Applied Research Scientist · Psychometrician
Jiayi Deng, Ph.D.
I design and evaluate fair, valid, and human-centered measurement systems—including AI-assisted decision workflows.
My work combines psychometrics, behavioral data, and responsible AI evaluation to study how high-impact assessment and selection systems perform across people, contexts, and decision workflows. I build reproducible methods for validation, subgroup comparability, human oversight, and evidence-based risk assessment.
Featured project
Human-AI Fairness Audit Lab
A public, reproducible audit of a synthetic AI-assisted candidate-review workflow.
The lab evaluates not only model predictions, but the complete human-AI decision system: held-out performance, subgroup and intersectional outcomes, calibration, counterfactual behavior, human overrides, escalation policies, and monitoring regressions.
Synthetic, non-hiring demonstration: the lab uses synthetic applicants and simulated model/reviewer decisions. It is an evaluation and research demonstration, not a hiring recommendation system, and its statistical disparities do not establish real-world discrimination or legal compliance.
- Synthetic cases
- 6,000 / 2,000 / 2,400 / 800
- Automated tests
- 56
- Held-out AUC
- v1 0.736 · v2 0.746
- Sensitivity suites
- Direct-input and end-to-end
Research focus
Measurement science for human-centered AI and assessment systems
AI fairness and human-AI evaluation
Evaluation of model outputs, end-to-end workflows, human overrides, counterfactual behavior, monitoring, and responsible-use boundaries.
Psychometric validity and subgroup comparability
Item response theory, measurement invariance, differential item and distractor functioning, reliability, equating, and calibration.
Behavioral and process data
Response time, rapid guessing, aberrant response patterns, cognitive labs, think-aloud protocols, and structured behavioral coding.
Evidence synthesis
Systematic review and meta-analysis, including current work on how AI-enabled interventions affect student motivation and engagement.
Experience
Applied research across assessment, selection, and AI-assisted expert workflows
At the Human Resources Research Organization, Jiayi contributes to psychometric research and independent quality review across state assessment programs and a medical-school admissions situational judgment assessment. She also evaluates generative-AI-assisted expert workflows for standards alignment and evidence mapping.
Research Scientist · Human Resources Research Organization
Psychometric validation, independent quality review, AI-assisted alignment evaluation, and response-process research.
Applied Psychometric Roles · Pearson / Pearson VUE
Applied psychometric analysis, progress-monitoring evaluation, simulation research, and conference proposal development.
Research and Teaching · University of Minnesota
IRT, equating, invariance, response-time, simulation, meta-analytic research, and measurement/statistics teaching.
Selected research
Publications and current manuscripts
Under review
Can AI Motivate Students? A Systematic Review and Meta-analysis on AI's Role in Student Motivation and Engagement.
Applied Psychological Measurement · 2024
Gorney, K., & Deng, J. `aberrance`: An R package for detecting aberrant behavior in test data.
Educational and Psychological Measurement · 2024
Rios, J. A., & Deng, J. Is effort-moderated scoring robust to multidimensional rapid guessing?
Applied Psychological Measurement · 2022
Deng, J., & Rios, J. A. Investigating the effect of differential rapid guessing on population invariance in equating.
Let's connect