Convergent Validation of Trait Induction in LLMs

2026/05/01 by Yasin Edin, Leon Rosenplänter, Martin Kersting · 1 voice
Decision Sciences · Psychology · #Psychometric Methodologies and Testing #Personality Traits and Psychology #Mental Health via Writing

paper · doi:10.1027/2698-1866/a000127

openalex publication_date 2026/05/01 · openalex created_date 2026/05/05 · openalex updated_date 2026/06/26

Abstract

Abstract: This study investigates an approach to test a key requirement for using large language models (LLMs) to simulate valid data sets of item-responses, namely the effect of trait induction on another related construct. Using OpenAI's GPT-4, we used zero-shot prompting to induce high- and low-agreeableness profiles in the model's output. We then investigated, in terms of convergent construct validation, the effect of trait induction on another theoretically and empirically related construct: emotion understanding (EU). To measure EU, we utilized a situational judgment test developed to test LLMs. Using n = 680 simulated data sets, we show that a prompt on a trait (agreeableness) has a plausible effect on unprompted behavior in a skill (EU). These results provide initial support for evaluating the effects of trait induction through methods of convergent construct validation when assessing LLM-generated responses, and for using LLMs to generate data for foundational steps of psychometric test development.

Citations

• Convergent and discriminant validation by the multitrait-multimethod matrix.
• The Geographic Distribution of Big Five Personality Traits
• Gender bias and stereotypes in Large Language Models
• Bias and Fairness in Large Language Models: A Survey
• Exploring the Frontiers of LLMs in Psychological Applications: A Comprehensive Review
• Bias and Fairness in Large Language Models: A Survey
• Emotional Intelligence of Large Language Models
• Out of One, Many: Using Language Models to Simulate Human Samples

Discussions

• Happy to share that my first co-first-authored paper, "Convergent Validation of Trait Induction in LLMs", is now out in PTAD! A big thank you to my co-authors Yasin Edin & Martin Kersting (@psydiagjlu [bsky, 4 points, 0 comments]

Related

• Evaluating LLM Alignment on Personality Inference from Real-World Interview Data
• Do LLMs Have Distinct and Consistent Personality? TRAIT: Personality Testset designed for LLMs with Psychometrics
• Humanizing LLMs: A Survey of Psychological Measurements with Tools, Datasets, and Human-Agent Applications
• BIG5-CHAT: Shaping LLM Personalities Through Training on Human-Grounded Data
• From Prompts to Constructs: A Dual-Validity Framework for LLM Research in Psychology