MindShift: Analyzing Language Models' Reactions to Psychological Prompts

2025/12/09 by Anton Vasiliuk, Vasiliuk, Anton, Irina Abdullaeva +7
Psychology · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Digital Mental Health Interventions #FOS: Computer and information sciences #Mental Health via Writing #Personality Traits and Psychology

paper · pdf · doi:10.48550/arxiv.2512.09149

openalex publication_date 2025/12/09 · openalex created_date 2025/12/12 · openalex updated_date 2026/07/28

Abstract

Large language models (LLMs) hold the potential to absorb and reflect personality traits and attitudes specified by users. In our study, we investigated this potential using robust psychometric measures. We adapted the most studied test in psychological literature, namely Minnesota Multiphasic Personality Inventory (MMPI) and examined LLMs' behavior to identify traits. To asses the sensitivity of LLMs' prompts and psychological biases we created personality-oriented prompts, crafting a detailed set of personas that vary in trait intensity. This enables us to measure how well LLMs follow these roles. Our study introduces MindShift, a benchmark for evaluating LLMs' psychological adaptability. The results highlight a consistent improvement in LLMs' role perception, attributed to advancements in training datasets and alignment techniques. Additionally, we observe significant differences in responses to psychometric assessments across different model types and families, suggesting variability in their ability to emulate human-like personality traits. MindShift prompts and code for LLM evaluation will be publicly available.

Citations

• Assessment and manipulation of latent constructs in pre-trained language models using psychometric scales
• The Better Angels of Machine Personality: How Personality Relates to LLM Safety
• Do LLMs Have Distinct and Consistent Personality? TRAIT: Personality Testset designed for LLMs with Psychometrics
• The Falcon Series of Open Language Models
• You don't need a personality test to know these models are unreliable: Assessing the Reliability of Large Language Models on Psychometric Instruments
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Have Large Language Models Developed a Personality?: Applicability of Self-Assessment Tests in Measuring Personality in LLMs
• Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies
• Personalizing Dialogue Agents: I have a dog, do you have pets too?
• Inducing anxiety in large language models can induce bias
• CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge

Related

• Evaluating Large Language Models with Psychometrics
• Illuminating the Black Box: A Psychometric Investigation into the Multifaceted Nature of Large Language Models
• A Comparative Study of Large Language Models and Human Personality Traits
• Humanizing LLMs: A Survey of Psychological Measurements with Tools, Datasets, and Human-Agent Applications
• Revisiting the Reliability of Psychological Scales on Large Language Models