Automated Item Neutralization for Non-Cognitive Scales: A Large Language Model Approach to Reducing Social-Desirability Bias

2025/09/09 by Wu, Sirui, Yang, Daijin
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computers and Society (cs.CY) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2509.19314

Abstract

This study evaluates item neutralization assisted by the large language model (LLM) to reduce social desirability bias in personality assessment. GPT-o3 was used to rewrite the International Personality Item Pool Big Five Measure (IPIP-BFM-50), and 203 participants completed either the original or neutralized form along with the Marlowe-Crowne Social Desirability Scale. The results showed preserved reliability and a five-factor structure, with gains in Conscientiousness and declines in Agreeableness and Openness. The correlations with social desirability decreased for several items, but inconsistently. Configural invariance held, though metric and scalar invariance failed. Findings support AI neutralization as a potential but imperfect bias-reduction method.

Citations

• The Relationship Between Reasoning and Performance in Large Language Models -- o3 (mini) Thinks Harder, Not Longer
• Exploring Social Desirability Response Bias in Large Language Models: Evidence from GPT-4 Simulations
• Large Language Models Show Human-like Social Desirability Biases in Survey Responses
• Better Zero-Shot Reasoning with Role-Play Prompting
• Prompt Sapper: A LLM-Empowered Production Tool for Building AI Chains
• GPT-3: Its Nature, Scope, Limits, and Consequences
• Measurement invariance conventions and reporting: The state of the art and future directions for psychological research
• The international personality item pool and the future of public-domain personality measures
• Common method biases in behavioral research: A critical review of the literature and recommended remedies.
• IMPROVE: Iterative Model Pipeline Refinement and Optimization Leveraging LLM Experts

Related

• Large Language Models Show Human-like Social Desirability Biases in Survey Responses
• Investigating the Impact of LLM Personality on Cognitive Bias Manifestation in Automated Decision-Making Tasks
• Mitigating Social Desirability Bias in Random Silicon Sampling
• Exploring Social Desirability Response Bias in Large Language Models: Evidence from GPT-4 Simulations
• An IRT Examination of the Psychometric Functioning of Negatively Worded Personality Items