Large Language Models in Human Subject Research, and the Presence of Idiosyncratic Human Behaviors

2025/11/01 by Dionysios S. Demetis, Jeandri Robertson, Andrew Flostrand +6 · 1 voice
Social Sciences · Medicine · #Language and cultural evolution #Artificial Intelligence in Healthcare and Education #Computational and Text Analysis Methods

paper · doi:10.1109/mitp.2025.3633473

openalex publication_date 2025/11/01 · openalex created_date 2025/12/22 · openalex updated_date 2026/07/02

Abstract

Large language models (LLMs) not only mimic human language generally but are also showing the ability to exhibit nuanced, idiosyncratic human behavior. This could lead to the expanded use of LLMs in research and a diminution of the influence of humans. It also has implications for AI alignment and the role of the technology in society.

Citations

• Factuality challenges in the era of large language models and opportunities for fact-checking
• Evaluating large language models in theory of mind tasks
• Resisting the Algorithmic Management of Science: Craft and Community After Generative AI
• Twenty Years of Personality Computing: Threats, Challenges and Future Directions
• Investigating the Impact of LLM Personality on Cognitive Bias Manifestation in Automated Decision-Making Tasks
• Designing AI-Agents with Personalities: A Psychometric Approach
• Take Caution in Using LLMs as Human Surrogates: Scylla Ex Machina
• Large Language Models Show Human-like Social Desirability Biases in Survey Responses
• Factuality Challenges in the Era of Large Language Models

Discussions

• Some signs of the presence of subtle but important behavioral traits in LLMs, typically attributed to humans. This could profoundly affect both behavioral research and agentic use of AI in society. d [bsky, 4 points, 0 comments]

Related

• The Emergence of Social Science of Large Language Models
• The Quo Vadis of the Relationship between Language and Large Language Models
• Talking About Large Language Models
• Large Language Models: The Need for Nuance in Current Debates and a Pragmatic Perspective on Understanding
• Large Language Models and the Reverse Turing Test