Evaluating LLM Adaptation to Sociodemographic Factors: User Profile vs. Dialogue History

2025/05/27 by Qishuai Zhong, Zhong, Qishuai, Zhi Li +5 · 2 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Human-Computer Interaction (cs.HC) #Natural Language Processing Techniques #Text Readability and Simplification

paper · pdf · doi:10.48550/arxiv.2505.21362

openalex publication_date 2025/05/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Effective engagement by large language models (LLMs) requires adapting responses to users' sociodemographic characteristics, such as age, occupation, and education level. While many real-world applications leverage dialogue history for contextualization, existing evaluations of LLMs' behavioral adaptation often focus on single-turn prompts. In this paper, we propose a framework to evaluate LLM adaptation when attributes are introduced either (1) explicitly via user profiles in the prompt or (2) implicitly through multi-turn dialogue history. We assess the consistency of model behavior across these modalities. Using a multi-agent pipeline, we construct a synthetic dataset pairing dialogue histories with distinct user profiles and employ questions from the Value Survey Module (VSM 2013) (Hofstede and Hofstede, 2016) to probe value expression. Our findings indicate that most models adjust their expressed values in response to demographic changes, particularly in age and education level, but consistency varies. Models with stronger reasoning capabilities demonstrate greater alignment, indicating the importance of reasoning in robust sociodemographic adaptation.

Citations

• An Evaluation of Cultural Value Alignment in LLM
• A Survey on LLM-as-a-Judge
• Does Prompt Formatting Have Any Impact on LLM Performance?
• Benchmarking Bias in Large Language Models during Role-Playing
• The Llama 3 Herd of Models
• CLAVE: An Adaptive Framework for Evaluating Values of LLM Generated Responses
• Are Large Language Models Consistent over Value-laden Questions?
• How Well Do LLMs Represent Values Across Cultures? Empirical Analysis of LLM Responses Based on Hofstede Cultural Dimensions
• A Complete Survey on LLM-based AI Chatbots
• MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
• WorldValuesBench: A Large-Scale Benchmark Dataset for Multi-Cultural Value Awareness of Language Models
• LLM Task Interference: An Initial Study on the Impact of Task-Switch in Conversational History
• Synthetic Dialogue Dataset Generation using LLM Agents
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Cultural Alignment in Large Language Models: An Explanatory Analysis Based on Hofstede's Cultural Dimensions
• Large Language Models as Superpositions of Cultural Perspectives
• Towards Measuring the Representation of Subjective Global Opinions in Language Models
• LiveChat: A Large-Scale Personalized Dialogue Dataset Automatically Constructed from Live Streaming
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• HumBEL: A Human-in-the-Loop Approach for Evaluating Demographic Factors of Language Models in Human-Machine Conversations
• Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
• Probing Pre-Trained Language Models for Cross-Cultural Differences in Values
• Pchatbot: A Large-Scale Dataset for Personalized Chatbot
• BART: Denoising Sequence-to-Sequence Pre-training for Natural Language\n Generation, Translation, and Comprehension
• Personalizing Dialogue Agents: I have a dog, do you have pets too?
• A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
• Can LLMs Grasp Implicit Cultural Values? Benchmarking LLMs' Cultural Intelligence with CQ-Bench

Cited by

• Prompt Fairness: Sub-group Disparities in LLMs
• Flipping the Dialogue: Training and Evaluating User Language Models

Related

• Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptions
• Exploring Robustness of LLMs to Sociodemographically-Conditioned Paraphrasing
• Should We Fine-Tune or RAG? Evaluating Different Techniques to Adapt LLMs for Dialogue
• Linear socio-demographic representations emerge in Large Language Models from indirect cues
• Topics as Proxies for Sociodemographics: How Conversational Context Affects LLM Answers