Exploring Chain-of-Thought Reasoning for Steerable Pluralistic Alignment

2025/10/05 by Zhang, Yunfan, Kathleen McKeown, Smaranda Muresan +2
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multi-Agent Systems and Negotiation #Semantic Web and Ontologies

paper · pdf · doi:10.48550/arxiv.2510.04045

openalex publication_date 2025/10/05 · openalex created_date 2025/10/09 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) are typically trained to reflect a relatively uniform set of values, which limits their applicability to tasks that require understanding of nuanced human perspectives. Recent research has underscored the importance of enabling LLMs to support steerable pluralism -- the capacity to adopt a specific perspective and align generated outputs with it. In this work, we investigate whether Chain-of-Thought (CoT) reasoning techniques can be applied to building steerable pluralistic models. We explore several methods, including CoT prompting, fine-tuning on human-authored CoT, fine-tuning on synthetic explanations, and Reinforcement Learning with Verifiable Rewards (RLVR). We evaluate these approaches using the Value Kaleidoscope and OpinionQA datasets. Among the methods studied, RLVR consistently outperforms others and demonstrates strong training sample efficiency. We further analyze the generated CoT traces with respect to faithfulness and safety.

Citations

• Reasoning Models Don't Always Say What They Think
• Competitive Programming with Large Reasoning Models
• Qwen2.5 Technical Report
• Large Language Models Reflect the Ideology of their Creators
• Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration
• Focus on Your Question! Interpreting and Mitigating Toxic CoT Problems in Commonsense Reasoning
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• Aligning Language Models with Human Preferences via a Bayesian Approach
• Factuality Challenges in the Era of Large Language Models
• Factuality challenges in the era of large language models and opportunities for fact-checking
• Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
• Whose Opinions Do Language Models Reflect?
• On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning
• Evaluating the Factual Consistency of Large Language Models Through News Summarization
• Training language models to follow instructions with human feedback
• Finetuned Language Models Are Zero-Shot Learners

Related

• Chain-of-Thought as a Lens: Evaluating Structured Reasoning Alignment between Human Preferences and Large Language Models
• On the Hardness of Faithful Chain-of-Thought Reasoning in Large Language Models
• A Closer Look at Bias and Chain-of-Thought Faithfulness of Large (Vision) Language Models
• A Geometric Perspective on Stabilizing Value Conflict Resolution
• Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens