Chain-of-Thought as a Lens: Evaluating Structured Reasoning Alignment between Human Preferences and Large Language Models

2025/11/09 by Wang, Boxuan, Li, Zhuoyun, Huang, Xinmiao +2 · 1 citation
Computer Science · #Advanced Graph Neural Networks #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Topic Modeling

paper · doi:10.48550/arxiv.2511.06168

openalex publication_date 2025/11/09 · openalex created_date 2025/11/12 · openalex updated_date 2026/07/28

Abstract

This paper primarily demonstrates a method to quantitatively assess the alignment between multi-step, structured reasoning in large language models and human preferences. We introduce the Alignment Score, a semantic-level metric that compares a model-produced chain of thought traces with a human-preferred reference by constructing semantic-entropy-based matrices over intermediate steps and measuring their divergence. Our analysis shows that Alignment Score tracks task accuracy across models and hop depths, and peaks at 2-hop reasoning. Empirical results further indicate that misalignment at greater reasoning depths is driven mainly by alignment errors such as thematic shift and redundant reasoning. Viewing chain sampling as drawing from a distribution over reasoning paths, we empirically demonstrate a strong and consistent correlation between Alignment Score and accuracy, readability, and coherence, supporting its use as a diagnostic signal. The code is available.

Citations

• Stop Reducing Responsibility in LLM-Powered Multi-Agent Systems to Local Alignment
• Semantic Energy: Detecting LLM Hallucination Beyond Entropy
• Tapas Are Free! Training-Free Adaptation of Programmatic Agents via LLM-Guided Program Synthesis in Dynamic Environments
• An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliability
• Verbosity-Aware Rationale Reduction: Effective Reduction of Redundant Rationale via Principled Criteria
• Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse
• Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs
• Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities
• General Purpose Verification for Chain of Thought Prompting
• Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph
• Direct Evaluation of Chain-of-Thought in Multi-hop Reasoning with Knowledge Graphs
• A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chains
• Over-Reasoning and Redundant Calculation of Large Language Models
• GPQA: A Graduate-Level Google-Proof Q&A Benchmark
• JudgeLM: Fine-tuned Large Language Models are Scalable Judges
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
• ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning
• Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
• ReAct: Synergizing Reasoning and Acting in Language Models
• Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
• Large Language Models are Zero-Shot Reasoners
• Can language models learn from explanations in context?
• DREAM: Improving Situational QA by First Elaborating the Situation
• DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing
• Training Verifiers to Solve Math Word Problems
• TruthfulQA: Measuring How Models Mimic Human Falsehoods
• Measuring Mathematical Problem Solving With the MATH Dataset
• Learning to summarize from human feedback
• BART: Denoising Sequence-to-Sequence Pre-training for Natural Language\n Generation, Translation, and Comprehension
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• A Diversity-Promoting Objective Function for Neural Conversation Models
• Divergence measures based on the Shannon entropy

Cited by

• Rethinking Multi-Agent Intelligence Through the Lens of Small-World Networks

Related

• Making Large Language Models Better Reasoners with Alignment
• Exploring Chain-of-Thought Reasoning for Steerable Pluralistic Alignment
• CASE: Causal Alignment and Structural Enforcement for Improving Chain-of-Thought Faithfulness
• FRIT: Using Causal Importance to Improve Chain-of-Thought Faithfulness
• Improving Consistency in Large Language Models through Chain of Guidance