Self-Improvement of Language Models by Post-Training on Multi-Agent Debate

2025/09/18 by Ankur Samanta, Akshayaa Magesh, Samanta, Ankur +16
Computer Science · #Artificial Intelligence (cs.AI) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2509.15172

openalex publication_date 2025/09/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/30

Abstract

Self-improvement, where models improve beyond their current performance without external supervision, remains a challenge. The core difficulty is sourcing a training signal stronger than what the model itself can currently produce. Majority voting has been shown to provide such a signal by aggregating over multiple samples, helping mitigate some of the inconsistencies in LM reasoning. In this work, we show that multi-agent debate--where models collaborate and exchange reasoning over multiple rounds--provides an even richer signal than single-round majority voting. We introduce Multi-Agent Consensus Alignment (MACA), which uses reinforcement learning (RL) to post-train models to effectively utilize multi-agent debate. We find that preference learning over full reasoning traces, learning to differentiate between majority and minority reasoning, is more effective than binary consensus rewards or SFT-based approaches for leveraging these debate signals. This produces three key improvements: models are (1) better at utilizing the multi-agent debate setting (+26.87% on MATH), (2) individually more accurate (+21.51% on MathQA), and (3) more self-consistent (+27.6% on GSM8K). We also see strong generalization to unseen benchmarks (+16.3% on GPQA, +11.6% on CommonsenseQA).

Citations

• Robust Reward Modeling via Causal Rubrics
• Arbiters of Ambivalence: Challenges of Using LLMs in No-Consensus Tasks
• Nemotron-CrossThink: Scaling Self-Learning beyond Math Reasoning
• Preference Leakage: A Contamination Problem in LLM-as-a-judge
• Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains
• Exploiting Structure in Offline Multi-Agent RL: The Benefits of Low Interaction Rank
• Path-Consistency with Prefix Enhancement for Efficient Inference in LLMs
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• Universal Self-Consistency for Large Language Model Generation
• GPQA: A Graduate-Level Google-Proof Q&A Benchmark
• Large Language Models Cannot Self-Correct Reasoning Yet
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• MADiff: Offline Multi-agent Learning with Diffusion Models
• Improving Factuality and Reasoning in Language Models through Multiagent Debate
• QLoRA: Efficient Finetuning of Quantized LLMs
• Self-Prompting Large Language Models for Zero-Shot Open-Domain QA
• Self-Consistency Improves Chain of Thought Reasoning in Language Models
• Training Verifiers to Solve Math Word Problems
• Evaluating Large Language Models Trained on Code
• Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning
• Are NLP Models really able to Solve Simple Math Word Problems?
• Measuring Mathematical Problem Solving With the MATH Dataset
• Measuring and Improving Consistency in Pretrained Language Models
• MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms
• The Curious Case of Neural Text Degeneration
• A Theory of Social Comparison Processes
• CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge
• An Integrative Theory of Prefrontal Cortex Function

Related

• Self-Consistency Is Losing Its Edge: Diminishing Returns and Rising Costs in Modern LLMs
• When Disagreements Elicit Robustness: Investigating Self-Repair Capabilities under LLM Multi-Agent Disagreements
• Just Ask One More Time! Self-Agreement Improves Reasoning of Language Models in (Almost) All Scenarios
• ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs
• Understanding Bias Reinforcement in LLM Agents Debate