Making Large Language Models Better Reasoners with Step-Aware Verifier

2022/06/06 by Yifei Li, Zeqi Lin, Li, Yifei +12 · 2 voices · 64 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling #cs.AI #cs.CL

paper · pdf · doi:10.48550/arxiv.2206.02336

openalex publication_date 2022/06/06 · arxiv published 2022/06/06 · arxiv updated 2023/05/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Few-shot learning is a challenging task that requires language models to generalize from limited examples. Large language models like GPT-3 and PaLM have made impressive progress in this area, but they still face difficulties in reasoning tasks such as GSM8K, a benchmark for arithmetic problems. To improve their reasoning skills, previous work has proposed to guide the language model with prompts that elicit a series of reasoning steps before giving the final answer, achieving a significant improvement on GSM8K from 17.9% to 58.1% in problem-solving rate. In this paper, we present DIVERSE (Diverse Verifier on Reasoning Step), a novel approach that further enhances the reasoning capability of language models. DIVERSE has three main components: first, it generates diverse prompts to explore different reasoning paths for the same question; second, it uses a verifier to filter out incorrect answers based on a weighted voting scheme; and third, it verifies each reasoning step individually instead of the whole chain. We evaluate DIVERSE on the latest language model code-davinci-002 and show that it achieves new state-of-the-art results on six of eight reasoning benchmarks (e.g., GSM8K 74.4% to 83.2%).

Cited by

• AutoICE: Automatically Synthesizing Verifiable C Code via LLM-driven Evolution
• From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Models
• In-Token Rationality Optimization: Towards Accurate and Concise LLM Reasoning via Self-Feedback
• HalluClean: A Unified Framework to Combat Hallucinations in LLMs
• Test-time Diverse Reasoning by Riemannian Activation Steering
• Synthetic Data-Driven Prompt Tuning for Financial QA over Tables and Documents
• CGES: Confidence-Guided Early Stopping for Efficient and Accurate Self-Consistency
• Efficient Test-Time Retrieval Augmented Generation
• Verifying Large Language Models' Reasoning Paths via Correlation Matrix Rank
• Can Language Models Compose Skills In-Context?
• RETuning: Upgrading Inference-Time Scaling for Stock Movement Prediction with Large Language Models
• UniCode: Augmenting Evaluation for Code Reasoning
• Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanism
• Hallucination Detection via Internal States and Structured Reasoning Consistency in Large Language Models
• Cog-Rethinker: Hierarchical Metacognitive Reinforcement Learning for LLM Reasoning
• PHANTOM RECALL: When Familiar Puzzles Fool Smart Models
• Tracing the Traces: Latent Temporal Signals for Efficient and Accurate Reasoning
• ReTraceQA: Evaluating Reasoning Traces of Small Language Models in Commonsense Question Answering
• Best-of-Majority: Minimax-Optimal Strategy for Pass@k Inference Scaling
• SoT: Structured-of-Thought Prompting Guides Multilingual Reasoning in Large Language Models
• Generalized Parallel Scaling with Interdependent Generations
• GRPO-λ: Credit Assignment improves LLM Reasoning
• DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Search
• SCI-Verifier: Scientific Verifier with Thinking
• ContextPRM: Leveraging Contextual Coherence for multi-domain Test-Time Scaling
• Scaling LLM Test-Time Compute with Mobile NPU on Smartphones
• Audited Reasoning Refinement: Fine-Tuning Language Models via LLM-Guided Step-Wise Evaluation and Correction
• A Probabilistic Inference Scaling Theory for LLM Self-Correction
• Trust but Verify! A Survey on Verification Design for Test-time Scaling
• Your Reward Function for RL is Your Best PRM for Search: Unifying RL and Search-Based TTS
• Improving Value-based Process Verifier via Low-Cost Variance Reduction
• Cooper: Co-Optimizing Policy and Reward Models in Reinforcement Learning for Large Language Models
• Learning to Reason Across Parallel Samples for LLM Reasoning
• Enhancing Test-Time Scaling of Large Language Models with Hierarchical Retrieval-Augmented MCTS
• Harnessing the Wisdom of LLM Crowds through Complementarity-Driven Iterative Collaboration
• DynScaling: Efficient Verifier-free Inference Scaling via Dynamic and Integrated Sampling
• SPARE: Single-Pass Annotation with Reference-Guided Evaluation for Automatic Process Supervision and Reward Modelling
• Multi-Step Visual Reasoning with Visual Tokens Scaling and Verification
• Med-PRM: Medical Reasoning Models with Stepwise, Guideline-verified Process Rewards
• The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity
• Leveraging Large Language Models for Bengali Math Word Problem Solving with Chain of Thought Reasoning
• Enhancing Logical Reasoning in Language Models via Symbolically-Guided Monte Carlo Process Supervision
• Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewards
• SIPDO: Closed-Loop Prompt Optimization via Synthetic Data Feedback
• The Quest for Efficient Reasoning: A Data-Centric Benchmark to CoT Distillation
• ProgRM: Build Better GUI Agents with Progress Rewards
• Stepwise Reasoning Checkpoint Analysis: A Test Time Scaling Method to Enhance LLMs' Reasoning
• T²: An Adaptive Test-Time Scaling Strategy for Contextual Question Answering
• Optimal Policy Minimum Bayesian Risk
• Generalizable Process Reward Models via Formally Verified Training Data
• SATBench: Benchmarking LLMs' Logical Reasoning via Automated Puzzle Generation from SAT Formulas
• AdaBoN: Adaptive Best-of-N Alignment
• Rethinking the Role of Prompting Strategies in LLM Test-Time Scaling: A Perspective of Probability Theory
• HAPO: Training Language Models to Reason Concisely via History-Aware Policy Optimization
• Real-Time Verification of Embodied Reasoning for Generative Skill Acquisition
• Learning from Peers in Reasoning Models
• When Reasoning Beats Scale: A 1.5B Reasoning Model Outranks 13B LLMs as Discriminator
• PAMT: Process-Aligned Reinforcement Learning for Multi-Domain Machine Translation
• Process Reward Models That Think
• CoT-RAG: Integrating Chain of Thought and Retrieval-Augmented Generation to Enhance Reasoning in Large Language Models
• Could Thinking Multilingually Empower LLM Reasoning?
• Refining Over Resampling: Test-Time Self-Correction for LLM Reasoning
• A Survey of Reasoning with Foundation Models: Concepts, Methodologies, and Outlook
• Sample, Don't Search: Rethinking Test-Time Alignment for Language Models

Discussions

• DiVeRSe - On the Advance of Making Language Models Better Reasoners [hn, 2 points, 1 comments]
• On the Advance of Making Language Models Better Reasoners [hn, 2 points, 0 comments]

Related

• Not All Votes Count! Programs as Verifiers Improve Self-Consistency of Language Models for Math Reasoning
• Patience Is The Key to Large Language Model Reasoning
• BeyondBench: Benchmark-Free Evaluation of Reasoning in Language Models
• Masked Thought: Simply Masking Partial Reasoning Steps Can Improve Mathematical Reasoning Learning of Language Models
• Multi-Step Reasoning with Large Language Models, a Survey