Beyond Correctness: Confidence-Aware Reward Modeling for Enhancing Large Language Model Reasoning

2025/11/09 by Qianxi He, He, Qianxi, Ren, Qingyu +5
Computer Science · #Topic Modeling #Multimodal Machine Learning Applications #Explainable Artificial Intelligence (XAI)

paper · pdf · doi:10.48550/arxiv.2511.07483

Abstract

Recent advancements in large language models (LLMs) have shifted the post-training paradigm from traditional instruction tuning and human preference alignment toward reinforcement learning (RL) focused on reasoning capabilities. However, numerous technical reports indicate that purely rule-based reward RL frequently results in poor-quality reasoning chains or inconsistencies between reasoning processes and final answers, particularly when the base model is of smaller scale. During the RL exploration process, models might employ low-quality reasoning chains due to the lack of knowledge, occasionally producing correct answers randomly and receiving rewards based on established rule-based judges. This constrains the potential for resource-limited organizations to conduct direct reinforcement learning training on smaller-scale models. We propose a novel confidence-based reward model tailored for enhancing STEM reasoning capabilities. Unlike conventional approaches, our model penalizes not only incorrect answers but also low-confidence correct responses, thereby promoting more robust and logically consistent reasoning. We validate the effectiveness of our approach through static evaluations, Best-of-N inference tests, and PPO-based RL training. Our method outperforms several state-of-the-art open-source reward models across diverse STEM benchmarks. We release our codes and model in https://github.com/qianxiHe147/C2RM.

Citations

• Qwen3 Technical Report
• What Makes a Reward Model a Good Teacher? An Optimization Perspective
• Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs
• JudgeBench: A Benchmark for Evaluating LLM-based Judges
• Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown
• The Llama 3 Herd of Models
• Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts
• Break the Chain: Large Language Models Can be Shortcut Reasoners
• Secrets of RLHF in Large Language Models Part II: Reward Modeling
• GPQA: A Graduate-Level Google-Proof Q&A Benchmark
• Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs
• Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
• Do Language Models Know When They're Hallucinating References?
• Discovering Latent Knowledge in Language Models Without Supervision
• Language Models (Mostly) Know What They Know
• Teaching Models to Express Their Uncertainty in Words
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• Measuring Mathematical Problem Solving With the MATH Dataset
• Deep reinforcement learning from human preferences

Related

• Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty
• Confidence as a Reward: Transforming LLMs into Reward Models
• Confidence Is All You Need: Few-Shot RL Fine-Tuning of Language Models
• Cooper: Co-Optimizing Policy and Reward Models in Reinforcement Learning for Large Language Models
• Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Models