Reasoning Models Hallucinate More: Factuality-Aware Reinforcement Learning for Large Reasoning Models

2025/05/30 by Junyi Li, Li, Junyi, Hwee Tou Ng +1 · 4 citations
Arts and Humanities · Computer Science · #Artificial Intelligence (cs.AI) #Blockchain Technology Applications and Security #Computability, Logic, AI Algorithms #Computation and Language (cs.CL) #FOS: Computer and information sciences #Mental Health and Psychiatry

paper · pdf · doi:10.48550/arxiv.2505.24630

openalex publication_date 2025/05/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/30

Abstract

Large language models (LLMs) have significantly advanced in reasoning tasks through reinforcement learning (RL) optimization, achieving impressive capabilities across various challenging benchmarks. However, our empirical analysis reveals a critical drawback: reasoning-oriented RL fine-tuning significantly increases the prevalence of hallucinations. We theoretically analyze the RL training dynamics, identifying high-variance gradient, entropy-induced randomness, and susceptibility to spurious local optima as key factors leading to hallucinations. To address this drawback, we propose Factuality-aware Step-wise Policy Optimization (FSPO), an innovative RL fine-tuning algorithm incorporating explicit factuality verification at each reasoning step. FSPO leverages automated verification against given evidence to dynamically adjust token-level advantage values, incentivizing factual correctness throughout the reasoning process. Experiments across mathematical reasoning and hallucination benchmarks using Qwen2.5 and Llama models demonstrate that FSPO effectively reduces hallucinations while enhancing reasoning accuracy, substantially improving both reliability and performance.

Cited by

• Understanding New-Knowledge-Induced Factual Hallucinations in LLMs: Analysis and Interpretation
• MR-Align: Meta-Reasoning Informed Factuality Alignment for Large Reasoning Models
• Knowledge-Level Consistency Reinforcement Learning: Dual-Fact Alignment for Long-Form Factuality
• Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewards
• A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models

Related

• KnowRL: Exploring Knowledgeable Reinforcement Learning for Factuality
• Detection and Mitigation of Hallucination in Large Reasoning Models: A Mechanistic Perspective
• Reasoning Error from Known Fact: Step-Level Self-Consistency Group Relative Policy Optimization for LLM
• The Hallucination Tax of Reinforcement Finetuning
• Thinking, Faithful and Stable: Mitigating Hallucinations in LLMs