Teaching Large Language Models to Maintain Contextual Faithfulness via Synthetic Tasks and Reinforcement Learning

2025/05/22 by Si, Shuzheng, Zhao, Haozhe, Gao, Cheng +11 · 3 citations
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2505.16483

Abstract

Teaching large language models (LLMs) to be faithful in the provided context is crucial for building reliable information-seeking systems. Therefore, we propose a systematic framework, CANOE, to reduce faithfulness hallucinations of LLMs across different downstream tasks without human annotations. Specifically, we first synthesize short-form question-answering (QA) data with four diverse tasks to construct high-quality and easily verifiable training data without human annotation. Also, we propose Dual-GRPO, a rule-based reinforcement learning method that includes three tailored rule-based rewards derived from synthesized short-form QA data, while simultaneously optimizing both short-form and long-form response generation. Notably, Dual-GRPO eliminates the need to manually label preference data to train reward models and avoids over-optimizing short-form generation when relying only on the synthesized short-form QA data. Experimental results show that CANOE greatly improves the faithfulness of LLMs across 11 different tasks, even outperforming the most advanced LLMs, e.g., GPT-4o and OpenAI o1.

Cited by

• From Context to EDUs: Faithful and Structured Context Compression via Elementary Discourse Unit Decomposition
• RhinoInsight: Improving Deep Research through Control Mechanisms for Model Behavior and Context
• Copy-Paste to Mitigate Large Language Model Hallucinations

Related

• Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
• Lessons from Training Grounded LLMs with Verifiable Rewards
• SPELL: Self-Play Reinforcement Learning for evolving Long-Context Language Models
• Supervised Knowledge Makes Large Language Models Better In-context Learners
• A Technical Survey of Reinforcement Learning Techniques for Large Language Models