Harnessing RLHF for Robust Unanswerability Recognition and Trustworthy Response Generation in LLMs

2025/07/22 by Shuyuan Lin, Lei Duan, Lin, Shuyuan +5
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2507.16951

openalex publication_date 2025/07/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Conversational Information Retrieval (CIR) systems, while offering intuitive access to information, face a significant challenge: reliably handling unanswerable questions to prevent the generation of misleading or hallucinated content. Traditional approaches often rely on external classifiers, which can introduce inconsistencies with the core generative Large Language Models (LLMs). This paper introduces Self-Aware LLM for Unanswerability (SALU), a novel approach that deeply integrates unanswerability detection directly within the LLM's generative process. SALU is trained using a multi-task learning framework for both standard Question Answering (QA) and explicit abstention generation for unanswerable queries. Crucially, it incorporates a confidence-score-guided reinforcement learning with human feedback (RLHF) phase, which explicitly penalizes hallucinated responses and rewards appropriate abstentions, fostering intrinsic self-awareness of knowledge boundaries. Through extensive experiments on our custom-built C-IR_Answerability dataset, SALU consistently outperforms strong baselines, including hybrid LLM-classifier systems, in overall accuracy for correctly answering or abstaining from questions. Human evaluation further confirms SALU's superior reliability, achieving high scores in factuality, appropriate abstention, and, most importantly, a dramatic reduction in hallucination, demonstrating its ability to robustly "know when to say 'I don't know'."

Citations

• Draw ALL Your Imagine: A Holistic Benchmark and Agent Framework for Complex Instruction-based Image Generation
• Rethinking Visual Dependency in Long-Context Reasoning for Large Vision-Language Models
• U-shaped and Inverted-U Scaling behind Emergent Abilities of Large Language Models
• OLAPH: Improving Factuality in Biomedical Long-form Question Answering
• Visual In-Context Learning for Large Vision-Language Models
• Uncertainty-aware Language Modeling for Selective Question Answering
• A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
• Hint-enhanced In-Context Learning wakes Large Language Models up for knowledge-intensive tasks
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners
• PaLM: Scaling Language Modeling with Pathways
• ClarET: Pre-training a Correlation-Aware Context-To-Event Transformer for Event-Centric Generation and Classification
• EventBERT: A Pre-Trained Model for Event Correlation Reasoning
• TruthfulQA: Measuring How Models Mimic Human Falsehoods
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Evaluating the Factual Consistency of Abstractive Text Summarization
• Say What I Want: Towards the Dark Side of Neural Dialogue Models
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• Contextual Candor: Enhancing LLM Trustworthiness Through Hierarchical Unanswerability Detection
• Uncertainty-Based Abstention in LLMs Improves Safety and Reduces Hallucinations
• TruthRL: Incentivizing Truthful LLMs via Reinforcement Learning
• AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions
• Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedback