Evaluating LLMs at Detecting Errors in LLM Responses

2024/04/04 by Ryo Kamoi, Sarkar Snigdha Sarathi Das, Kamoi, Ryo +27 · 16 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Imbalanced Data Classification Techniques #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2404.03602

openalex publication_date 2024/04/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

With Large Language Models (LLMs) being widely used across various tasks, detecting errors in their responses is increasingly crucial. However, little research has been conducted on error detection of LLM responses. Collecting error annotations on LLM responses is challenging due to the subjective nature of many NLP tasks, and thus previous research focuses on tasks of little practical value (e.g., word sorting) or limited error types (e.g., faithfulness in summarization). This work introduces ReaLMistake, the first error detection benchmark consisting of objective, realistic, and diverse errors made by LLMs. ReaLMistake contains three challenging and meaningful tasks that introduce objectively assessable errors in four categories (reasoning correctness, instruction-following, context-faithfulness, and parameterized knowledge), eliciting naturally observed and diverse errors in responses of GPT-4 and Llama 2 70B annotated by experts. We use ReaLMistake to evaluate error detectors based on 12 LLMs. Our findings show: 1) Top LLMs like GPT-4 and Claude 3 detect errors made by LLMs at very low recall, and all LLM-based error detectors perform much worse than humans. 2) Explanations by LLM-based error detectors lack reliability. 3) LLMs-based error detection is sensitive to small changes in prompts but remains challenging to improve. 4) Popular approaches to improving LLMs, including self-consistency and majority vote, do not improve the error detection performance. Our benchmark and code are provided at https://github.com/psunlpgroup/ReaLMistake.

Cited by

• FLAWS: A Benchmark for Error Identification and Localization in Scientific Papers
• IF-CRITIC: Towards a Fine-Grained LLM Critic for Instruction-Following Evaluation
• Improving Human Verification of LLM Reasoning through Interactive Explanation Interfaces
• Towards Scalable Oversight with Collaborative Multi-Agent Debate in Error Detection
• Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMs
• Beyond Consensus: Mitigating the Agreeableness Bias in LLM Judge Evaluations
• Do LLMs Know They Are Being Tested? Evaluation Awareness and Incentive-Sensitive Failures in GPT-OSS-20B
• Where Did It All Go Wrong? A Hierarchical Look into Multi-Agent Error Attribution
• Sample, Align, Synthesize: Graph-Based Response Synthesis with ConGrs
• Hop, Skip, and Overthink: Diagnosing Why Reasoning Models Fumble during Multi-Hop Analysis
• Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in Large Language Models
• Diagnosing Failures in Large Language Models' Answers: Integrating Error Attribution into Evaluation Framework
• Self-Correction Bench: Uncovering and Addressing the Self-Correction Blind Spot in Large Language Models
• Evaluation of LLMs for mathematical problem solving
• LLMs are Better Than You Think: Label-Guided In-Context Learning for Named Entity Recognition
• Mis-prompt: Benchmarking Large Language Models for Proactive Error Handling

Related

• See What LLMs Cannot Answer: A Self-Challenge Framework for Uncovering LLM Weaknesses
• CLEAR: Error Analysis via LLM-as-a-Judge Made Easy
• LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond
• EVADE: LLM-Based Explanation Generation and Validation for Error Detection in NLI
• Mis-prompt: Benchmarking Large Language Models for Proactive Error Handling