NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark

2023/10/27 by Oscar Sainz, Sainz, Oscar, Jon Ander Campos +9 · 52 citations
Computer Science · #Computation and Language (cs.CL) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2310.18018

openalex publication_date 2023/10/27 · openalex created_date 2023/11/01 · openalex updated_date 2026/07/28

Abstract

In this position paper, we argue that the classical evaluation on Natural Language Processing (NLP) tasks using annotated benchmarks is in trouble. The worst kind of data contamination happens when a Large Language Model (LLM) is trained on the test split of a benchmark, and then evaluated in the same benchmark. The extent of the problem is unknown, as it is not straightforward to measure. Contamination causes an overestimation of the performance of a contaminated model in a target benchmark and associated task with respect to their non-contaminated counterparts. The consequences can be very harmful, with wrong scientific conclusions being published while other correct ones are discarded. This position paper defines different levels of data contamination and argues for a community effort, including the development of automatic and semi-automatic measures to detect when data from a benchmark was exposed to a model, and suggestions for flagging papers with conclusions that are compromised by data contamination.

Cited by

• Reasoning Gets Harder for LLMs Inside A Dialogue
• Eval Factsheets: A Structured Framework for Documenting AI Evaluations
• FiMMIA: scaling semantic perturbation-based membership inference across modalities
• FLAWS: A Benchmark for Error Identification and Localization in Scientific Papers
• TrackList: Tracing Back Query Linguistic Diversity for Head and Tail Knowledge in Open Large Language Models
• Simulated Self-Assessment in Large Language Models: A Psychometric Approach to AI Self-Efficacy
• When Words Change the Model: Sensitivity of LLMs for Constraint Programming Modelling
• SPAN: Benchmarking and Improving Cross-Calendar Temporal Reasoning of Large Language Models
• SynQuE: Estimating Synthetic Dataset Quality Without Annotations
• EvalCards: A Framework for Standardized Evaluation Reporting
• DecompSR: A dataset for decomposed analyses of compositional multihop spatial reasoning
• Empirical and Sustainability Aspects of Software Engineering Research in the Era of Large Language Models: A Reflection
• Questionnaire meets LLM: A Benchmark and Empirical Study of Structural Skills for Understanding Questions and Responses
• Not ready for the bench: LLM legal interpretation is unstable and out of step with human judgments
• EsoLang-Bench: Evaluating Genuine Reasoning in Large Language Models via Esoteric Programming Languages
• Robustness as an Emergent Property of Task Performance
• SynthWorlds: Controlled Parallel Worlds for Disentangling Reasoning and Knowledge in Language Models
• What do model reports say about their ChemBio benchmark evaluations? Comparing recent releases to the STREAM framework
• Citation Failure: Definition, Analysis and Efficient Mitigation
• AcademicEval: Live Long-Context LLM Benchmark
• DMRetriever: A Family of Models for Improved Text Retrieval in Disaster Management
• Deep Associations, High Creativity: A Simple yet Effective Metric for Evaluating Large Language Models
• Syntactic Blind Spots: How Misalignment Leads to LLMs Mathematical Errors
• Provable Training Data Identification for Large Language Models
• Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Models
• Haystack Engineering: Context Engineering for Heterogeneous and Agentic Long-Context Evaluation
• Quantifying Data Contamination in Psychometric Evaluations of LLMs
• MatheMagic: Generating Dynamic Mathematics Benchmarks Robust to Memorization
• Language Model as Planner and Formalizer under Constraints
• The Flaw of Averages: Quantifying Uniformity of Performance on Benchmarks
• OpenAnt: LLM-Powered Vulnerability Discovery Through Code Decomposition, Adversarial Verification, and Dynamic Testing
• EngiBench: A Benchmark for Evaluating Large Language Models on Engineering Problem Solving
• Cross-Attention is Half Explanation in Speech-to-Text Models
• From Ground Trust to Truth: Disparities in Offensive Language Judgments on Contemporary Korean Political Discourse
• Data Leakage in Visual Datasets
• EvoEmpirBench: Dynamic Spatial Reasoning with Agent-ExpVer
• LVLMs are Bad at Overhearing Human Referential Communication
• MatQnA: A Benchmark Dataset for Multi-modal Large Language Models in Materials Characterization and Analysis
• Performance Assessment Strategies for Generative AI Applications in Healthcare
• The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMs
• Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Framework
• EvolMathEval: Towards Evolvable Benchmarks for Mathematical Reasoning via Evolutionary Testing
• Is GPT-OSS Good? A Comprehensive Evaluation of OpenAI's Latest Open Source Models
• An Investigation of Robustness of LLMs in Mathematical Reasoning: Benchmarking with Mathematically-Equivalent Transformation of Advanced Mathematical Problems
• A Survey on Non-Intrusive ASR Refinement: From Output-Level Correction to Full-Model Distillation
• Dynamic Benchmark Construction for Evaluating Large Language Models on Real-World Codes
• FinMMR: Make Financial Numerical Reasoning More Multimodal, Comprehensive, and Challenging
• Putnam-AXIOM: A Functional and Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs
• LLM-Crowdsourced: A Benchmark-Free Paradigm for Mutual Evaluation of Large Language Models
• MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks
• Pretraining on the Test Set Is No Longer All You Need: A Debate-Driven Approach to QA Benchmarks
• DCR: Quantifying Data Contamination in LLMs Evaluation

Related

• Benchmark Data Contamination of Large Language Models: A Survey
• The Vulnerability of Language Model Benchmarks: Do They Accurately Reflect True LLM Performance?
• The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination
• Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs
• Evaluation data contamination in LLMs: how do we measure it and (when) does it matter?