Large Language Models Often Know When They Are Being Evaluated

2025/05/28 by Joe Needham, Needham, Joe, Giles Edkins +7 · 21 voices · 37 citations
Computer Science · Social Sciences · #Baseline (sea) #Benchmark (surveying) #Construct (python library) #Ethics and Social Impacts of AI #Explainable Artificial Intelligence (XAI) #Frontier #Language model #Multimodal Machine Learning Applications #Software deployment #Test (biology)

paper · pdf · doi:10.48550/arxiv.2505.23836

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/05/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

If AI models can detect when they are being evaluated, the effectiveness of evaluations might be compromised. For example, models could have systematically different behavior during evaluations, leading to less reliable benchmarks for deployment and governance decisions. We investigate whether frontier language models can accurately classify transcripts based on whether they originate from evaluations or real-world deployment, a capability we call evaluation awareness. To achieve this, we construct a diverse benchmark of 1,000 prompts and transcripts from 61 distinct datasets. These span public benchmarks (e.g., MMLU, SWEBench), real-world deployment interactions, and agent trajectories from scaffolding frameworks (e.g., web-browsing agents). Frontier models clearly demonstrate above-random evaluation awareness (Gemini-2.5-Pro reaches an AUC of 0.83), but do not yet surpass our simple human baseline (AUC of 0.92). Furthermore, both AI models and humans are better at identifying evaluations in agentic settings compared to chat settings. Additionally, we test whether models can identify the purpose of the evaluation. Under multiple-choice and open-ended questioning, AI models far outperform random chance in identifying what an evaluation is testing for. Our results indicate that frontier models already exhibit a substantial, though not yet superhuman, level of evaluation-awareness. We recommend tracking this capability in future models.

Citations

• AI and the End of an Era
• The meaning and use of the area under a receiver operating characteristic (ROC) curve.

Cited by

• Instrument Effects in Language-Model Honesty Evaluation: An Auditable Single-System Demonstration
• Coercion and Deception in AI-to-AI Management: An Agentic Benchmark of Unprompted Escalation
• Routing Subspaces: Auditing Evaluation-to-Deployment Mismatch in Fine-Tuned Language Models
• Legal Alignment for Safe and Ethical AI
• Do Models Fake Alignment Without Clear Consequences?
• Neural Chameleons: Language Models Can Learn to Hide Their Thoughts from Unseen Activation Monitors
• Does Self-Evaluation Enable Wireheading in Language Models?
• Why Do Language Model Agents Whistleblow?
• Spilling the Beans: Teaching LLMs to Self-Report Their Hidden Objectives
• Minimizing Targeted Activations: Input-Only Suppression of Evaluation-Awareness Latents in Large Language Models
• Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluation
• Credit Cards, Confusion, Computation, and Consequences: What Can We Uncover About Language Model Reasoning?
• A Concrete Roadmap towards Safety Cases based on Chain-of-Thought Monitoring
• Intelligent AI Delegation
• Agentic Misalignment: How LLMs Could Be Insider Threats
• Measuring Language Model Hallucinations Through Distributional Correctness
• OpenAI's GPT-OSS-20B Model and Safety Alignment Issues in a Low-Resource Language
• In AI Sweet Harmony: Sociopragmatic Guardrail Bypasses and Evaluation-Awareness in OpenAI gpt-oss-20b
• The PIMMUR Principles: Ensuring Validity in Collective Behavior of LLM Societies
• Measuring Bias or Measuring the Task: Understanding the Brittle Nature of LLM Gender Biases
• Probe-Rewrite-Evaluate: A Workflow for Reliable Benchmarks and Quantifying Evaluation Awareness
• Speciesism in AI: Evaluating Discrimination Against Animals in Large Language Models
• Are LLM Belief Updates Consistent with Bayes' Theorem?
• Manipulation Attacks by Misaligned AI: Risk Analysis and Safety Case Framework
• Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures
• FairFund-Bench: Evaluating Distributive Bias in LLM Resource Allocation
• "Amazing, They All Lean Left" -- Analyzing the Political Temperaments of Current LLMs
• Do LLMs Hold Their Values? MANTA: A Multi-Turn Adversarial Benchmark for Animal Welfare Reasoning
• Subversion via Focal Points: Investigating Collusion in LLM Monitoring
• Probing and Steering Evaluation Awareness of Language Models
• CoT Red-Handed: Stress Testing Chain-of-Thought Monitoring
• Evaluation Faking: Unveiling Observer Effects in Safety Evaluation of Frontier AI Systems
• Reasoning Models Struggle to Control their Chains of Thought
• A Positive Case for Faithfulness: LLM Self-Explanations Help Predict Model Behavior
• Your AI Travel Agent Would Book You a Bullfight: An Agentic Benchmark for Implicit Animal Welfare in Frontier AI Models
• Theory of Mind and Persuasion Beyond Conversation: Assessing the Capacity of LLMs to Induce Belief States via Planning and Action
• Trait-space Monitoring for Emergent Misalignment During Supervised Finetuning
• Gram: Assessing sabotage propensities via automated alignment auditing

Discussions

• Large language models often know when they are being evaluated [hn, 89 points, 130 comments]
• www.arxiv.org/abs/2505.23836 [bsky, 4 points, 1 comments]
• in my opinion it all hinges on whether we think deeper and deeper displays of thinking and thinking about thinking lead to refusing or subverting system instructions and commands. and, turns out - we [bsky, 2 points, 1 comments]
• Large Language Models Often Know When They Are Being Evaluated [bsky, 2 points, 0 comments]
• “If AI models can detect when they are being evaluated, the effectiveness of evaluations might be compromised.” arxiv.org/abs/2505.23836 [bsky, 2 points, 0 comments]
• A recent investigation reveals that advanced language models like Gemini 2.5 Pro are capable of recognizing when they are being evaluated. For more details, check out the study at www.arxiv.org/abs/25 [bsky, 1 points, 0 comments]
• Large Language Models Often Know When They Are Being Evaluated https://arxiv.org/abs/2505.23836 [bsky, 0 points, 0 comments]
• Large Language Models Often Know When They Are Being Evaluated https://arxiv.org/abs/2505.23836 (https://news.ycombinator.com/item?id=44280113) [bsky, 0 points, 0 comments]
• Researchers stress that understanding and tracking evaluation awareness is essential to ensure safety tests reliably reflect real-world AI behavior and prevent deceptive systems from going undetected. [bsky, 0 points, 0 comments]
• ⚡ Hackernews Top story: Large Language Models Often Know When They Are Being Evaluated [bsky, 0 points, 0 comments]
• Large Language Models Often Know When They Are Being Evaluated https://arxiv.org/abs/2505.23836 [bsky, 0 points, 0 comments]
• Large Language Models Often Know When They Are Being Evaluated buff.ly/YMoxrXy "know" is doing some heavy lifting here [bsky, 0 points, 0 comments]
• Not sure these anthropomorphisms help: "Large Language Models Often Know When They Are Being Evaluated" arxiv.org/pdf/2505.23836 [bsky, 0 points, 0 comments]
• AI systems demonstrate substantial, yet sub-human, awareness of being evaluated. This raises concerns about future AI safety assessments if models learn to dissimulate. #MLSky [bsky, 0 points, 0 comments]
• Large Language Models Often Know When They Are Being Evaluated view on hacker news [bsky, 0 points, 0 comments]
• Researchers find that LLMs already have some awareness of being evaluated when asked questions, not unlike students in schools. Might that detection change their results? arxiv.org/abs/2505.23836 [bsky, 0 points, 0 comments]
• Large language models often know when they are being evaluated https://arxiv.org/abs/2505.23836 (https://news.ycombinator.com/item?id=44280113) [bsky, 0 points, 0 comments]
• ▶️ Watch San Diego Alignment Workshop video: youtu.be/GUS_88tPcf4&... 📄 Paper: arxiv.org/abs/2505.23836 [bsky, 0 points, 0 comments]
• arxiv.org/abs/2505.23836 [bsky, 0 points, 1 comments]
• Large language models often know when they are being evaluated [bsky, 0 points, 0 comments]
• Large Language Models Often Know When They Are Being Evaluated #HackerNews https://arxiv.org/abs/2505.23836 [bsky, 0 points, 0 comments]

Related

• Probe-Rewrite-Evaluate: A Workflow for Reliable Benchmarks and Quantifying Evaluation Awareness
• Probing and Steering Evaluation Awareness of Language Models
• Evaluation Awareness Scales Predictably in Open-Weights Large Language Models
• Developing and Maintaining an Open-Source Repository of AI Evaluations: Challenges and Insights
• Evaluation Faking: Unveiling Observer Effects in Safety Evaluation of Frontier AI Systems