AI Testing Should Account for Sophisticated Strategic Behaviour

2025/08/19 by Vojtech Kovarik, Vojtěch Kovařík, Eric Olav Chen +9 · 1 voice · 2 citations
Computer Science · Psychology · Social Sciences · #Adversarial Robustness in Machine Learning #Ethics and Social Impacts of AI #Human-Automation Interaction and Safety #cs.AI #cs.GT

paper · pdf · doi:10.48550/arxiv.2508.14927

openalex publication_date 2025/08/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This position paper argues for two claims regarding AI testing and evaluation. First, to remain informative about deployment behaviour, evaluations need account for the possibility that AI systems understand their circumstances and reason strategically. Second, game-theoretic analysis can inform evaluation design by formalising and scrutinising the reasoning in evaluation-based safety cases. Drawing on examples from existing AI systems, a review of relevant research, and formal strategic analysis of a stylised evaluation scenario, we present evidence for these claims and motivate several research directions.

Citations

• Evaluation Faking: Unveiling Observer Effects in Safety Evaluation of Frontier AI Systems
• Evaluating Frontier Models for Stealth and Situational Awareness
• Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation
• Demonstrating specification gaming in reasoning models
• Alignment faking in large language models
• AI and the End of an Era
• Mastering Board Games by External and Internal Planning with Language Models
• Towards evaluations-based safety cases for AI scheming
• Game Theory with Simulation in the Presence of Unpredictable Randomisation
• The Shutdown Problem: An AI Engineering Puzzle for Decision Theorists
• Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
• Thousands of AI Authors on the Future of AI
• AI Control: Improving Safety Despite Intentional Subversion
• Provably safe systems: the only path to controllable AGI
• Taken out of context: On measuring situational awareness in LLMs
• Game Theory with Simulation of Other Players
• Should the Timing of Inspections be Predictable?
• Poisoning Web-Scale Training Datasets is Practical
• Circumventing interpretability: How to defeat mind-readers
• Will we run out of data? Limits of LLM scaling based on human-generated data
• Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals
• Training language models to follow instructions with human feedback
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling
• Optimal Policies Tend to Seek Power
• Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model
• Fine-Tuning Language Models from Human Preferences
• Risks from Learned Optimization in Advanced Machine Learning Systems
• SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
• Approval-directed agency and the decision theory of Newcomb-like problems
• Categorizing Variants of Goodhart's Law
• The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities
• The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities
• Deep reinforcement learning from human preferences
• DeepStack: Expert-Level Artificial Intelligence in No-Limit Poker
• Implementation without commitment in moral hazard environments
• Evolution of digital organisms at high mutation rates leads to survival of the flattest
• Cooperation under the Security Dilemma

Cited by

• Shutdown Safety Valves for Advanced AI
• Scheming Ability in LLM-to-LLM Strategic Interactions

Discussions

• Excited to present our position paper in the 4:30pm poster session at @neuripsconf.bsky.social today! We discuss the problem that AI systems may suspect they're being tested and act differently as a r [bsky, 9 points, 0 comments]

Related

• Evaluating AI Evaluation: Perils and Prospects
• Declare and Justify: Explicit assumptions in AI evaluations are necessary for effective regulation
• Reasons to Doubt the Impact of AI Risk Evaluations
• AI Evaluation: past, present and future
• Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems