The Limits of Predicting Agents from Behaviour

2025/06/03 by Alexis Bellot, Bellot, Alexis, Jonathan G. Richens +3 · 1 citation
Computer Science · Social Sciences · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Ethics and Social Impacts of AI #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (stat.ML)

paper · pdf · doi:10.48550/arxiv.2506.02923

openalex publication_date 2025/06/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

As the complexity of AI systems and their interactions with the world increases, generating explanations for their behaviour is important for safely deploying AI. For agents, the most natural abstractions for predicting behaviour attribute beliefs, intentions and goals to the system. If an agent behaves as if it has a certain goal or belief, then we can make reasonable predictions about how it will behave in novel situations, including those where comprehensive safety evaluations are untenable. How well can we infer an agent's beliefs from their behaviour, and how reliably can these inferred beliefs predict the agent's behaviour in novel situations? We provide a precise answer to this question under the assumption that the agent's behaviour is guided by a world model. Our contribution is the derivation of novel bounds on the agent's behaviour in new (unseen) deployment environments, which represent a theoretical limit for predicting intentional agents from behavioural data alone. We discuss the implications of these results for several research areas including fairness and safety.

Citations

• Scientist AI Needs a Government: Structural Governance as the Missing Layer in Non-Agentic AI Safety
• Can a Bayesian Oracle Prevent Harm from an Agent?
• Does ChatGPT Have a Mind?
• Evaluating the World Model Implicit in a Generative Model
• Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems
• Robust agents learn causal world models
• Towards Bounding Causal Effects under Markov Equivalence
• Language Models Represent Space and Time
• Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations
• Talking About Large Language Models
• Talking about Large Language Models
• Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task
• A Causal Analysis of Harm
• Personalized Decision Making -- A Conceptual Introduction
• Counterfactual harm
• Invariance in Policy Optimisation and Partial Identifiability in Reward Learning
• Partial Counterfactual Identification from Observational and Experimental Data
• Causal Abstractions of Neural Networks
• Chess as a Testbed for Language Model State Tracking
• Agent Incentives: A Causal Perspective
• Risks from Learned Optimization in Advanced Machine Learning Systems
• Bounds on the conditional and average treatment effect with unobserved confounding factors
• Counterfactual Fairness
• Towards Resolving Unidentifiability in Inverse Reinforcement Learning
• Preference elicitation and inverse reinforcement learning
• Probabilities of Causation: Bounds and Identification
• The Construction of Utility Functions from Expenditure Data

Cited by

• A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents

Related

• AI Agent Behavioral Science
• Safety from Honesty in a Disinterested AI Predictor
• Causal Analysis of Agent Behavior for AI Safety
• Intent-aligned AI systems deplete human agency: the need for agency foundations research in AI safety
• Limits to Verification and Validation of Agentic Behavior