Interpretable Risk Mitigation in LLM Agent Systems

2025/05/15 by Jan Chojnacki, Chojnacki, Jan · 4 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computer Science and Game Theory (cs.GT) #Computers and Society (cs.CY) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2505.10670

openalex publication_date 2025/05/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Autonomous agents powered by large language models (LLMs) enable novel use cases in domains where responsible action is increasingly important. Yet the inherent unpredictability of LLMs raises safety concerns about agent reliability. In this work, we explore agent behaviour in a toy, game-theoretic environment based on a variation of the Iterated Prisoner's Dilemma. We introduce a strategy-modification method-independent of both the game and the prompt-by steering the residual stream with interpretable features extracted from a sparse autoencoder latent space. Steering with the good-faith negotiation feature lowers the average defection probability by 28 percentage points. We also identify feasible steering ranges for several open-source LLM agents. Finally, we hypothesise that game-theoretic evaluation of LLM agents, combined with representation-steering alignment, can generalise to real-world applications on end-user devices and embodied platforms.

Citations

• Moral Alignment for LLM Agents
• Integrated Design and Governance of Agentic AI Systems through Adaptive Information Modulation
• Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2
• Large Model Strategic Thinking, Small Model Efficiency: Transferring Theory of Mind in Large Language Models
• Nicer Than Humans: How do Large Language Models Behave in the Prisoner's Dilemma?
• Large Language Models for UAVs: Current State and Pathways to the Future
• Non-Linear Inference Time Intervention: Improving LLM Truthfulness
• Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception
• CogAgent: A Visual Language Model for GUI Agents
• GPQA: A Graduate-Level Google-Proof Q&A Benchmark
• Representation Engineering: A Top-Down Approach to AI Transparency
• You Only Look at Screens: Multimodal Chain-of-Action Agents
• Sparse Autoencoders Find Highly Interpretable Features in Language Models
• Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
• Playing repeated games with large language models
• Generative Agents: Interactive Simulacra of Human Behavior
• Toolformer: Language Models Can Teach Themselves to Use Tools
• Towards Reasoning in Large Language Models: A Survey
• RT-1: Robotics Transformer for Real-World Control at Scale
• Distributing Accountability, Not Capability: Phase Separation and the LLM Workflow Quadrant in Autonomous AI Agent Architectures
• Toy Models of Superposition
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
• Training language models to follow instructions with human feedback
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Finetuned Language Models Are Zero-Shot Learners
• On the Opportunities and Risks of Foundation Models
• Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing
• Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing
• Evaluating Large Language Models Trained on Code
• Measuring Mathematical Problem Solving With the MATH Dataset
• On the Dangers of Stochastic Parrots
• Measuring Massive Multitask Language Understanding
• Language Models are Few-Shot Learners
• A Primer in BERTology: What we know about how BERT works
• Artificial Intelligence, Values and Alignment
• Emergent Tool Use From Multi-Agent Autocurricula
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Concrete Problems in AI Safety
• The Mythos of Model Interpretability
• Research Priorities for Robust and Beneficial Artificial Intelligence

Cited by

• LLM Harms: A Taxonomy and Discussion
• A Call for Collaborative Intelligence: Why Human-Agent Systems Should Precede AI Autonomy
• The AI in the Mirror: LLM Self-Recognition in an Iterated Public Goods Game
• When Truthful Representations Flip Under Deceptive Instructions?

Related

• Understanding LLM Agent Behaviours via Game Theory: Strategy Recognition, Biases and Multi-Agent Dynamics
• Safety Game: Balancing Safe and Informative Conversations with Blackbox Agentic AI using LP Solvers
• LLMs as Policy-Agnostic Teammates: A Case Study in Human Proxy Design for Heterogeneous Agent Teams
• Do LLMs trust AI regulation? Emerging behaviour of game-theoretic LLM agents
• R-Judge: Benchmarking Safety Risk Awareness for LLM Agents