Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety

2025/07/15 by Tomek Korbak, Korbak, Tomek, Mikita Balesni +81 · 25 voices · 90 citations
Computer Science · Mathematics · #Anomaly Detection Techniques and Applications #Causal chain #Chain (unit) #Frontier #Imperfect #Investment (military) #cs.AI #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2507.11473

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/07/15 · openalex created_date 2025/10/09 · openalex updated_date 2026/08/05

Abstract

AI systems that "think" in human language offer a unique opportunity for AI safety: we can monitor their chains of thought (CoT) for the intent to misbehave. Like all other known AI oversight methods, CoT monitoring is imperfect and allows some misbehavior to go unnoticed. Nevertheless, it shows promise and we recommend further research into CoT monitorability and investment in CoT monitoring alongside existing safety methods. Because CoT monitorability may be fragile, we recommend that frontier model developers consider the impact of development decisions on CoT monitorability.

Citations

• Latent Iteration as Renormalization: Inference-Time Recurrence in a Depth-Recurrent Transformer Flows Attention Geometry Toward the SYK Conformal Fixed Point
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Reasoning Beyond Words ? Exploring framework for hidden state reasoning
• AI and the End of an Era
• Chain of Thought Empowers Transformers to Solve Inherently Serial Problems
• Attention Is All You Need

Cited by

• Operational Proto-Introspection in Looped Language Models: Process-Quality Taps, Executable Branching, and the Readout-Control Boundary
• Length Penalties Make Chain-of-Thought Less Monitorable
• Reading Between the Dots: Decoding Hidden Computation across Filler Tokens
• Thinking Without Words: Efficient Latent Reasoning with Abstract Chain-of-Thought
• Legal Alignment for Safe and Ethical AI
• Distributional AGI Safety
• Training LLMs for Honesty via Confessions
• Humans Perceive Wrong Narratives from AI Reasoning Texts
• Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokens
• Position: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!
• Is Chain-of-Thought Really Not Explainability? Chain-of-Thought Can Be Faithful without Hint Verbalization
• Practical challenges of control monitoring in frontier AI deployments
• Not All LLM Reasoning is Visible in the Chain-of-Thought
• The Agony of Opacity: Foundations for Reflective Interpretability in AI-Mediated Mental Health Support
• Learning to Extract Context for Context-Aware LLM Inference
• Async Control: Stress-testing Asynchronous Control Measures for LLM Agents
• Can LLMs Understand What We Cannot Say? Measuring Multilevel Alignment Through Abortion Stigma Across Cognitive, Interpersonal, and Structural Levels
• Asynchronous Reasoning: Training-Free Interactive Thinking LLMs
• Chain-of-Image Generation: Toward Monitorable and Controllable Image Generation
• Training Language Models to Use Prolog as a Tool
• Reliable agent engineering should integrate machine-compatible organizational principles
• Natural Language Summarization Enables Multi-Repository Bug Localization by LLMs in Microservice Architectures
• Unsupervised decoding of encoded reasoning using language model interpretability
• Mitigating Indirect Prompt Injection via Instruction-Following Intent Analysis
• Steering Awareness: Detecting Activation Steering from Within
• Large Language Models' Complicit Responses to Illicit Instructions across Socio-Legal Contexts
• Do LLMs Really Struggle at NL-FOL Translation? Revealing their Strengths via a Novel Benchmarking Strategy
• Investigating CoT Monitorability in Large Reasoning Models
• Thought Branches: Interpreting LLM Reasoning Requires Resampling
• Measuring Chain-of-Thought Monitorability Through Faithfulness and Verbosity
• Reasoning Models Sometimes Output Illegible Chains of Thought
• Distributed Attacks in Persistent-State AI Control
• On the Use of Large Language Models for Qualitative Synthesis
• MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in Language Models, More than Outcomes
• A Pragmatic Way to Measure Chain-of-Thought Monitorability
• Mapping Faithful Reasoning in Language Models
• A Concrete Roadmap towards Safety Cases based on Chain-of-Thought Monitoring
• Can Reasoning Models Obfuscate Reasoning? Stress-Testing Chain-of-Thought Monitorability
• The Ends Justify the Thoughts: RL-Induced Motivated Reasoning in LLM CoTs
• AI Alignment Strategies from a Risk Perspective: Independent Safety Mechanisms or Shared Failures?
• RECON: Reasoning with Condensation for Efficient Retrieval-Augmented Generation
• Output Supervision Can Obfuscate the Chain of Thought
• Tool Use Enables Undetectable Steganography in Multi-Agent LLM Systems
• All Code, No Thought: Current Language Models Struggle to Reason in Ciphered Language
• Verifying Chain-of-Thought Reasoning via Its Computational Graph
• Off-Trajectory Reasoning: Can LLMs Collaborate on Reasoning Trajectory?
• From surveillance to signalling: escalation channels as environmental controls for agentic AI
• Typed Chain-of-Thought: A Curry-Howard Framework for Verifying LLM Reasoning
• Towards Safe Reasoning in Large Reasoning Models via Corrective Intervention
• SafeSearch: Automated Red-Teaming for the Safety of LLM-Based Search Agents
• Scaling Policy Compliance Assessment in Language Models with Policy Reasoning Traces
• Self-Blinding and Counterfactual Self-Simulation Mitigate Biases and Sycophancy in Large Language Models
• Talking Trees: Reasoning-Assisted Induction of Decision Trees for Tabular Data
• (When) Should We Delegate AI Governance to AIs? Some Lessons from Administrative Law
• Conversable Complexity: Agentic LLM Collectives as Interpretable Substrates
• Beyond the Commitment Boundary: Probing Epiphenomenal Chain-of-Thought in Large Reasoning Models
• CORE: Contrastive Reflection Enables Rapid Improvements in Reasoning
• Towards mitigating information leakage when evaluating safety monitors
• Reasoning Language Model for Personalized Lung Cancer Screening
• AI reasoning effort predicts human decision time in content moderation
• Reliable Weak-to-Strong Monitoring of LLM Agents
• Long Chain-of-Thought Reasoning Across Languages
• Lexical Hints of Accuracy in LLM Reasoning Chains
• Retrieval-augmented reasoning with lean language models
• AURA: Affordance-Understanding and Risk-aware Alignment Technique for Large Language Models
• Persistent Instability in LLM's Personality Measurements: Effects of Scale, Reasoning, and Conversation History
• Cognitive Loop via In-Situ Optimization: Self-Adaptive Reasoning for Science
• LLMs Can Covertly Sandbag on Capability Evaluations Against Chain-of-Thought Monitoring
• Outcome Rewards Do Not Guarantee Verifiable or Causally Important Reasoning
• Shaping capabilities with token-level data filtering
• Towards Evaluting Fake Reasoning Bias in Language Models
• Critique of impure reason: Unveiling the reasoning behaviour of medical large language models
• Benchmarking Deception Probes via Black-to-White Performance Boosts
• On the Generalization of Steering Vectors for Chain-of-Thought Faithfulness
• Where Do Reasoning Models Refuse?
• Thought Anchors: Which LLM Reasoning Steps Matter?
• Latent Concept Disentanglement in Transformer-based Language Models
• Case-Based Reasoning Enhances the Predictive Power of LLMs in Drug-Drug Interaction
• ReasoningShield: Safety Detection over Reasoning Traces of Large Reasoning Models
• A False Average: Chain-of-Thought Monitors Collapse Where They Are the Only Defense
• Reasoning Models Struggle to Control their Chains of Thought
• Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought
• Do explanations generalize across large reasoning models?
• Training Language Models to Explain Their Own Computations
• Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack
• Analyzing and Improving Chain-of-Thought Monitorability Through Information Theory
• Online Safety Monitoring for LLMs
• Introspective Coupling: Self-Explanation Training Tracks Behavioral Change Despite Fixed Supervision
• Do Thinking Tokens Help with Safety?
• How Transparent is DiffusionGemma?
• From Sycophantic Consensus to Pluralistic Repair: Why AI Alignment Must Surface Disagreement
• How Vulnerable Are AI Agents to Indirect Prompt Injections? Insights from a Large-Scale Public Competition
• Who Controls the Conversation? User Perspectives On Generative AI (LLM) System Prompts
• VERITAS: A Multi-Agent Co-Scientist for Verifiable Image-Derived Hypothesis Testing
• LatentGuard: Efficient and Inspectable Latent Reasoning for LLM Safeguards
• The Tell-Tale Trace: Detecting Reasoning Failures in LLMs Using Chain-of-Thought Dynamics
• Chain-of-Thought Monitoring Can Be Unreliable in Implicit-Influence Settings
• Position: It's Time to Optimize LLMs for Self-Consistency

Discussions

• Chain of thought monitorability: A new and fragile opportunity for AI safety [hn, 134 points, 64 comments]
• OpenAI backs a new cross-organizational paper emphasizing the promise of Chain of Thought (CoT) monitoring agents are generally black boxes, but the one way we’ve identified to monitor them is their C [bsky, 8 points, 0 comments]
• Indeed, it would have been much less safe if they hadn't gotten the transparency right first. arxiv.org/abs/2507.11473 [bsky, 7 points, 0 comments]
• 요즘 이런 생각을 종종합니다. "윤리는 안보의 문제다. 철학은 생존의 문제다." 어제 이런 것들을 읽었습니다. 인상적이었어요. "사고의 사슬 모니터링: AI 안전을 위한 새롭고 취약한 기회" arxiv.org/abs/2507.114... "사고의 사슬이 필요할 때, 언어 모델은 평가자를 회피하기 위해 고군분투한다." arxiv.org/abs/2507.052 [bsky, 3 points, 0 comments]
• Qualitatively, we also expect it to be harder to rule out that models are manipulating evaluations as situational awareness and strategic reasoning capabilities improve. This will be especially concer [bsky, 3 points, 1 comments]
• But more RL fine-tuning, scaling or a jump to 𝗹𝗮𝘁𝗲𝗻𝘁 𝗿𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 (all math, no words) could slam that window shut. Treat CoT legibility as a perishable safety asset ( the godfather of AI [bsky, 2 points, 1 comments]
• Recent works have stressed importance of monitoring CoTs arxiv.org/abs/2507.11473 & anthropic.com/research/tra... (@anthropic.com). Erasing information from parameters makes FUR a very precise tool fo [bsky, 2 points, 1 comments]
• [2507.11473] Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety arxiv.org/abs/2507.11473 [bsky, 1 points, 0 comments]
• Checking out this paper: Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety. arxiv.org/pdf/2507.11473 [bsky, 1 points, 0 comments]
• Chain of thought monitorability: A new and fragile opportunity for AI safety https://arxiv.org/abs/2507.11473 https://news.ycombinator.com/item?id=44582855 [bsky, 0 points, 0 comments]
• Chain of thought monitorability: A new and fragile opportunity for AI safety https://arxiv.org/abs/2507.11473 [bsky, 0 points, 0 comments]
• Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety. arxiv.org/abs/2507.11473 [bsky, 0 points, 0 comments]
• Chain of Thought Monitorability: A New and Fragile Opportunity for Al Safety arxiv.org/abs/2507.11473 arxiviq.substack.com/p/chain-of-t... [bsky, 0 points, 0 comments]
• Read the paper here: arxiv.org/abs/2507.11473 [bsky, 0 points, 0 comments]
• Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety All monitoring and oversight methods have limitations that allow some misbehavior to go unnoticed. Thus, safety measures fo [bsky, 0 points, 0 comments]
• https://arxiv.org/abs/2507.11473 この論文では、Chain of Thought (CoT) のモニタリング可能性について議論しています。 AIの安全性を向上させるための新たな機会として捉えられています。 CoTの脆弱性についても指摘されています。 [bsky, 0 points, 0 comments]
• Chain of thought monitorability: A new and fragile opportunity for AI safety #HackerNews https://arxiv.org/abs/2507.11473 [bsky, 0 points, 0 comments]
• The UK AI safety institute released an interesting paper proposing monitoring of chain-of-thoughts where LLMs are used in particularly critical or sensitive topics, as this can allow for stronger guar [bsky, 0 points, 0 comments]
• Chain of thought monitorability: A new and fragile opportunity for AI safety https://arxiv.org/abs/2507.11473 (https://news.ycombinator.com/item?id=44582855) [bsky, 0 points, 0 comments]
• Chain of thought monitorability: A new and fragile opportunity for AI safety https://arxiv.org/abs/2507.11473 (https://news.ycombinator.com/item?id=44582855) [bsky, 0 points, 0 comments]
• https://bsky.app/profile/buzzing.cc.web.brid.gy/post/3lu4szhxqmkh2 [bsky, 0 points, 0 comments]
• Current #AI reasoning models often explicitly state phrases like “Let’s hack,” or “Let’s sabotage,” in their internal "chain-of-thoughts". 40+ Researchers from OpenAI, Google DeepMind, Anthropic, and [bsky, 0 points, 0 comments]
• ▶️ Watch San Diego Alignment Workshop video: youtu.be/wa1XIJ6NmiA&... 📄 Read paper: arxiv.org/abs/2507.11473 [bsky, 0 points, 0 comments]
• Chain of thought monitorability: A new and fragile opportunity for AI safety [bsky, 0 points, 0 comments]
• #AI systems that reason in natural language can expose their intentions, making Chain-of-Thought (#CoT) monitoring a promising safety lever 🔗 arxiv.org/abs/2507.11473 #ResponsibleAI [bsky, 0 points, 0 comments]

Related

• When Chain of Thought is Necessary, Language Models Struggle to Evade Monitors
• A Pragmatic Way to Measure Chain-of-Thought Monitorability
• A Concrete Roadmap towards Safety Cases based on Chain-of-Thought Monitoring
• CoT Red-Handed: Stress Testing Chain-of-Thought Monitoring
• Monitoring Monitorability