Scientist AI Needs a Government: Structural Governance as the Missing Layer in Non-Agentic AI Safety

2025/02/21 by Bengio, Yoshua, Michael K. Cohen, Damiano Fornasiere +22 · 19 citations
Physics and Astronomy · #Space Science and Extraterrestrial Life

paper · pdf · doi:10.48550/arxiv.2502.15657

Abstract

Bengio et al. (2025) propose Scientist AI — a non-agentic, Bayesian AI architecture — as a safer alternative to autonomous AI agents. This paper identifies structural governance — governance embedded in system architecture rather than imposed by external institutions — as the missing layer. Drawing on the Neural Knowledge Base (NKB), a production knowledge graph with 1.27 million nodes and a functioning citizenship architecture, we propose a three-layer model for AI safety: honest computation (Scientist AI), structural governance (citizenship architecture), and institutional regulation. No single layer is sufficient.

Cited by

• The Cartesian Cut in Agentic AI
• STAMP/STPA informed characterization of Factors Leading to Loss of Control in AI Systems
• Password-Activated Shutdown Protocols for Misaligned Frontier Agents
• Toward a Safe Internet of Agents
• Toward Honest Language Models for Deductive Reasoning
• Not Everything That Counts Can Be Counted: A Case for Safe Qualitative AI
• A Criminology of Machines
• Natural Building Blocks for Structured World Models: Theory, Evidence, and Scaling
• The Oversight Game: Learning to Cooperatively Balance an AI Agent's Safety and Autonomy
• Shutdown Safety Valves for Advanced AI
• Safety from Honesty in a Disinterested AI Predictor
• AI Alignment Strategies from a Risk Perspective: Independent Safety Mechanisms or Shared Failures?
• Regulating the Agency of LLM-based Agents
• A dataset of rated conceptual arguments
• OpenFake: An Open Dataset and Platform Toward Real-World Deepfake Detection
• Dimensional Characterization and Pathway Modeling for Catastrophic AI Risks
• Bench-2-CoP: Can We Trust Benchmarking for EU AI Compliance?
• How Far Are AI Scientists from Changing the World?
• Model-Based Soft Maximization of Suitable Metrics of Long-Term Human Power

Related

• Risks of AI Scientists: Prioritizing Safeguarding Over Autonomy
• An Overview of Catastrophic AI Risks
• Arguments about Highly Reliable Agent Designs as a Useful Path to Artificial Intelligence Safety
• The Decision Path to Control AI Risks Completely: Fundamental Control Mechanisms for AI Governance
• Benefits and Risks of Using AI Agents in Research