Provably safe systems: the only path to controllable AGI

2023/09/05 by Max Tegmark, Steve Omohundro, Tegmark, Max +1 · 2 voices · 17 citations
Computer Science · Engineering · Mathematics · #Advanced Malware Detection Techniques #Adversarial Robustness in Machine Learning #Artificial intelligence #Computability, Logic, AI Algorithms #Computer science #Engineering #Humanity #Interpretability #Law #Mathematical economics #Mathematics #Outcome (game theory) #Path (computing) #Political science #Programming language #Sociology #Theoretical computer science #Thriving #Work (physics) #cs.AI #cs.CY #cs.LG

paper · pdf · doi:10.48550/arxiv.2309.01933

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2023/09/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We describe a path to humanity safely thriving with powerful Artificial General Intelligences (AGIs) by building them to provably satisfy human-specified requirements. We argue that this will soon be technically feasible using advanced AI for formal verification and mechanistic interpretability. We further argue that it is the only path which guarantees safe controlled AGI. We end with a list of challenge problems whose solution would contribute to this positive outcome and invite readers to join in this work.

Cited by

• International Agreements to Limit Frontier AI: Objectives and Exit
• Distributional AGI Safety
• STAMP/STPA informed characterization of Factors Leading to Loss of Control in AI Systems
• Verifying Memoryless Sequential Decision-making of Large Language Models
• Logical Consistency Between Disagreeing Experts and Its Role in AI Safety
• ASGuard: Activation-Scaling Guard to Mitigate Targeted Jailbreaking Attack
• Limitations on Accurate, Trusted, Human-level Reasoning
• Bounded PCTL Model Checking of Large Language Model Outputs
• No-Knowledge Alarms for Misaligned LLMs-as-Judges
• AI Testing Should Account for Sophisticated Strategic Behaviour
• Lessons from complex systems science for AI governance
• Safety Features for a Centralised AGI Project
• Black-Box Access is Insufficient for Rigorous AI Audits
• Towards provable probabilistic safety for scalable embodied AI systems
• RedDebate: Safer Responses Through Multi-Agent Red Teaming Debates
• On the creation of narrow AI: hierarchy and nonlocality of neural network skills
• Owner-Harm: A Missing Threat Model for AI Agent Safety

Discussions

• Provably safe systems: the only path to controllable AGI [hn, 45 points, 64 comments]
• Provably safe systems: the only path to controllable AGI [hn, 2 points, 1 comments]

Related

• Limitations on Accurate, Trusted, Human-level Reasoning
• Towards AI-45^∘ Law: A Roadmap to Trustworthy AGI
• Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems
• Safe AI -- How is this Possible?
• Building Safer AGI by introducing Artificial Stupidity