Responses to catastrophic AGI risk: a survey

2014/12/19 by Kaj Sotala, Roman V. Yampolskiy · 4 citations
Social Sciences · Physics and Astronomy · Computer Science · Mathematics · #Ethics and Social Impacts of AI #Space Science and Extraterrestrial Life #Explainable Artificial Intelligence (XAI) #Humanity #Scale (ratio) #Field (mathematics) #Risk analysis (engineering) #Computer science #Political science #Business #Geography #Law #Mathematics #Cartography

paper · doi:10.1088/0031-8949/90/1/018001

openalex publication_date 2014/12/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Many researchers have argued that humanity will create artificial general intelligence (AGI) within the next twenty to one hundred years. It has been suggested that AGI may pose a catastrophic risk to humanity. After summarizing the arguments for why AGI may pose such a risk, we survey the field’s proposed responses to AGI risk. We consider societal proposals, proposals for external constraints on AGI behaviors, and proposals for creating AGIs that are safe due to their internal design.

Citations

• Better Never to Have Been
• Speculations Concerning the First Ultraintelligent Machine
• The Perils of Cognitive Enhancement and the Urgent Imperative to Enhance the Moral Character of Humanity
• Model-based Utility Functions
• Human agency and responsible computing: Implications for computer system design
• Prolegomena to any future artificial moral agent
• AI Armageddon and the Three Laws of Robotics
• Construal-level theory of psychological distance.
• A note on the confinement problem
• Defeating Dr. Evil with Self‐Locating Belief
• The conscious access hypothesis: origins and recent evidence
• Artificial Intelligence as a positive and negative factor in global risk
• Validity of the single processor approach to achieving large scale computing capabilities
• Implications and consequences of robots with biological brains
• Intelligent Machinery, A Heretical Theory*
• What Might Cognition Be, If Not Computation?
• Freedom of the Will and the Concept of a Person
• Toward the Human–Robot Co-Existence Society: On Safety Intelligence for Next Generation Robots
• Protecting the Endangered Human: Toward an International Treaty Prohibiting Cloning and Inheritable Alterations
• Moral Enhancement
• Singularity Hypotheses

Cited by

• The Key Concepts of Ethics of Artificial Intelligence
• A model of pathways to artificial superintelligence catastrophe for risk and decision analysis
• Modelos Dinâmicos Aplicados à Aprendizagem de Valores em Inteligência Artificial
• Artificial general intelligence [wikipedia]

Related

• Artificial General Intelligence, Existential Risk, and Human Risk Perception
• Artificial Intelligence: Arguments for Catastrophic Risk
• AGI Safety Literature Review
• Preparing for the Intelligence Explosion
• Safety Features for a Centralised AGI Project