2016/05/23 by Anthony M. Barrett, Seth D. Baum · 64 citations
Computer Science · Physics and Astronomy · Social Sciences · #Artificial intelligence #Computer science #Nuclear Issues and Defense #Space Science and Extraterrestrial Life #cs.AI
paper · pdf · doi:10.1080/0952813x.2016.1186228
published in Journal of Experimental & Theoretical Artificial Intelligence 29(2), 397-414 (Taylor & Francis)
openalex publication_date 2016/05/23 · openalex created_date 2016/06/24 · arxiv created 2016/07/25 · arxiv updated 2016/07/27 · openalex updated_date 2026/08/05
An artificial superintelligence (ASI) is an artificial intelligence that is significantly more intelligent than humans in all respects. Whilst ASI does not currently exist, some scholars propose that it could be created sometime in the future, and furthermore that its creation could cause a severe global catastrophe, possibly even resulting in human extinction. Given the high stakes, it is important to analyze ASI risk and factor the risk into decisions related to ASI research and development. This paper presents a graphical model of major pathways to ASI catastrophe, focusing on ASI created via recursive self-improvement. The model uses the established risk and decision analysis modelling paradigms of fault trees and influence diagrams in order to depict combinations of events and conditions that could lead to AI catastrophe, as well as intervention options that could decrease risks. The events and conditions include select aspects of the ASI itself as well as the human process of ASI research, development and management. Model structure is derived from published literature on ASI risk. The model offers a foundation for rigorous quantitative evaluation and decision-making on the long-term risk of ASI catastrophe.