2024/02/06 by Xiangru Tang, Tang, Xiangru, Qiao Jin +24 · 2 voices · 26 citations
Computer Science · Engineering · Medicine · #Artificial Intelligence (cs.AI) #Autonomy #Biomedical Ethics and Regulation #Business #Computation and Language (cs.CL) #Computers and Society (cs.CY) #Engineering #Engineering ethics #FOS: Computer and information sciences #Law #Law, AI, and Intellectual Property #Machine Learning (cs.LG) #Medicine #Political science #Risk analysis (engineering) #Safeguarding #cs.AI #cs.CL #cs.CY #cs.LG
paper · pdf · doi:10.48550/arxiv.2402.04247
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2024/02/06 · arxiv published 2024/02/06 · openalex created_date 2024/02/09 · arxiv updated 2025/07/21 · openalex updated_date 2026/07/28
AI scientists powered by large language models have demonstrated substantial promise in autonomously conducting experiments and facilitating scientific discoveries across various disciplines. While their capabilities are promising, these agents also introduce novel vulnerabilities that require careful consideration for safety. However, there has been limited comprehensive exploration of these vulnerabilities. This perspective examines vulnerabilities in AI scientists, shedding light on potential risks associated with their misuse, and emphasizing the need for safety measures. We begin by providing an overview of the potential risks inherent to AI scientists, taking into account user intent, the specific scientific domain, and their potential impact on the external environment. Then, we explore the underlying causes of these vulnerabilities and provide a scoping review of the limited existing works. Based on our analysis, we propose a triadic framework involving human regulation, agent alignment, and an understanding of environmental feedback (agent regulation) to mitigate these identified risks. Furthermore, we highlight the limitations and challenges associated with safeguarding AI scientists and advocate for the development of improved models, robust benchmarks, and comprehensive regulations.