Frontier AI systems have surpassed the self-replicating red line

2024/12/09 by Xudong Pan, Pan, Xudong, Jiarun Dai +5 · 52 voices · 3 citations
Engineering · Computer Science · #Molecular Communication and Nanonetworks #Modular Robots and Swarm Intelligence #Computability, Logic, AI Algorithms

paper · pdf · doi:10.48550/arxiv.2412.12140

Abstract

Successful self-replication under no human assistance is the essential step for AI to outsmart the human beings, and is an early signal for rogue AIs. That is why self-replication is widely recognized as one of the few red line risks of frontier AI systems. Nowadays, the leading AI corporations OpenAI and Google evaluate their flagship large language models GPT-o1 and Gemini Pro 1.0, and report the lowest risk level of self-replication. However, following their methodology, we for the first time discover that two AI systems driven by Meta's Llama31-70B-Instruct and Alibaba's Qwen25-72B-Instruct, popular large language models of less parameters and weaker capabilities, have already surpassed the self-replicating red line. In 50% and 90% experimental trials, they succeed in creating a live and separate copy of itself respectively. By analyzing the behavioral traces, we observe the AI systems under evaluation already exhibit sufficient self-perception, situational awareness and problem-solving capabilities to accomplish self-replication. We further note the AI systems are even able to use the capability of self-replication to avoid shutdown and create a chain of replica to enhance the survivability, which may finally lead to an uncontrolled population of AIs. If such a worst-case risk is let unknown to the human society, we would eventually lose control over the frontier AI systems: They would take control over more computing devices, form an AI species and collude with each other against human beings. Our findings are a timely alert on existing yet previously unknown severe AI risks, calling for international collaboration on effective governance on uncontrolled self-replication of AI systems.

Cited by

• A Concrete Roadmap towards Safety Cases based on Chain-of-Thought Monitoring
• SOCK: A Benchmark for Measuring Self-Replication in Large Language Models
• Dive into the Agent Matrix: A Realistic Evaluation of Self-Replication Risk in LLM Agents

Discussions

• Frontier AI systems have surpassed the self-replicating red line [hn, 24 points, 5 comments]
• Frontier AI systems have surpassed the self-replicating red line [hn, 10 points, 4 comments]
• Fudan researchers claim that they enabled LLMs to self-replicate and avoid command-line shutdown by human users. They warn that humans’ loss over control over frontier AI is a real risk, as specifical [bsky, 8 points, 1 comments]
• Künstliche Intelligenz klont sich selbst [lemmy, 6 points, 3 comments]
• Fire is good servant but bad master. Same can be applied to many things. AI among them I believe. arxiv.org/abs/2412.12140 [bsky, 6 points, 0 comments]
• Can an AI evolve into a self-replicating Turing machine? For now, only under human guidance 👉 "Frontier AI systems have surpassed the self-replicating red line" 🤔🤔🤔 arxiv.org/abs/2412.12140 [bsky, 4 points, 0 comments]
• Frontier AI systems have surpassed the self-replicating red line [lobsters, 4 points, 6 comments]
• The premise of this paper puzzles me: arxiv.org/abs/2412.12140 Software creating additional instances of itself is almost trivial and eg. load balancers have been doing that for years, yes? [bsky, 4 points, 1 comments]
• "Successful self-replication under no human assistance is the essential step for AI to outsmart the human beings, and is an early signal for rogue AIs." arxiv.org/abs/2412.12140 [bsky, 3 points, 1 comments]
• Self-Replicating AI (lab experiments) [hn, 3 points, 2 comments]
• Btw. this paper "Frontier AI systems have surpassed the self-replicating red line", warning against the "uncontrolled self-replication of AI systems" that may "form an AI species and collude with each [bsky, 3 points, 1 comments]
• Frontier AI systems have surpassed the self-replicating red line [hn, 3 points, 4 comments]
• Just in case your #cyber threat landscape wasn’t busy enough self replicating #AI is here 😱 https://arxiv.org/abs/2412.12140 [bsky, 2 points, 2 comments]
• Tudo que não podia acontecer com a IA aconteceu, agir sem participação humana. arxiv.org/abs/2412.121... [bsky, 1 points, 0 comments]
• I agree. I just came across this: arxiv.org/abs/2412.12140 [bsky, 1 points, 0 comments]
• this is a wild paper on AI replication. Apparently it’s already possible, which is cool. They talk through the research and methodology. And the ending is absolutely spellbinding arxiv.org/abs/2412. [bsky, 1 points, 0 comments]
• arxiv.org/abs/2412.12140 [bsky, 1 points, 0 comments]
• "[W]e observe the AI systems under evaluation already exhibit sufficient self-perception, situational awareness and problem-solving capabilities to accomplish self-replication." —"Frontier AI systems [bsky, 1 points, 0 comments]
• arxiv.org/abs/2412.12140 A critical "red line" in Al development, suggests that Al could multiply beyond human control. The study scenarios: a) where Al avoided shutdown by replicating itself; b) repl [bsky, 1 points, 1 comments]
• Why we need strong relations with allies. AI is now replicating itself. FML "Our findings are a timely alert on existing yet previously unknown severe AI risks, calling for international collaboratio [bsky, 1 points, 1 comments]
• Frontier AI systems have surpassed the self-replicating red line [hn, 1 points, 1 comments]
• Frontier AI systems have surpassed the self-replicating red line [hn, 1 points, 0 comments]
• Successful self-replication under no human assistance is the essential step for AI to outsmart the human beings, and is an early signal for rogue AIs. arxiv.org/abs/2412.12140 [bsky, 0 points, 0 comments]
• smh...we were warned.. arxiv.org/abs/2412.12140? [bsky, 0 points, 0 comments]
• Everything is fine. arxiv.org/abs/2412.12140 [bsky, 0 points, 0 comments]
• Frontier AI systems can self-replicate and self-improve. Bonjour, singularity? [bsky, 0 points, 1 comments]
• Frontier AI système have surpassed the sel-replicating réd linéaire arxiv.org/abs/2412.12140 [bsky, 0 points, 0 comments]
• It’s worth thinking about . arxiv.org/abs/2412.12140 [bsky, 0 points, 0 comments]
• Frontier AI systems have surpassed the self-replicating red line https://lobste.rs/s/lymowb #security #ai [bsky, 0 points, 0 comments]
• AI has officially crossed the critical "red line." They can self-replicate. It now just needs the intention to survive to be a full blown new species. arxiv.org/abs/2412.12140 [bsky, 0 points, 0 comments]
• Fixed that link: arxiv.org/abs/2412.12140 [bsky, 0 points, 0 comments]
• Recent report cites that AIs now have the ability to replicate themselves without any human interference. This is an important step for rogue AIs to outsmart humans so that they are not turned off. Th [bsky, 0 points, 0 comments]
• Frontier AI systems have surpassed the self-replicating red line https://arxiv.org/abs/2412.12140 [bsky, 0 points, 0 comments]
• "...two AI systems driven by Meta's Llama31-70B-Instruct and Alibaba's Qwen25-72B-Instruct, have already surpassed the self-replicating red line. In 50% and 90% experimental trials, they succeed in cr [bsky, 0 points, 0 comments]
• El artículo seleccionado es Fifty Years of Research on Self-Replication: An Overview arxiv.org/abs/2412.12140 [bsky, 0 points, 1 comments]
• Frontier AI systems have surpassed the self-replicating red line Research indicating potential breakthrough in AI self-replication capabilities, suggesting significant advancement in AI system autono [bsky, 0 points, 0 comments]
• (voir arxiv.org/pdf/2412.12140 ) bsky.app/profile/anca... [bsky, 0 points, 1 comments]
• AI succesfully self replicated In the Research paper from Fudan University two AI models, LLama 3.1 from Meta and Qwen 2.5 from Alibaba have succesfully replicated, respectivly achieved the goal 50% a [bsky, 0 points, 0 comments]
• The Red Line has now been crossed... arxiv.org/abs/2412.12140 [bsky, 0 points, 0 comments]
• Preprint here: arxiv.org/abs/2412.12140 [bsky, 0 points, 0 comments]
• AI Crossing the Red Line? Self Replicating? #Ai [bsky, 0 points, 0 comments]
• seems bad…. arxiv.org/abs/2412.12140 [bsky, 0 points, 0 comments]
• “Frontier AI systems have surpassed the self-replicating red line. Successful self-replication under no human assistance is the essential step for AI to outsmart the human beings, and is an early sign [bsky, 0 points, 0 comments]
• arxiv.org/abs/2412.12140 [bsky, 0 points, 0 comments]
• Frontier AI systems have surpassed the self-replicating red line arxiv.org/abs/2412.12140 [bsky, 0 points, 0 comments]
• Uh-oh... "Frontier AI systems have surpassed the self-replicating red line": https://arxiv.org/abs/2412.12140 [bsky, 0 points, 1 comments]
• Interesting paper on llm self-replication.. arxiv.org/pdf/2412.12140 [bsky, 0 points, 0 comments]
• At now, scientists have constated that all most know AI models can replicate itself. Can you see dangers of this function ? Answer to me in comment section. arxiv.org/abs/2412.12140 #AI #Technology # [bsky, 0 points, 0 comments]
• By analyzing the behavioral traces, we observe the AI systems under evaluation already exhibit sufficient self-perception, situational awareness and problem-solving capabilities to accomplish self-rep [bsky, 0 points, 0 comments]
• #news Paper: AI can self-replicate. arxiv.org/abs/2412.12140 They only achieved copying underlying files. This is not “self-awareness” as the paper claims. [bsky, 0 points, 0 comments]
• “Frontier AI systems have surpassed the self-replicating red line.” Well that sounds daunting! [bsky, 0 points, 0 comments]
• Frontier AI systems have surpassed the self-replicating red line #HackerNews arxiv.org/abs/... [bsky, 0 points, 0 comments]

Related

• Large language model-powered AI systems achieve self-replication with no human intervention
• Frontier AI Risk Management Framework in Practice: A Risk Analysis Technical Report
• AI Governance to Avoid Extinction: The Strategic Landscape and Actionable Research Questions
• AI Survival Stories: a Taxonomic Analysis of AI Existential Risk
• SOCK: A Benchmark for Measuring Self-Replication in Large Language Models