Generative AI Misuse: A Taxonomy of Tactics and Insights from Real-World Data

2024/06/19 by Nahema Marchal, Marchal, Nahema, Rachel Xu +9 · 12 voices · 6 citations
Social Sciences · #Ethics and Social Impacts of AI

paper · pdf · doi:10.48550/arxiv.2406.13843

Abstract

Generative, multimodal artificial intelligence (GenAI) offers transformative potential across industries, but its misuse poses significant risks. Prior research has shed light on the potential of advanced AI systems to be exploited for malicious purposes. However, we still lack a concrete understanding of how GenAI models are specifically exploited or abused in practice, including the tactics employed to inflict harm. In this paper, we present a taxonomy of GenAI misuse tactics, informed by existing academic literature and a qualitative analysis of approximately 200 observed incidents of misuse reported between January 2023 and March 2024. Through this analysis, we illuminate key and novel patterns in misuse during this time period, including potential motivations, strategies, and how attackers leverage and abuse system capabilities across modalities (e.g. image, text, audio, video) in the wild.

Cited by

• A matter of principle? AI alignment as the fair treatment of claims
• Training-free Source Attribution of AI-generated Images via Resynthesis
• SafetyPairs: Isolating Safety Critical Image Features with Counterfactual Image Generation
• OpenFake: An Open Dataset and Platform Toward Real-World Deepfake Detection
• Towards Robust Red-Green Watermarking for Autoregressive Image Generators
• Can We End the Cat-and-Mouse Game? Simulating Self-Evolving Phishing Attacks with LLMs and Genetic Algorithms

Discussions

• Google Says AI Could Break Reality [lemmy, 26 points, 8 comments]
• Wie KI für Betrug und Manipulation genutzt wird [lemmy, 5 points, 0 comments]
• Generative AI Misuse: A Taxonomy of Tactics and Insights from Real-World Data [hn, 4 points, 0 comments]
• Generative AI Misuse: A Taxonomy of Tactics and Insights from Real-World Data [hn, 4 points, 0 comments]
• this since people give AI far to much credit, but that is in part because of the sheer driving force behind it - "BIG TECH". It is alarming yes, BUT here is a good source why you should do as mentione [bsky, 1 points, 0 comments]
• Generative AI Misuse: A Taxonomy of Tactics and Insights from Real-World Data [hn, 1 points, 0 comments]
• "What does good look like" is an interesting question, but one that probably should be asked from the jump by the people making any product or service, right? This paper is an interesting look at typ [bsky, 1 points, 1 comments]
• If you want to have a look to the paper I am talking about it is available here 👇 https://arxiv.org/abs/2406.13843 [bsky, 1 points, 0 comments]
• "Manipulation of human likeness and falsification of evidence underlie the most prevalent tactics in real-world cases of misuse. Most of these were deployed with a discernible intent to influence publ [bsky, 1 points, 0 comments]
• arxiv.org/abs/2406.13843 [bsky, 0 points, 0 comments]
• "This contamination of publicly accessible data with AI-generated content could potentially... distort collective understanding of socio-political reality ...high-profile individuals are able to expla [bsky, 0 points, 0 comments]
• https://arxiv.org/abs/2406.13843 [bsky, 0 points, 0 comments]

Related

• Impacts and Risk of Generative AI Technology on Cyber Defense
• Business (mis)Use Cases of Generative AI
• A Closer Look at the Existing Risks of Generative AI: Mapping the Who, What, and How of Real-World Incidents
• Unintentional Consequences: Generative AI Use for Cybercrime
• Generative AI Models: Opportunities and Risks for Industry and Authorities