On the Risk of Misinformation Pollution with Large Language Models

2023/05/23 by Yikang Pan, Pan, Yikang, Liangming Pan +9 · 24 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Hate Speech and Cyberbullying Detection #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2305.13661

openalex publication_date 2023/05/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we comprehensively investigate the potential misuse of modern Large Language Models (LLMs) for generating credible-sounding misinformation and its subsequent impact on information-intensive applications, particularly Open-Domain Question Answering (ODQA) systems. We establish a threat model and simulate potential misuse scenarios, both unintentional and intentional, to assess the extent to which LLMs can be utilized to produce misinformation. Our study reveals that LLMs can act as effective misinformation generators, leading to a significant degradation in the performance of ODQA systems. To mitigate the harm caused by LLM-generated misinformation, we explore three defense strategies: prompting, misinformation detection, and majority voting. While initial results show promising trends for these defensive strategies, much more work needs to be done to address the challenge of misinformation pollution. Our work highlights the need for further research and interdisciplinary collaboration to address LLM-generated misinformation and to promote responsible use of LLMs.

Cited by

• Identifying Bias in Machine-generated Text Detection
• PVMark: Enabling Public Verifiability for LLM Watermarking Schemes
• RAGuard: A Layered Defense Framework for Retrieval-Augmented Generation Systems Against Data Poisoning
• NeuroGenPoisoning: Neuron-Guided Attacks on Retrieval-Augmented Generation of LLM via Genetic Optimization of External Knowledge
• Explainability of Large Language Models: Opportunities and Challenges toward Generating Trustworthy Explanations
• All You Need is One: Capsule Prompt Tuning with a Single Vector
• SimKey: A Semantically Aware Key Module for Watermarking Language Models
• ADMIT: Few-shot Knowledge Poisoning Attacks on RAG-based Fact Checking
• Detecting LLM-Generated Spam Reviews by Integrating Language Model Embeddings and Graph Neural Network
• Using Grok to Avoid Personal Attacks While Correcting Misinformation on X
• Incentive-Aligned Multi-Source LLM Summaries
• SafeSearch: Automated Red-Teaming for the Safety of LLM-Based Search Agents
• Security and Privacy Challenges of Large Language Models: A Survey
• MPCG: Multi-Round Persona-Conditioned Generation for Modeling the Evolution of Misinformation with LLMs
• LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systems
• Character-Level Perturbations Disrupt LLM Watermarks
• HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistants
• Mind Your Server: A Systematic Study of Parasitic Toolchain Attacks on the MCP Ecosystem
• A Multi-Strategy Approach for AI-Generated Text Detection
• ConspirED: A Dataset for Cognitive Traits of Conspiracy Theories and Large Language Model Safety
• Embodied AI: Emerging Risks and Opportunities for Policy Action
• Adversarial Attacks against Neural Ranking Models via In-Context Learning
• A Survey on Data Security in Large Language Models
• A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directions

Related

• Combating Misinformation in the Age of LLMs: Opportunities and Challenges
• Combating misinformation in the age of LLMs: Opportunities and challenges
• Attacking Open-domain Question Answering by Injecting Misinformation
• How does Misinformation Affect Large Language Model Behaviors and Preferences?
• A Survey on Proactive Defense Strategies Against Misinformation in Large Language Models