MIND: A Multi-agent Framework for Zero-shot Harmful Meme Detection

2025/07/09 by Chunxiao Fan, Liu, Ziyan, Haoran Lou +6 · 1 citation
Computer Science · Social Sciences · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Hate Speech and Cyberbullying Detection #Misinformation and Its Impacts #Sentiment Analysis and Opinion Mining

paper · pdf · doi:10.48550/arxiv.2507.06908

openalex publication_date 2025/07/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The rapid expansion of memes on social media has highlighted the urgent need for effective approaches to detect harmful content. However, traditional data-driven approaches struggle to detect new memes due to their evolving nature and the lack of up-to-date annotated data. To address this issue, we propose MIND, a multi-agent framework for zero-shot harmful meme detection that does not rely on annotated data. MIND implements three key strategies: 1) We retrieve similar memes from an unannotated reference set to provide contextual information. 2) We propose a bi-directional insight derivation mechanism to extract a comprehensive understanding of similar memes. 3) We then employ a multi-agent debate mechanism to ensure robust decision-making through reasoned arbitration. Extensive experiments on three meme datasets demonstrate that our proposed framework not only outperforms existing zero-shot approaches but also shows strong generalization across different model architectures and parameter scales, providing a scalable solution for harmful meme detection. The code is available at https://github.com/destroy-lonely/MIND.

Citations

• Towards Low-Resource Harmful Meme Detection with LMM Agents
• MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution
• Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
• AutoDefense: Multi-Agent LLM Defense against Jailbreak Attacks
• MARG: Multi-Agent Review Generation for Scientific Papers
• InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
• Improved Baselines with Visual Instruction Tuning
• ExpeL: LLM Agents Are Experiential Learners
• Pro-Cap: Leveraging a Frozen Vision-Language Model for Hateful Meme Detection
• SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoning
• MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework
• ChatDev: Communicative Agents for Software Development
• Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate
• AdaPlanner: Adaptive Planning from Feedback with Language Models
• EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought
• Improving Factuality and Reasoning in Language Models through Multiagent Debate
• InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
• Generative Agents: Interactive Simulacra of Human Behavior
• Self-Refine: Iterative Refinement with Self-Feedback
• Prompting for Multimodal Hateful Meme Classification
• On the Evolution of (Hateful) Memes by Means of Multimodal Contrastive Learning
• Distributing Accountability, Not Capability: Phase Separation and the LLM Workflow Quadrant in Autonomous AI Agent Architectures
• Large Language Models are Zero-Shot Reasoners
• Detecting and Understanding Harmful Memes: A Survey
• On Explaining Multimodal Hateful Meme Detection Models
• MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targets
• Learning Transferable Visual Models From Natural Language Supervision
• Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challenge
• Enhance Multimodal Transformer With External Label And In-Domain Pretrain: Hateful Meme Challenge Winning Solution
• Vilio: State-of-the-art Visio-Linguistic Models applied to Hateful Memes
• Multimodal Learning for Hateful Memes Detection
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
• REALM: Retrieval-Augmented Language Model Pre-Training
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Supervised Multimodal Bitransformers for Classifying Images and Text
• Deep Residual Learning for Image Recognition
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation
• Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• From Evidence to Verdict: An Agent-Based Forensic Framework for AI-Generated Image Detection

Related

• Towards Low-Resource Harmful Meme Detection with LMM Agents
• From Shallow Humor to Metaphor: Towards Label-Free Harmful Meme Detection via LMM Agent Self-Improvement
• MemeMind: A Large-Scale Multimodal Dataset with Chain-of-Thought Reasoning for Harmful Meme Detection
• AdamMeme: Adaptively Probe the Reasoning Capacity of Multimodal Large Language Models on Harmfulness
• Detecting Harmful Memes with Decoupled Understanding and Guided CoT Reasoning