2023/10/12 by Giovanni Burbi, Alberto Baldrati, Burbi, Giovanni +7 · 3 citations
Computer Science · Psychology · Social Sciences · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Hate Speech and Cyberbullying Detection #Humor Studies and Applications #Misinformation and Its Impacts
paper · pdf · doi:10.48550/arxiv.2310.08368
openalex publication_date 2023/10/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Multimodal image-text memes are prevalent on the internet, serving as a unique form of communication that combines visual and textual elements to convey humor, ideas, or emotions. However, some memes take a malicious turn, promoting hateful content and perpetuating discrimination. Detecting hateful memes within this multimodal context is a challenging task that requires understanding the intertwined meaning of text and images. In this work, we address this issue by proposing a novel approach named ISSUES for multimodal hateful meme classification. ISSUES leverages a pre-trained CLIP vision-language model and the textual inversion technique to effectively capture the multimodal semantic content of the memes. The experiments show that our method achieves state-of-the-art results on the Hateful Memes Challenge and HarMeme datasets. The code and the pre-trained models are publicly available at https://github.com/miccunifi/ISSUES.