Vilio: State-of-the-art Visio-Linguistic Models applied to Hateful Memes

2020/12/14 by Niklas Muennighoff, Muennighoff, Niklas · 3 citations
Computer Science · Psychology · Social Sciences · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Digital Games and Media #FOS: Computer and information sciences #Hate Speech and Cyberbullying Detection #Humor Studies and Applications

paper · pdf · doi:10.48550/arxiv.2012.07788

openalex publication_date 2020/12/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This work presents Vilio, an implementation of state-of-the-art visio-linguistic models and their application to the Hateful Memes Dataset. The implemented models have been fitted into a uniform code-base and altered to yield better performance. The goal of Vilio is to provide a user-friendly starting point for any visio-linguistic problem. An ensemble of 5 different V+L models implemented in Vilio achieves 2nd place in the Hateful Memes Challenge out of 3,300 participants. The code is available at https://github.com/Muennighoff/vilio.

Citations

• Averaging Weights Leads to Wider Optima and Better Generalization
• Pythia v0.1: the Winning Entry to the VQA Challenge 2018
• ERNIE: Enhanced Language Representation with Informative Entities
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Cited by

• Unpacking Hateful Memes: Presupposed Context and False Claims
• Fall into a Pit, Gain in a Wit: Cognitive-Guided Harmful Meme Detection via Misjudgment Risk Pattern Retrieval
• Diagnosing the Impact of AI on Radiology in China

Related

• Detecting and Correcting Hate Speech in Multimodal Memes with Large Visual Language Model
• Detecting and Mitigating Hateful Content in Multimodal Memes with Vision-Language Models
• Labels or Input? Rethinking Augmentation in Multimodal Hate Detection
• Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challenge
• Hateful Memes Detection via Complementary Visual and Linguistic Networks