Moderating Harm: Benchmarking Large Language Models for Cyberbullying Detection in YouTube Comments

2025/05/25 by Amel Muminovic, Muminovic, Amel · 2 citations
Computer Science · Psychology · #Artificial Intelligence (cs.AI) #Benchmarking #Best practice #Bullying, Victimization, and Aggression #Computation and Language (cs.CL) #Data quality #FOS: Computer and information sciences #Hate Speech and Cyberbullying Detection #Language model #Moderation #Precision and recall #Recall #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2505.18927

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/05/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

As online platforms grow, comment sections increasingly host harassment that undermines user experience and well-being. This study benchmarks three leading large language models, OpenAI GPT-4.1, Google Gemini 1.5 Pro, and Anthropic Claude 3 Opus, on a corpus of 5,080 YouTube comments sampled from high-abuse threads in gaming, lifestyle, food vlog, and music channels. The dataset comprises 1,334 harmful and 3,746 non-harmful messages in English, Arabic, and Indonesian, annotated independently by two reviewers with substantial agreement (Cohen's kappa = 0.83). Using a unified prompt and deterministic settings, GPT-4.1 achieved the best overall balance with an F1 score of 0.863, precision of 0.887, and recall of 0.841. Gemini flagged the highest share of harmful posts (recall = 0.875) but its precision fell to 0.767 due to frequent false positives. Claude delivered the highest precision at 0.920 and the lowest false-positive rate of 0.022, yet its recall dropped to 0.720. Qualitative analysis showed that all three models struggle with sarcasm, coded insults, and mixed-language slang. These results underscore the need for moderation pipelines that combine complementary models, incorporate conversational context, and fine-tune for under-represented languages and implicit abuse. A de-identified version of the dataset and full prompts is publicly released to promote reproducibility and further progress in automated content moderation.

Citations

• Automated Content Moderation Increases Adherence to Community Guidelines
• Assessing the impact of contextual information in hate speech detection
• Challenges in Detoxifying Language Models
• Just Say No: Analyzing the Stance of Neural Dialogue Generation in Offensive Contexts
• Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets
• A systematic review of Hate Speech automatic detection using Natural Language Processing
• On the Dangers of Stochastic Parrots
• Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection
• HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection
• Multilingual Offensive Language Identification with Cross-lingual Embeddings
• RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
• Toxicity Detection: Does Context Really Matter?
• GLUECoS : An Evaluation Benchmark for Code-Switched NLP
• Social Bias Frames: Reasoning about Social and Power Implications of Language
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• Predicting the Type and Target of Offensive Posts in Social Media
• Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior
• All You Need is "Love": Evading Hate-speech Detection
• Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• Large Language Models for Toxic Language Detection in Low-Resource Balkan Languages
• Cyberbullying Detection via Aggression-Enhanced Prompting

Related

• A Multi-Task Benchmark for Abusive Language Detection in Low-Resource Settings
• Large Language Models for Toxic Language Detection in Low-Resource Balkan Languages
• Explain Thyself Bully: Sentiment Aided Cyberbullying Detection with Explanation
• Harmful YouTube Video Detection: A Taxonomy of Online Harm and MLLMs as Alternative Annotators
• Advancing Content Moderation: Evaluating Large Language Models for Detecting Sensitive Content Across Text, Images, and Videos