The CRINGE Loss: Learning what language not to model

2022/11/10 by Leonard Adolphs, Tianyu Gao, Adolphs, Leonard +9 · 4 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2211.05826

openalex publication_date 2022/11/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Standard language model training employs gold human documents or human-human interaction data, and treats all training data as positive examples. Growing evidence shows that even with very large amounts of positive training data, issues remain that can be alleviated with relatively small amounts of negative data -- examples of what the model should not do. In this work, we propose a novel procedure to train with such data called the CRINGE loss (ContRastive Iterative Negative GEneration). We show the effectiveness of this approach across three different experiments on the tasks of safe generation, contradiction avoidance, and open-domain dialogue. Our models outperform multiple strong baselines and are conceptually simple, easy to train and implement.

Cited by

• A Survey on Generative Model Unlearning: Fundamentals, Taxonomy, Evaluation, and Future Direction
• Teaching Models to Understand (but not Generate) High-risk Data
• Reward Model Overoptimisation in Iterated RLHF
• Exploring Criteria of Loss Reweighting to Enhance LLM Unlearning

Related

• Large Language Model Unlearning
• Towards Transfer Unlearning: Empirical Evidence of Cross-Domain Bias Mitigation
• CLINE: Contrastive Learning with Semantic Negative Examples for Natural Language Understanding
• Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning
• CURE: Controlled Unlearning for Robust Embeddings -- Mitigating Conceptual Shortcuts in Pre-Trained Language Models