Preference Tuning For Toxicity Mitigation Generalizes Across Languages

2024/06/23 by Xiaochen Li, Zheng-Xin Yong, Li, Xiaochen +3 · 1 voice · 6 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Cryptography and Security (cs.CR) #FOS: Computer and information sciences #Machine Learning (cs.LG) #cs.AI #cs.CL #cs.CR #cs.LG

paper · pdf · doi:10.48550/arxiv.2406.16235

arxiv published 2024/06/23 · arxiv updated 2024/11/08

Abstract

Detoxifying multilingual Large Language Models (LLMs) has become crucial due to their increasing global use. In this work, we explore zero-shot cross-lingual generalization of preference tuning in detoxifying LLMs. Unlike previous studies that show limited cross-lingual generalization for other safety tasks, we demonstrate that Direct Preference Optimization (DPO) training with only English data can significantly reduce toxicity in multilingual open-ended generations. For example, the probability of mGPT-1.3B generating toxic continuations drops from 46.8% to 3.9% across 17 different languages after training. Our results also extend to other multilingual LLMs, such as BLOOM, Llama3, and Aya-23. Using mechanistic interpretability tools like causal intervention and activation analysis, we identified the dual multilinguality property of MLP layers in LLMs, which explains the cross-lingual generalization of DPO. Finally, we show that bilingual sentence retrieval can predict the cross-lingual transferability of DPO preference tuning.

Cited by

• Beyond a Global Norm: Personalizing Toxicity Sensitivity in Language Models Without Retraining
• Patching LLM Like Software: A Lightweight Method for Improving Safety Policy in Large Language Models
• Self-Jailbreaking: Language Models Can Reason Themselves Out of Safety Alignment After Benign Reasoning Training
• Social Bias in Multilingual Language Models: A Survey
• ToxiFrench: Benchmarking and Enhancing Language Models via CoT Fine-Tuning for French Toxicity Detection
• Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLM

Discussions

• Debiasing in English transfers to a variety of other languages. Really neat result, accruing evidence of an "interlingua" representation in LLMs. arxiv.org/abs/2406.16235 [bsky, 4 points, 0 comments]

Related

• Cross-Lingual Transfer of Debiasing and Detoxification in Multilingual LLMs: An Extensive Investigation
• From One to Many: Expanding the Scope of Toxicity Mitigation in Language Models
• PolygloToxicityPrompts: Multilingual Evaluation of Neural Toxic Degeneration in Large Language Models
• Breaking mBad! Supervised Fine-tuning for Cross-Lingual Detoxification
• The Hidden Space of Safety: Understanding Preference-Tuned LLMs in Multilingual context