A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity

2024/01/03 by Andrew Lee, Lee, Andrew, Bai, Xiaoyan +8 · 27 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning and Data Classification #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2401.01967

openalex publication_date 2024/01/03 · openalex created_date 2024/01/10 · openalex updated_date 2026/07/28

Abstract

While alignment algorithms are now commonly used to tune pre-trained language models towards a user's preferences, we lack explanations for the underlying mechanisms in which models become ``aligned'', thus making it difficult to explain phenomena like jailbreaks. In this work we study a popular algorithm, direct preference optimization (DPO), and the mechanisms by which it reduces toxicity. Namely, we first study how toxicity is represented and elicited in a pre-trained language model, GPT2-medium. We then apply DPO with a carefully crafted pairwise dataset to reduce toxicity. We examine how the resulting model averts toxic outputs, and find that capabilities learned from pre-training are not removed, but rather bypassed. We use this insight to demonstrate a simple method to un-align the model, reverting it back to its toxic behavior.

Cited by

• Better World Models Can Lead to Better Post-Training Performance
• An Empirical Study of Sample Selection Strategies for Large Language Model Repair
• Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting
• Executable Knowledge Graphs for Replicating AI Research
• Explainability of Large Language Models: Opportunities and Challenges toward Generating Trustworthy Explanations
• DSCD: Large Language Model Detoxification with Self-Constrained Decoding
• From Defender to Devil? Unintended Risk Interactions Induced by LLM Defenses
• On the Convergence of Moral Self-Correction in Large Language Models
• Refusal Falls off a Cliff: How Safety Alignment Fails in Reasoning?
• Defeating Cerberus: Concept-Guided Privacy-Leakage Mitigation in Multimodal Language Models
• Task Vectors, Learned Not Extracted: Performance Gains and Mechanistic Insight
• Detoxifying Large Language Models via Autoregressive Reward Guided Representation Editing
• Understanding Post-Training Structural Changes in Large Language Models
• Collapse of Irrelevant Representations (CIR) Ensures Robust and Non-Disruptive LLM Unlearning
• NeuroBreak: Unveil Internal Jailbreak Mechanisms in Large Language Models
• When Thinking Backfires: Mechanistic Insights Into Reasoning-Induced Misalignment
• Safety Alignment Should Be Made More Than Just A Few Attention Heads
• NEAT: Concept driven Neuron Attribution in LLMs
• The AI-Fraud Diamond: A Novel Lens for Auditing Algorithmic Deception
• ToxiFrench: Benchmarking and Enhancing Language Models via CoT Fine-Tuning for French Toxicity Detection
• Towards Integrated Alignment
• How Chain-of-Thought Works? Tracing Information Flow from Decoding, Projection, and Activation
• AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning
• CREME: Robustness Enhancement of Code LLMs via Layer-Aware Model Editing
• On the Impossibility of Separating Intelligence from Judgment: The Computational Intractability of Filtering for AI Alignment
• Discourse Heuristics For Paradoxically Moral Self-Correction
• Persona Features Control Emergent Misalignment

Related

• More is Less: The Pitfalls of Multi-Model Synthetic Preference Data in DPO Safety Alignment
• Model Editing as a Robust and Denoised variant of DPO: A Case Study on Toxicity
• Refined Direct Preference Optimization with Synthetic Data for Behavioral Alignment of LLMs
• D-STEER - Preference Alignment Techniques Learn to Behave, not to Believe -- Beneath the Surface, DPO as Steering Vector Perturbation in Activation Space
• sDPO: Don't Use Your Data All at Once