Zephyr: Direct Distillation of LM Alignment

2023/10/25 by Lewis Tunstall, Tunstall, Lewis, Edward Beeching +25 · 45 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Recommender Systems and Techniques #Speech and dialogue systems #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2310.16944

openalex publication_date 2023/10/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We aim to produce a smaller language model that is aligned to user intent. Previous research has shown that applying distilled supervised fine-tuning (dSFT) on larger models significantly improves task accuracy; however, these models are unaligned, i.e. they do not respond well to natural prompts. To distill this property, we experiment with the use of preference data from AI Feedback (AIF). Starting from a dataset of outputs ranked by a teacher model, we apply distilled direct preference optimization (dDPO) to learn a chat model with significantly improved intent alignment. The approach requires only a few hours of training without any additional sampling during fine-tuning. The final result, Zephyr-7B, sets the state-of-the-art on chat benchmarks for 7B parameter models, and requires no human annotation. In particular, results on MT-Bench show that Zephyr-7B surpasses Llama2-Chat-70B, the best open-access RLHF-based model. Code, models, data, and tutorials for the system are available at https://github.com/huggingface/alignment-handbook.

Cited by

• Towards a Relevance Posterior in Neural Information Access
• Towards Benchmarking Privacy Vulnerabilities in Selective Forgetting with Large Language Models
• Comparative Analysis of LLM Abliteration Methods: A Cross-Architecture Evaluation
• GLaD: Geometric Latent Distillation for Vision-Language-Action Models
• SPACE: Noise Contrastive Estimation Stabilizes Self-Play Fine-Tuning for Large Language Models
• On Evaluating LLM Alignment by Evaluating LLMs as Judges
• Geometric-disentangelment Unlearning
• Genomic Next-Token Predictors are In-Context Learners
• Defending Unauthorized Model Merging via Dual-Stage Weight Protection
• AMaPO: Adaptive Margin-attached Preference Optimization for Language Model Alignment
• CAPO: Confidence Aware Preference Optimization Learning for Multilingual Preferences
• DRAGON: Guard LLM Unlearning in Context via Negative Detection and Reasoning
• Advancing Equitable AI: Evaluating Cultural Expressiveness in LLMs for Latin American Contexts
• Efficiency vs. Alignment: Investigating Safety and Fairness Risks in Parameter-Efficient Fine-Tuning of LLMs
• Value Drifts: Tracing Value Alignment During LLM Post-Training
• Probing Knowledge Holes in Unlearned LLMs
• DETECT: Determining Ease and Textual Clarity of German Text Simplifications
• Adversarial Déjà Vu: Jailbreak Dictionary Learning for Stronger Generalization to Unseen Attacks
• Robust Preference Alignment via Directional Neighborhood Consensus
• Forgetting to Forget: Attention Sink as A Gateway for Backdooring LLM Unlearning
• Towards Understanding Valuable Preference Data for Large Language Model Alignment
• LLM Unlearning on Noisy Forget Sets: A Study of Incomplete, Rewritten, and Watermarked Data
• Active Model Selection for Large Language Models
• Large Language Models can extract morphological data from taxonomic descriptions, but their stochastic nature makes automation challenging: a test on Australian Asteraceae
• On the Role of Difficult Prompts in Self-Play Preference Optimization
• Making, not Taking, the Best of N
• Scaling with Collapse: Efficient and Predictable Training of LLM Families
• OrthAlign: Orthogonal Subspace Decomposition for Non-Interfering Multi-Objective Alignment
• Helpful assistant or fruitful facilitator? Investigating how personas affect language model behavior
• Who's Laughing Now? An Overview of Computational Humour Generation and Explanation
• WEE-Therapy: A Mixture of Weak Encoders Framework for Psychological Counseling Dialogue Analysis
• BridgeAlign: Bridging Preference Alignment for Humanities and Social Sciences
• MedLLM: An Open Medical Language Model at the Sub-Billion Scale
• References Improve LLM Alignment in Non-Verifiable Domains
• Improving User Interface Generation Models from Designer Feedback
• Decoding Uncertainty: The Impact of Decoding Strategies for Uncertainty Estimation in Large Language Models
• Evaluating Large Language Models for Cross-Lingual Retrieval
• Bhaasha, Bhasa, Zaban: A Survey for Low-Resourced Languages in South Asia -- Current Stage and Challenges
• Guarding Your Conversations: Privacy Gatekeepers for Secure Interactions with Cloud-Based AI Models
• An Epidemiological Knowledge Graph extracted from the World Health Organization's Disease Outbreak News
• The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authors
• Political Ideology Shifts in Large Language Models
• Improving Fisher Information Estimation and Efficiency for LoRA-based LLM Unlearning
• Better Language Model-Based Judging Reward Modeling through Scaling Comprehension Boundaries
• Rule2Text: A Framework for Generating and Evaluating Natural Language Explanations of Knowledge Graph Rules

Related

• Diversity First, Quality Later: A Two-Stage Assumption for Language Model Alignment
• Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignment
• Direct Large Language Model Alignment Through Self-Rewarding Contrastive Prompt Distillation
• Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback
• Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing