Decoupled Alignment for Robust Plug-and-Play Adaptation

2024/06/03 by Haozheng Luo, Luo, Haozheng, Jiahao Yu +21
Computer Science · #Advanced Vision and Imaging #Human Pose and Action Recognition #Video Surveillance and Tracking Methods #cs.AI #cs.CL #cs.CR

paper · pdf · doi:10.48550/arxiv.2406.01514

openalex publication_date 2024/06/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We introduce a training-free safety enhancement method for aligning large language models (LLMs) without the need for supervised fine-tuning or reinforcement learning from human feedback. Our main idea is to provide a robust plug-and-play approach to prevent shadow alignment when models are adapted to downstream tasks. Specifically, we leverage knowledge distillation to extract alignment signals from well-aligned LLMs and inject them into shadow-aligned models via model fusion, enabling plug-and-play alignment correction. In our methodology, we employ delta debugging to identify the critical components of knowledge necessary for effective distillation. On the harmful question dataset, our method significantly enhances the average defense success rate by approximately 14.42%, reaching as high as 51.39% across 17 influenced LLMs, without compromising performance. Our code is available at https://github.com/NWULIST/DAPA.

Citations

• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Training language models to follow instructions with human feedback
• Hopfield Networks is All You Need
• Language Models are Few-Shot Learners
• Attention Is All You Need

Related

• Aligners: Decoupling LLMs and Alignment
• Contrastive Knowledge Transfer and Robust Optimization for Secure Alignment of Large Language Models
• Emulated Disalignment: Safety Alignment for Large Language Models May Backfire!
• Aligner: Efficient Alignment by Learning to Correct
• Large Language Model Alignment: A Survey