One-Shot Safety Alignment for Large Language Models via Optimal Dualization

2024/05/29 by Xinmeng Huang, Huang, Xinmeng, Shuo Li +9 · 4 citations
Computer Science · #Topic Modeling #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2405.19544

Abstract

The growing safety concerns surrounding large language models raise an urgent need to align them with diverse human preferences to simultaneously enhance their helpfulness and safety. A promising approach is to enforce safety constraints through Reinforcement Learning from Human Feedback (RLHF). For such constrained RLHF, typical Lagrangian-based primal-dual policy optimization methods are computationally expensive and often unstable. This paper presents a perspective of dualization that reduces constrained alignment to an equivalent unconstrained alignment problem. We do so by pre-optimizing a smooth and convex dual function that has a closed form. This shortcut eliminates the need for cumbersome primal-dual policy iterations, greatly reducing the computational burden and improving training stability. Our strategy leads to two practical algorithms in model-based and preference-based settings (MoCAN and PeCAN, respectively). A broad range of experiments demonstrate the effectiveness and merits of our algorithms.

Cited by

• Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimization
• SafeSearch: Do Not Trade Safety for Utility in LLM Search Agents
• Primal-Dual Direct Preference Optimization for Constrained LLM Alignment
• MEUV: Achieving Fine-Grained Capability Activation in Large Language Models via Mutually Exclusive Unlock Vectors

Related

• SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety
• Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimization
• Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization
• Stepwise Alignment for Constrained Language Model Policy Optimization
• Enhancing LLM Safety via Constrained Direct Preference Optimization