Course-Correction: Safety Alignment Using Synthetic Preferences

2024/07/23 by Xu, Rongwu, Cai, Yishuo, Zhou, Zhenhong +6 · 5 citations
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2407.16637

Abstract

The risk of harmful content generated by large language models (LLMs) becomes a critical concern. This paper presents a systematic study on assessing and improving LLMs' capability to perform the task of course-correction, \ie, the model can steer away from generating harmful content autonomously. To start with, we introduce the C²-Eval benchmark for quantitative assessment and analyze 10 popular LLMs, revealing varying proficiency of current safety-tuned LLMs in course-correction. To improve, we propose fine-tuning LLMs with preference learning, emphasizing the preference for timely course-correction. Using an automated pipeline, we create C²-Syn, a synthetic dataset with 750K pairwise preferences, to teach models the concept of timely course-correction through data-driven preference learning. Experiments on 2 LLMs, Llama2-Chat 7B and Qwen2 7B, show that our method effectively enhances course-correction skills without affecting general performance. Additionally, it effectively improves LLMs' safety, particularly in resisting jailbreak attacks.

Cited by

• FinTrust: A Comprehensive Benchmark of Trustworthiness Evaluation in Finance Domain
• UnsafeChain: Enhancing Reasoning Model Safety via Hard Cases
• From Threat to Tool: Leveraging Refusal-Aware Injection Attacks for Safety Alignment
• Towards Understanding the Cognitive Habits of Large Reasoning Models
• Exploring Multimodal Challenges in Toxic Chinese Detection: Taxonomy, Benchmark, and Findings

Related

• Mission Impossible: A Statistical Perspective on Jailbreaking LLMs
• Robustifying Safety-Aligned Large Language Models through Clean Data Curation
• Fake Alignment: Are LLMs Really Aligned Well?
• Separate the Wheat from the Chaff: A Post-Hoc Approach to Safety Re-Alignment for Fine-Tuned Language Models
• Safety Arithmetic: A Framework for Test-time Safety Alignment of Language Models by Steering Parameters and Activations