2024/09/12 by Réda Alami, Alami, Reda, Ali Khalifa Almansoori +9 · 2 citations
Engineering · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Safety Systems Engineering in Autonomy
paper · pdf · doi:10.48550/arxiv.2409.07772
openalex publication_date 2024/09/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
We demonstrate that preference optimization methods can effectively enhance LLM safety. Applying various alignment techniques to the Falcon 11B model using safety datasets, we achieve a significant boost in global safety score (from 57.64% to 99.90%) as measured by LlamaGuard 3 8B, competing with state-of-the-art models. On toxicity benchmarks, average scores in adversarial settings dropped from over 0.6 to less than 0.07. However, this safety improvement comes at the cost of reduced general capabilities, particularly in math, suggesting a trade-off. We identify noise contrastive alignment (Safe-NCA) as an optimal method for balancing safety and performance. Our study ultimately shows that alignment techniques can be sufficient for building safe and robust models.