Rethinking the Evaluation of Alignment Methods: Insights into Diversity, Generalisation, and Safety

2025/09/16 by Denis Janiak, Janiak, Denis, Julia Moska +11
Decision Sciences · Pharmacology, Toxicology and Pharmaceutics · #Computation and Language (cs.CL) #Evaluation and Performance Assessment #FOS: Computer and information sciences #Machine Learning (cs.LG) #Pharmacy and Medical Practices

paper · pdf · doi:10.48550/arxiv.2509.12936

openalex publication_date 2025/09/16 · openalex created_date 2025/10/18 · openalex updated_date 2026/07/28

Abstract

Large language models (LLMs) require careful alignment to balance competing objectives - factuality, safety, conciseness, proactivity, and diversity. Existing studies focus on individual techniques or specific dimensions, lacking a holistic assessment of the inherent trade-offs. We propose a unified evaluation framework that compares LLM alignment methods (PPO, DPO, ORPO, KTO) across these five axes, using both in-distribution and out-of-distribution datasets. Leveraging a specialized LLM-as-Judge prompt, validated through human studies, we reveal that DPO and KTO excel in factual accuracy, PPO and DPO lead in safety, and PPO best balances conciseness with proactivity. Our findings provide insights into trade-offs of common alignment methods, guiding the development of more balanced and reliable LLMs.

Citations

• Look Before You Leap: Enhancing Attention and Vigilance Regarding Harmful Content with GuidelineLLM
• One fish, two fish, but not the whole sea: Alignment reduces language models’ conceptual diversity
• Taming Overconfidence in LLMs: Reward Calibration in RLHF
• CS4: Measuring the Creativity of Large Language Models Automatically by Controlling the Number of Story-Writing Constraints
• The Llama 3 Herd of Models
• Benchmarking Language Model Creativity: A Case Study on Code Generation
• PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference
• Creativity Has Left the Chat: The Price of Debiasing Language Models
• ORPO: Monolithic Preference Optimization without Reference Model
• Exploring Precision and Recall to assess the quality and diversity of LLMs
• Tradeoffs Between Alignment and Helpfulness in Language Models with Steering Methods
• Understanding the Effects of RLHF on LLM Generalisation and Diversity
• BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback
• AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback
• A Survey on Proactive Dialogue Systems: Problems, Methods, and Prospects
• LLaMA: Open and Efficient Foundation Language Models
• Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkit
• Rethinking and Refining the Distinct Metric
• Learning to summarize from human feedback
• Fine-Tuning Language Models from Human Preferences
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• Proximal Policy Optimization Algorithms
• A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

Related

• A Comprehensive Evaluation framework of Alignment Techniques for LLMs
• Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
• ABC Align: Large Language Model Alignment for Safety & Accuracy
• Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks
• A Comprehensive Survey of LLM Alignment Techniques: RLHF, RLAIF, PPO, DPO and More