Decoding Human Preferences in Alignment: An Improved Approach to Inverse Constitutional AI

2025/01/28 by Carl-Leander Henneking, Claas Beger, Henneking, Carl-Leander +1
Computer Science · Social Sciences · #Artificial Intelligence in Law #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2501.17112

openalex publication_date 2025/01/28 · openalex created_date 2025/01/30 · openalex updated_date 2026/07/28

Abstract

Traditional methods for aligning Large Language Models (LLMs), such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), rely on implicit principles, limiting interpretability. Constitutional AI (CAI) offers an explicit, rule-based framework for guiding LLM alignment. Building on this, we refine the Inverse Constitutional AI (ICAI) algorithm, which extracts constitutions from preference datasets. By improving principle generation, clustering, and embedding processes, our approach enhances the accuracy and generalizability of extracted principles across synthetic and real-world datasets. Our results highlight the potential of these principles to foster more transparent and adaptable alignment methods, offering a promising direction for future advancements beyond traditional fine-tuning.

Related

• Inverse Constitutional AI: Compressing Preferences into Principles
• IterAlign: Iterative Constitutional Alignment of Large Language Models
• Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
• Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback
• Aligning Large Language Models With Human Feedback: Mathematical foundations and algorithm design [Special Issue on the Mathematics of Deep Learning]