Multi-objective Large Language Model Alignment with Hierarchical Experts

2025/05/27 by Zhuo Li, Li, Zhuo, Guodong Du +18 · 1 citation
Computer Science · #Topic Modeling #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2505.20925

Abstract

Aligning large language models (LLMs) to simultaneously satisfy multiple objectives remains a significant challenge, especially given the diverse and often conflicting nature of human preferences. Existing alignment methods struggle to balance trade-offs effectively, often requiring costly retraining or yielding suboptimal results across the Pareto frontier of preferences. In this paper, we introduce HoE(Hierarchical Mixture-of-Experts), a lightweight, parameter-efficient, and plug-and-play approach that eliminates the need for model training, while enabling LLMs to adapt across the entire Pareto frontier and accommodate diverse user preferences. In particular, HoE consists of three hierarchical components: LoRA Experts, Router Experts and Preference Routing, reaching optimal Pareto frontiers and achieving a trade-off between parameter size, training cost, and performance. We evaluate HoE across various tasks on 14 objectives and 200 different preferences among 6 benchmarks, demonstrating superior performance over 15 recent baselines. Code is available in the supplementary materials.

Citations

• Function-to-Style Guidance of LLMs for Code Translation
• Adaptive Detoxification: Safeguarding General Capabilities of LLMs through Toxicity-Aware Knowledge Editing
• Safety Alignment via Constrained Knowledge Unlearning
• Neural Parameter Search for Slimmer Fine-Tuned Models and Better Transfer
• MTSA: Multi-turn Safety Alignment for LLMs through Multi-round Red-teaming
• Multi-Modality Expansion and Retention for LLMs through Parameter Merging and Decoupling
• PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model
• Delta Decompression for MoE-based LLMs Compression
• Knowledge Editing with Dynamic Knowledge Graphs for Multi-Hop Question Answering
• Multi-Objective Alignment of Large Language Models Through Hypervolume Maximization
• Impromptu Cybercrime Euphemism Detection
• Task Singular Vectors: Reducing Task Interference in Model Merging
• Adaptive Online Mirror Descent for Tchebycheff Scalarization in Multi-Objective Learning
• GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment
• Parameter Competition Balancing for Model Merging
• Traversing Pareto Optimal Policies: Provably Efficient Multi-Objective Reinforcement Learning
• Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuning
• Decoding-Time Language Model Alignment with Multiple Objectives
• Knowledge Fusion By Evolving Weights of Language Models
• Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language Models
• Multimodal Reasoning with Multimodal Knowledge Graph
• OR-Bench: An Over-Refusal Benchmark for Large Language Models
• One-Shot Safety Alignment for Large Language Models via Optimal Dualization
• MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models
• SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression
• Smooth Tchebycheff Scalarization for Multi-Objective Optimization
• Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards
• Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment
• Higher Layers Need More LoRA Experts
• X-LoRA: Mixture of Low-Rank Adapter Experts, a Flexible Framework for Large Language Models with Applications in Protein Mechanics and Molecular Design
• Aligner: Efficient Alignment by Learning to Correct
• Style Vectors for Steering Generative Large Language Model
• ARGS: Alignment as Reward-Guided Search
• ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models
• Steering Llama 2 via Contrastive Activation Addition
• Chain of Thought with Explicit Evidence Reasoning for Few-shot Relation Extraction
• Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging
• Merging Experts into One: Improving Computational Efficiency of Mixture of Experts
• Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization
• AdaMerging: Adaptive Model Merging for Multi-Task Learning
• Query-Dependent Prompt Evaluation and Optimization with Offline Inverse RL
• Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards
• TIES-Merging: Resolving Interference When Merging Models
• Efficient Storage of Fine-Tuned Models via Low-Rank Approximation of Weight Residuals
• GPT-4 Technical Report
• Dataless Knowledge Fusion by Merging Weights of Language Models
• Editing Models with Task Arithmetic
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• Merging Models with Fisher-Weighted Averaging
• Training Verifiers to Solve Math Word Problems
• Learning to summarize from human feedback
• Safe Policies for Reinforcement Learning via Primal-Dual Methods
• Fine-Tuning Language Models from Human Preferences
• Proximal Policy Optimization Algorithms
• The CMA Evolution Strategy: A Tutorial
• A Stochastic Approximation Method
• PAD: Personalized Alignment of LLMs at Decoding-Time
• FREE-Merging: Fourier Transform for Efficient Model Merging

Cited by

• Learning to Optimize Multi-Objective Alignment Through Dynamic Reward Weighting
• Knowledge Grafting of Large Language Models

Related

• T-REX: Mixture-of-Rank-One-Experts with Semantic-aware Intuition for Multi-task Large Language Model Finetuning
• GMoE: Empowering LLMs Fine-Tuning via MoE Graph Collaboration
• OrdMoE: Preference Alignment via Hierarchical Expert Group Ranking in Multimodal Mixture-of-Experts LLMs
• Mixture of Experts in Large Language Models
• Hierarchical Alignment: Surgical Fine-Tuning via Functional Layer Specialization in Large Language Models