MAVIS: Multi-Objective Alignment via Value-Guided Inference-Time Search

2025/08/19 by Carleton, Jeremy, Mukherjee, Debajoy, Shakkottai, Srinivas +1
#FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2508.13415

Abstract

Large Language Models (LLMs) are increasingly deployed across diverse applications that demand balancing multiple, often conflicting, objectives -- such as helpfulness, harmlessness, or humor. Aligning outputs to user-specific preferences in such multi-objective settings typically requires fine-tuning models for each objective or preference configuration, which is computationally expensive and inflexible. We introduce MAVIS -- Multi-Objective Alignment via Value-Guided Inference-Time Search -- a lightweight inference-time alignment framework that enables dynamic control over LLM behavior without modifying the base model's weights. MAVIS trains a set of small value models, each corresponding to a distinct objective. At inference time, these value models are combined using user-specified weights to produce a tilting function that adjusts the base model's output distribution toward desired trade-offs. The value models are trained using a simple iterative algorithm that ensures monotonic improvement of the KL-regularized policy. We show empirically that MAVIS outperforms baselines that fine-tune per-objective models and combine them post hoc, and even approaches the performance of the idealized setting where models are fine-tuned for a user's exact preferences.

Citations

• On Evaluating LLM Alignment by Evaluating LLMs as Judges
• Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewards
• Bounded Rationality for LLMs: Satisficing Alignment at Inference-Time
• Value-Guided Search for Efficient Chain-of-Thought Reasoning
• Multi-Objective Preference Optimization: Improving Human Alignment of Generative Models
• PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model
• Robust Multi-Objective Controlled Decoding of Large Language Models
• Robust Multi-Objective Preference Alignment with Online DPO
• Q\sharp: Provably Optimal Distributional RL for LLM Post-Training
• Amulet: ReAlignment During Test Time for Personalized Preference Adaptation of LLMs
• Towards Cost-Effective Reward Guided Text Generation
• QLASS: Boosting Language Agent Inference via Q-Guided Stepwise Search
• On Almost Surely Safe Alignment of Large Language Models at Inference-Time
• Entropy-Regularized Process Reward Model
• GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment
• VerifierQ: Enhancing LLM Test Time Compute with Q-Learning-based Verifiers
• VinePPO: Refining Credit Assignment in RL Training of LLMs
• Inference-Time Language Model Alignment via Integrated Value Guidance
• Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuning
• Q-Adapter: Customizing Pre-trained LLMs to New Preferences with Forgetting Mitigation
• Decoding-Time Language Model Alignment with Multiple Objectives
• PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference
• Aligning Large Language Models with Representation Editing: A Control Perspective
• Transfer Q Star: Principled Decoding for LLM Alignment
• Value Augmented Sampling for Language Model Alignment and Personalization
• Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment
• Direct Language Model Alignment from Online AI Feedback
• Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback
• Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training
• Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decoding
• Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Scaling Laws for Reward Model Overoptimization
• Anchor-Changing Regularized Natural Policy Gradient for Multi-Objective Reinforcement Learning
• Offline RL for Natural Language Generation with Implicit Language Q Learning
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• Training language models to follow instructions with human feedback
• Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selection
• Learning to summarize from human feedback
• A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation
• Deep Reinforcement Learning and the Deadly Triad
• Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Dynamic Policy Programming

Related

• Inference time LLM alignment in single and multidomain preference spectrum
• Bounded Rationality for LLMs: Satisficing Alignment at Inference-Time
• MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models
• PITA: Preference-Guided Inference-Time Alignment for LLM Post-Training
• MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time