Don't Mix Rewards, Mix Policies: Policy Decomposition and Optimization for Multi-Reward RL

2026/07/31 by Ruiming Liang, Yi Zhong, Yizhen Yuan +6
Computer Science · #cs.AI

paper · pdf

arxiv created 2026/07/31 · arxiv updated 2026/08/03

Abstract

Modern large language models (LLMs) are expected not just to answer correctly, but to adapt their behavior to different human values and use cases. As a result, multi-reward reinforcement learning (RL) has become an increasingly important problem for LLMs, where each reward captures a different aspect of desired behavior. However, optimizing with multiple rewards suffers from a more severe alignment tax issue, where different optimization objectives can trade off or even conflict with each other, leading to unstable and inefficient post-training. In this work, we propose PRISM, a new multi-reward RL framework built upon the idea of policy-space decomposition and composition. Instead of compositing different rewards, PRISM optimizes a set of standalone positive policies and a global negative policy. This alleviates the potential conflict during multi-reward policy optimization, while enabling controllability during inference by flexible policy composition. Experiments on scientific reasoning, tool-use reasoning, and helpfulness-safety alignment show that PRISM consistently outperforms existing multi-reward RL baselines, with extra controllability for inference-time preference control.

Citations

• Dichotomous Diffusion Policy Optimization
• Group Sequence Policy Optimization
• Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models
• ToolRL: Reward is All Tool Learning Needs
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts
• Multi-objective Reinforcement learning from AI Feedback
• Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning
• DMoERM: Recipes of Mixture-of-Experts for Effective Reward Modeling
• Dynamic Multi-Reward Weighting for Multi-Style Controllable Generation
• Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• WARM: On the Benefits of Weight Averaged Reward Models
• Controlled Text Generation via Language Model Arithmetic
• GPQA: A Graduate-Level Google-Proof Q&A Benchmark
• Safe RLHF: Safe Reinforcement Learning from Human Feedback
• Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging
• Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization
• Contrastive Decoding Improves Reasoning in Large Language Models
• Mitigating the Alignment Tax of RLHF
• BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset
• Fine-Grained Human Feedback Gives Better Rewards for Language Model Training
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• GPT-4 Technical Report
• Efficiently Scaling Transformer Inference
• Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• Training language models to follow instructions with human feedback
• A General Language Assistant as a Laboratory for Alignment
• DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts
• The Power of Scale for Parameter-Efficient Prompt Tuning
• Fine-Tuning Language Models from Human Preferences
• Proximal Policy Optimization Algorithms
• Constrained Policy Optimization
• Training Products of Experts by Minimizing Contrastive Divergence
• SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Models
• Aligning Large Language Models with Human: A Survey
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Prefix-Tuning: Optimizing Continuous Prompts for Generation