PrefPoE: Advantage-Guided Preference Fusion for Learning Where to Explore

2025/11/11 by Zhihao Lin, Lin, Zhihao, Lin Wu +5
Computer Science · #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2511.08241

openalex publication_date 2025/11/11 · openalex created_date 2025/11/13 · openalex updated_date 2026/07/28

Abstract

Exploration in reinforcement learning remains a critical challenge, as naive entropy maximization often results in high variance and inefficient policy updates. We introduce PrefPoE, a novel Preference-Product-of-Experts framework that performs intelligent, advantage-guided exploration via the first principled application of product-of-experts (PoE) fusion for single-task exploration-exploitation balancing. By training a preference network to concentrate probability mass on high-advantage actions and fusing it with the main policy through PoE, PrefPoE creates a soft trust region that stabilizes policy updates while maintaining targeted exploration. Across diverse control tasks spanning both continuous and discrete action spaces, PrefPoE demonstrates consistent improvements: +321% on HalfCheetah-v4 (1276~→~5375), +69% on Ant-v4, +276% on LunarLander-v2, with consistently enhanced training stability and sample efficiency. Unlike standard PPO, which suffers from entropy collapse, PrefPoE sustains adaptive exploration through its unique dynamics, thereby preventing premature convergence and enabling superior performance. Our results establish that learning where to explore through advantage-guided preferences is as crucial as learning how to act, offering a general framework for enhancing policy gradient methods across the full spectrum of reinforcement learning domains. Code and pretrained models are available in supplementary materials.

Citations

• DIME:Diffusion-Based Maximum Entropy Reinforcement Learning
• Continuous Control with Coarse-to-fine Reinforcement Learning
• Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF
• Mixtures of Experts Unlock Parameter Scaling for Deep RL
• Optimal Exploration for Model-Based RL in Nonlinear Systems
• Optimistic Natural Policy Gradient: a Simple Efficient Policy Optimization Framework for Online RL
• Efficient Continuous Control with Double Actors and Regularized Critics
• Deep Reinforcement Learning with Population-Coded Spiking Neural Network for Continuous Control
• V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control
• Challenges of Real-World Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Divide-and-Conquer Reinforcement Learning
• Overcoming Exploration in Reinforcement Learning with Demonstrations
• Proximal Policy Optimization Algorithms
• Parameter Space Noise for Exploration
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic
• Continuous control with deep reinforcement learning
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Trust Region Policy Optimization
• Training Products of Experts by Minimizing Contrastive Divergence

Related

• Preference-Guided Reinforcement Learning for Efficient Exploration
• PPO-BR: Dual-Signal Entropy-Reward Adaptation for Trust Region Policy Optimization
• EEPO: Exploration-Enhanced Policy Optimization via Sample-Then-Forget
• Reasoning through Exploration: A Reinforcement Learning Framework for Robust Function Calling
• Entropy Augmented Reinforcement Learning