Reward-Weighted Regression Converges to a Global Optimum

2021/07/19 by Miroslav Štrupl, Francesco Faccio, Štrupl, Miroslav +7 · 1 citation
Biochemistry, Genetics and Molecular Biology · Computer Science · #68T05 #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Gene Regulatory Network Analysis #I.2.6 #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Receptor Mechanisms and Signaling #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2107.09088

openalex publication_date 2021/07/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/29

Abstract

Reward-Weighted Regression (RWR) belongs to a family of widely known iterative Reinforcement Learning algorithms based on the Expectation-Maximization framework. In this family, learning at each iteration consists of sampling a batch of trajectories using the current policy and fitting a new policy to maximize a return-weighted log-likelihood of actions. Although RWR is known to yield monotonic improvement of the policy under certain circumstances, whether and under which conditions RWR converges to the optimal policy have remained open questions. In this paper, we provide for the first time a proof that RWR converges to a global optimum when no function approximation is used, in a general compact setting. Furthermore, for the simpler case with finite state and action spaces we prove R-linear convergence of the state-value function to the optimum.

Cited by

• Generative Bayesian Optimization: Generative Models as Acquisition Functions

Related

• Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences
• The Local Optimality of Reinforcement Learning by Value Gradients, and\n its Relationship to Policy Gradient Learning
• Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policies
• Inverse Reinforcement Learning with Just Classification and a Few Regressions
• Horizon-Free and Instance-Dependent Regret Bounds for Reinforcement Learning with General Function Approximation