Rectifying Regression in Reinforcement Learning

2025/10/01 by Alex Ayoub, Ayoub, Alex, David Szepesvári +6
Computer Science · #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2510.00885

openalex publication_date 2025/10/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper investigates the impact of the loss function in value-based methods for reinforcement learning through an analysis of underlying prediction objectives. We theoretically show that mean absolute error is a better prediction objective than the traditional mean squared error for controlling the learned policy's suboptimality gap. Furthermore, we present results that different loss functions are better aligned with these different regression objectives: binary and categorical cross-entropy losses with the mean absolute error and squared loss with the mean squared error. We then provide empirical evidence that algorithms minimizing these cross-entropy losses can outperform those based on the squared loss in linear reinforcement learning.

Citations

• Switching the Loss Reduces the Cost in Batch (Offline) Reinforcement Learning
• Stop Regressing: Training Value Functions via Classification for Scalable Deep RL
• Investigating the Histogram Loss in Regression
• More Benefits of Being Distributional: Second-Order Bounds for Reinforcement Learning
• Efficient First-Order Contextual Bandits: Prediction, Allocation, and Triangular Discrimination
• Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithms
• Bandit Algorithms
• Model-Based Reinforcement Learning with Value-Targeted Regression
• Provably Efficient Reinforcement Learning with Linear Function Approximation
• Information-Theoretic Considerations in Batch Reinforcement Learning
• A Comparative Analysis of Expected and Distributional Reinforcement Learning
• Improving Regression Performance with Distributional Losses
• Distributional Reinforcement Learning with Quantile Regression
• A Distributional Perspective on Reinforcement Learning
• Fast Rates for General Unbounded Loss Functions: from ERM to Generalized Bayes
• Agnostic Learning of Monomials by Halfspaces is Hard
• Reinforcement Learning: An Introduction

Related

• The Central Role of the Loss Function in Reinforcement Learning
• A Note on Loss Functions and Error Compounding in Model-based Reinforcement Learning
• Robust Losses for Learning Value Functions
• RL as Regressor: A Reinforcement Learning Approach for Function Approximation
• Improving Regression Performance with Distributional Losses