Limitations of Scalarisation in MORL: A Comparative Study in Discrete Environments

2025/11/20 by Muhammad Sa'ood Shah, Shah, Muhammad Sa'ood, Asad Jeewa +1 · 1 voice
Computer Science · #68T05 (Primary) 90C29 (Secondary) #Adaptive Dynamic Programming Control #Advanced Multi-Objective Optimization Algorithms #FOS: Computer and information sciences #G.1.6 #I.2.6 #Machine Learning (cs.LG) #Reinforcement Learning in Robotics #cs.LG

paper · pdf · doi:10.48550/arxiv.2511.16476

openalex publication_date 2025/11/20 · arxiv published 2025/11/20 · arxiv updated 2025/11/20 · openalex created_date 2025/11/23 · openalex updated_date 2026/07/28

Abstract

Scalarisation functions are widely employed in MORL algorithms to enable intelligent decision-making. However, these functions often struggle to approximate the Pareto front accurately, rendering them unideal in complex, uncertain environments. This study examines selected Multi-Objective Reinforcement Learning (MORL) algorithms across MORL environments with discrete action and observation spaces. We aim to investigate further the limitations associated with scalarisation approaches for decision-making in multi-objective settings. Specifically, we use an outer-loop multi-policy methodology to assess the performance of a seminal single-policy MORL algorithm, MO Q-Learning implemented with linear scalarisation and Chebyshev scalarisation functions. In addition, we explore a pioneering inner-loop multi-policy algorithm, Pareto Q-Learning, which offers a more robust alternative. Our findings reveal that the performance of the scalarisation functions is highly dependent on the environment and the shape of the Pareto front. These functions often fail to retain the solutions uncovered during learning and favour finding solutions in certain regions of the solution space. Moreover, finding the appropriate weight configurations to sample the entire Pareto front is complex, limiting their applicability in uncertain settings. In contrast, inner-loop multi-policy algorithms may provide a more sustainable and generalizable approach and potentially facilitate intelligent decision-making in dynamic and uncertain environments.

Citations

• Multi-Objective Reinforcement Learning Based on Decomposition: A Taxonomy and Framework
• PD-MORL: Preference-Driven Multi-Objective Reinforcement Learning Algorithm
• Optimistic Linear Support and Successor Features as a Basis for Optimal Policy Transfer
• Pareto Conditioned Networks
• A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation
• Dynamic Weights in Multi-Objective Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Successor Features for Transfer in Reinforcement Learning
• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning

Discussions

• Good to see some new empirical work on MORL. arxiv.org/pdf/2511.16476 [bsky, 0 points, 0 comments]

Related

• C-MORL: Multi-Objective Reinforcement Learning through Efficient Discovery of Pareto Front
• Addressing the issue of stochastic environments and local decision-making in multi-objective reinforcement learning
• Variance Reduced Policy Gradient Method for Multi-Objective Reinforcement Learning
• Traversing Pareto Optimal Policies: Provably Efficient Multi-Objective Reinforcement Learning
• On Generalization Across Environments In Multi-Objective Reinforcement Learning