Finite-Time Convergence and Sample Complexity of Actor-Critic Multi-Objective Reinforcement Learning

2024/05/05 by Tianchen Zhou, Zhou, Tianchen, FNU Hairi +13 · 2 citations
Computer Science · Neuroscience · #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural dynamics and brain function #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2405.03082

openalex publication_date 2024/05/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Reinforcement learning with multiple, potentially conflicting objectives is pervasive in real-world applications, while this problem remains theoretically under-explored. This paper tackles the multi-objective reinforcement learning (MORL) problem and introduces an innovative actor-critic algorithm named MOAC which finds a policy by iteratively making trade-offs among conflicting reward signals. Notably, we provide the first analysis of finite-time Pareto-stationary convergence and corresponding sample complexity in both discounted and average reward settings. Our approach has two salient features: (a) MOAC mitigates the cumulative estimation bias resulting from finding an optimal common gradient descent direction out of stochastic samples. This enables provable convergence rate and sample complexity guarantees independent of the number of objectives; (b) With proper momentum coefficient, MOAC initializes the weights of individual policy gradients using samples from the environment, instead of manual initialization. This enhances the practicality and robustness of our algorithm. Finally, experiments conducted on a real-world dataset validate the effectiveness of our proposed method.

Cited by

• FIRM: Federated In-client Regularized Multi-objective Alignment for Large Language Models
• Enabling Pareto-Stationarity Exploration in Multi-Objective Reinforcement Learning: A Multi-Objective Weighted-Chebyshev Actor-Critic Approach

Related

• Enabling Pareto-Stationarity Exploration in Multi-Objective Reinforcement Learning: A Multi-Objective Weighted-Chebyshev Actor-Critic Approach
• Theoretical Study of Conflict-Avoidant Multi-Objective Reinforcement Learning
• On the Sample Complexity of Actor-Critic Method for Reinforcement Learning with Function Approximation
• A Sharper Global Convergence Analysis for Average Reward Reinforcement Learning via an Actor-Critic Approach
• Beyond Exponentially Fast Mixing in Average-Reward Reinforcement Learning via Multi-Level Monte Carlo Actor-Critic