Prioritized Soft Q-Decomposition for Lexicographic Reinforcement Learning

2023/10/03 by Finn Rietz, Rietz, Finn, Erik Schaffernicht +5 · 1 citation
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Reinforcement Learning in Robotics #Robot Manipulation and Learning #Software Engineering Research

paper · pdf · doi:10.48550/arxiv.2310.02360

openalex publication_date 2023/10/03 · openalex created_date 2023/10/06 · openalex updated_date 2026/07/28

Abstract

Reinforcement learning (RL) for complex tasks remains a challenge, primarily due to the difficulties of engineering scalar reward functions and the inherent inefficiency of training models from scratch. Instead, it would be better to specify complex tasks in terms of elementary subtasks and to reuse subtask solutions whenever possible. In this work, we address continuous space lexicographic multi-objective RL problems, consisting of prioritized subtasks, which are notoriously difficult to solve. We show that these can be scalarized with a subtask transformation and then solved incrementally using value decomposition. Exploiting this insight, we propose prioritized soft Q-decomposition (PSQD), a novel algorithm for learning and adapting subtask solutions under lexicographic priorities in continuous state-action spaces. PSQD offers the ability to reuse previously learned subtask solutions in a zero-shot composition, followed by an adaptation step. Its ability to use retained subtask training data for offline learning eliminates the need for new environment interaction during adaptation. We demonstrate the efficacy of our approach by presenting successful learning, reuse, and adaptation results for both low- and high-dimensional simulated robot control tasks, as well as offline learning results. In contrast to baseline approaches, PSQD does not trade off between conflicting subtasks or priority constraints and satisfies subtask priorities during learning. PSQD provides an intuitive framework for tackling complex RL problems, offering insights into the inner workings of the subtask composition.

Citations

• Constrained Markov Decision Processes
• Monte Carlo sampling methods using Markov chains and their applications
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learning
• Reward Constrained Policy Optimization
• Soft Actor-Critic Algorithms and Applications
• Reinforcement Learning: An Introduction

Cited by

• LPPG-RL: Lexicographically Projected Policy Gradient Reinforcement Learning with Subproblem Exploration

Related

• Composite Q-learning: Multi-scale Q-function Decomposition and Separable Optimization
• Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution
• Deep Reinforcement Learning with Adjustments
• LPPG-RL: Lexicographically Projected Policy Gradient Reinforcement Learning with Subproblem Exploration
• Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning