AMPED: Adaptive Multi-objective Projection for balancing Exploration and skill Diversification

2025/06/06 by Geonwoo Cho, Cho, Geonwoo, Jaemoon Lee +9
Computer Science · #Adaptability #Adaptation (eye) #Artificial Intelligence (cs.AI) #Diversification (marketing strategy) #Diversity (politics) #Domain Adaptation and Few-Shot Learning #Exploit #FOS: Computer and information sciences #Machine Learning (cs.LG) #Projection (relational algebra) #Reinforcement Learning in Robotics #Reinforcement learning #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2506.05980

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/06/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Skill-based reinforcement learning (SBRL) enables rapid adaptation in environments with sparse rewards by pretraining a skill-conditioned policy. Effective skill learning requires jointly maximizing both exploration and skill diversity. However, existing methods often face challenges in simultaneously optimizing for these two conflicting objectives. In this work, we propose a new method, Adaptive Multi-objective Projection for balancing Exploration and skill Diversification (AMPED), which explicitly addresses both: during pre-training, a gradient-surgery projection balances the exploration and diversity gradients, and during fine-tuning, a skill selector exploits the learned diversity by choosing skills suited to downstream tasks. Our approach achieves performance that surpasses SBRL baselines across various benchmarks. Through an extensive ablation study, we identify the role of each component and demonstrate that each element in AMPED is contributing to performance. We further provide theoretical and empirical evidence that, with a greedy skill selector, greater skill diversity reduces fine-tuning sample complexity. These results highlight the importance of explicitly harmonizing exploration and diversity and demonstrate the effectiveness of AMPED in enabling robust and generalizable skill learning. Project Page: https://geonwoo.me/amped/

Citations

• Task Adaptation from Skills: Information Geometry, Disentanglement, and New Objectives for Unsupervised Reinforcement Learning
• InfoNCE: Identifying the Gap Between Theory and Practice
• Constrained Ensemble Exploration for Unsupervised Skill Discovery
• METRA: Scalable Unsupervised RL with Metric-Aware Abstraction
• Maximum State Entropy Exploration using Predecessor and Successor Representations
• Behavior Contrastive Learning for Unsupervised Skill Discovery
• Reward Design with Language Models
• Controllability-Aware Unsupervised Skill Discovery
• Anti-Exploration by Random Network Distillation
• Curiosity-Driven Multi-Agent Exploration with Mixed Objectives
• Skill-based Model-based Reinforcement Learning
• Multi-Task Learning as a Bargaining Game
• Lipschitz-constrained Unsupervised Skill Discovery
• CIC: Contrastive Intrinsic Control for Unsupervised Skill Discovery
• URLB: Unsupervised Reinforcement Learning Benchmark
• Conflict-Averse Gradient Descent for Multi-task Learning
• The Information Geometry of Unsupervised Reinforcement Learning
• APS: Active Pretraining with Successor Features
• Deep Reinforcement Learning at the Edge of the Statistical Precipice
• Learning more skills through optimistic exploration
• Mutual Information State Intrinsic Control
• Behavior From the Void: Unsupervised Active Pre-Training
• Relative Variational Intrinsic Control
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills
• Gradient Surgery for Multi-Task Learning
• Dynamics-Aware Unsupervised Discovery of Skills
• Self-Supervised Exploration via Disagreement
• Exploration by Random Network Distillation
• Representation Learning with Contrastive Predictive Coding
• Unsupervised Meta-Learning for Reinforcement Learning
• Diversity is All You Need: Learning Skills without a Reward Function
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• DeepMind Control Suite
• StarCraft II: A New Challenge for Reinforcement Learning
• Proximal Policy Optimization Algorithms
• Curiosity-driven Exploration by Self-supervised Prediction
• Variational Intrinsic Control
• Continuous control with deep reinforcement learning
• Curiosity and the pleasures of learning: Wanting and liking new information
• ComSD: Balancing Behavioral Quality and Diversity in Unsupervised Skill Discovery

Related

• Neuroevolution is a Competitive Alternative to Reinforcement Learning for Skill Discovery
• SkillS: Adaptive Skill Sequencing for Efficient Temporally-Extended Exploration
• VendiRL: A Framework for Self-Supervised Reinforcement Learning of Diversely Diverse Skills
• Trajectory First: A Curriculum for Discovering Diverse Policies
• APART: Diverse Skill Discovery using All Pairs with Ascending Reward and DropouT