Learning by Playing - Solving Sparse Reward Tasks from Scratch

2018/02/28 by Martin Riedmiller, Roland Hafner, Riedmiller, Martin +15 · 21 citations
Computer Science · Mathematics · #Adaptive Dynamic Programming Control #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics #Robotics (cs.RO) #cs.LG #cs.RO #stat.ML

paper · pdf · doi:10.48550/arxiv.1802.10567

A video of the rich set of learned behaviours can be found at https://youtu.be/mPKyvocNe_M

arxiv created 2018/02/28 · openalex publication_date 2018/02/28 · arxiv updated 2018/03/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose Scheduled Auxiliary Control (SAC-X), a new learning paradigm in the context of Reinforcement Learning (RL). SAC-X enables learning of complex behaviors - from scratch - in the presence of multiple sparse reward signals. To this end, the agent is equipped with a set of general auxiliary tasks, that it attempts to learn simultaneously via off-policy RL. The key idea behind our method is that active (learned) scheduling and execution of auxiliary policies allows the agent to efficiently explore its environment - enabling it to excel at sparse reward RL. Our experiments in several challenging robotic manipulation settings demonstrate the power of our approach.

Citations

• Learning Continuous Control Policies by Stochastic Value Gradients
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Auto-Encoding Variational Bayes
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Deep Successor Reinforcement Learning
• Guided Policy Search as Approximate Mirror Descent
• Emergence of Locomotion Behaviours in Rich Environments
• Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World
• A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning

Cited by

• Learning When to Switch: Adaptive Policy Selection via Reinforcement Learning
• Reinforcement learning of quantum circuit architectures for molecular potential energy curves
• Learning Parameterized Skills from Demonstrations
• Generating Auxiliary Tasks with Reinforcement Learning
• D-LiFT: Improving LLM-based Decompiler Backend via Code Quality-driven Fine-tuning
• SparseMap: A Sparse Tensor Accelerator Framework Based on Evolution Strategy
• Reinforcement Learning with Action Chunking
• Reinforcement Learning via Implicit Imitation Guidance
• Task-agnostic Exploration in Reinforcement Learning
• Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement
• An Introduction to Deep Reinforcement Learning
• Learning agile soccer skills for a bipedal robot with deep reinforcement learning
• An empirical investigation of the challenges of real-world reinforcement learning
• DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning
• Towards VM Rescheduling Optimization Through Deep Reinforcement Learning
• Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environments
• Amanuensis: The Programmer's Apprentice
• Multi-Task Reinforcement Learning with Soft Modularization
• Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
• Discovery of Options via Meta-Learned Subgoals
• Learning and Planning with a Semantic Model
• Recursive Optimization of Finite Blocklength Allocation to Mitigate Age-of-Information Outage
• ClinicalGPT-R1: Pushing reasoning capability of generalist disease diagnosis with large language model

Related

• Dealing with Sparse Rewards in Reinforcement Learning
• Reinforcement Learning via Auxiliary Task Distillation
• Reinforcement Learning with Unsupervised Auxiliary Tasks
• Dense Dynamics-Aware Reward Synthesis: Integrating Prior Experience with Demonstrations
• Revisiting Sparse Rewards for Goal-Reaching Reinforcement Learning