Learning with Training Wheels: Speeding up Training with a Simple Controller for Deep Reinforcement Learning

2018/12/12 by Linhai Xie, Sen Wang, Xie, Linhai +7 · 1 citation
Computer Science · Engineering · #Adversarial Robustness in Machine Learning #Artificial intelligence #Computer science #Context (archaeology) #Control engineering #Controller (irrigation) #Engineering #FOS: Computer and information sciences #Imitation #Mobile robot #Obstacle #Obstacle avoidance #PID controller #Reinforcement Learning in Robotics #Reinforcement learning #Robot #Robot Manipulation and Learning #Robotics (cs.RO) #Simple (philosophy) #Training (meteorology) #cs.RO

paper · pdf · doi:10.48550/arxiv.1812.05027

published in arXiv (Cornell University) (Cornell University) · Published in ICRA2018. The code is now available at https://github.com/xie9187/AsDDPG

arxiv created 2018/12/12 · openalex publication_date 2018/12/12 · arxiv updated 2018/12/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06

Abstract

Deep Reinforcement Learning (DRL) has been applied successfully to many robotic applications. However, the large number of trials needed for training is a key issue. Most of existing techniques developed to improve training efficiency (e.g. imitation) target on general tasks rather than being tailored for robot applications, which have their specific context to benefit from. We propose a novel framework, Assisted Reinforcement Learning, where a classical controller (e.g. a PID controller) is used as an alternative, switchable policy to speed up training of DRL for local planning and navigation problems. The core idea is that the simple control law allows the robot to rapidly learn sensible primitives, like driving in a straight line, instead of random exploration. As the actor network becomes more advanced, it can then take over to perform more complex actions, like obstacle avoidance. Eventually, the simple controller can be discarded entirely. We show that not only does this technique train faster, it also is less sensitive to the structure of the DRL network and consistently outperforms a standard Deep Deterministic Policy Gradient network. We demonstrate the results in both simulation and real-world experiments.

Citations

• Continuous control with deep reinforcement learning
• Prioritized Experience Replay
• A Deep Hierarchical Approach to Lifelong Learning in Minecraft
• Deep Reinforcement Learning with Successor Features for Navigation across Similar Environments
• One-Shot Imitation Learning
• Obstacle Avoidance through Deep Networks based Intermediate Perception
• Learning to Fly by Crashing

Cited by

• Uncertainty-Based Smooth Policy Regularisation for Reinforcement Learning with Few Demonstrations

Related

• Training in Task Space to Speed Up and Guide Reinforcement Learning
• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• PathRL: An End-to-End Path Generation Method for Collision Avoidance via Deep Reinforcement Learning
• On Training Flexible Robots using Deep Reinforcement Learning
• Using Deep Reinforcement Learning for the Continuous Control of Robotic Arms