Learning to Fly -- a Gym Environment with PyBullet Physics for Reinforcement Learning of Multi-agent Quadcopter Control

2021/03/02 by Jacopo Panerati, Panerati, Jacopo, Hehui Zheng +10 · 19 citations
Computer Science · Engineering · #Adversarial Robustness in Machine Learning #Reinforcement Learning in Robotics #Traffic control and management #cs.LG #cs.RO

paper · pdf · doi:10.48550/arxiv.2103.02142

8 pages, 11 figures, accepted for presentation at IROS 2021

arxiv created 2021/07/25 · arxiv updated 2021/07/27

Abstract

Robotic simulators are crucial for academic research and education as well as the development of safety-critical applications. Reinforcement learning environments -- simple simulations coupled with a problem specification in the form of a reward function -- are also important to standardize the development (and benchmarking) of learning algorithms. Yet, full-scale simulators typically lack portability and parallelizability. Vice versa, many reinforcement learning environments trade-off realism for high sample throughputs in toy-like problems. While public data sets have greatly benefited deep learning and computer vision, we still lack the software tools to simultaneously develop -- and fairly compare -- control theory and reinforcement learning approaches. In this paper, we propose an open-source OpenAI Gym-like environment for multiple quadcopters based on the Bullet physics engine. Its multi-agent and vision based reinforcement learning interfaces, as well as the support of realistic collisions and aerodynamic effects, make it, to the best of our knowledge, a first of its kind. We demonstrate its use through several examples, either for control (trajectory tracking with PID control, multi-robot flight with downwash, etc.) or reinforcement learning (single and multi-agent stabilization tasks), hoping to inspire future research that combines control theory and machine learning.

Cited by

• Dynamic Entropy Tuning in Reinforcement Learning Low-Level Quadcopter Control: Stochasticity vs Determinism
• Sim2Swim: Zero-Shot Velocity Control for Agile AUV Maneuvering in 3 Minutes
• Embodied Co-Design for Rapidly Evolving Agents: Taxonomy, Frontiers, and Challenges
• MagBotSim: Physics-Based Simulation and Reinforcement Learning Environments for Magnetic Robotics
• GO-Flock: Goal-Oriented Flocking in 3D Unknown Environments with Depth Maps
• A Simulation Evaluation Suite for Robust Adaptive Quadcopter Control
• ActivePose: Active 6D Object Pose Estimation and Tracking for Robotic Manipulation
• DiffAero: A GPU-Accelerated Differentiable Simulation Framework for Efficient Quadrotor Policy Learning
• DMPC-Swarm: Distributed Model Predictive Control on Nano UAV Swarms
• SwarmVLM: VLM-Guided Impedance Control for Autonomous Navigation of Heterogeneous Robots in Dynamic Warehousing
• Learning Agile Tensile Perching for Aerial Robots from Demonstrations
• ARMOR: Robust Reinforcement Learning-based Control for UAVs under Physical Attacks
• Dynamics and Control of Vision-Aided Multi-UAV-tethered Netted System Capturing Non-Cooperative Target
• Reinforcement Learning-based Fault-Tolerant Control for Quadrotor with Online Transformer Adaptation
• RoboVerse: Towards a Unified Platform, Dataset and Benchmark for Scalable and Generalizable Robot Learning
• A General Infrastructure and Workflow for Quadrotor Deep Reinforcement Learning and Reality Deployment
• Visual Grounding in Zero-Shot Vision-Language Control
• Sim-to-Real Transfer in Reinforcement Learning for Maneuver Control of a Variable-Pitch MAV
• Survey of Simulators for Aerial Robots: An Overview and In-Depth Systematic Comparisons [Survey]

Related

• Parallel Reinforcement Learning Simulation for Visual Quadrotor Navigation
• PyFlyt -- UAV Simulation Environments for Reinforcement Learning Research
• A Deep Reinforcement Learning Strategy for UAV Autonomous Landing on a Platform
• Multitask Reinforcement Learning for Quadcopter Attitude Stabilization and Tracking using Graph Policy
• Deep Reinforcement Learning-based Quadcopter Controller: A Practical Approach and Experiments