Learning robust perceptive locomotion for quadrupedal robots in the wild

2022/01/19 by Takahiro Miki, Joonho Lee, Jemin Hwangbo +3 · 754 citations
Computer Science · Engineering · Psychology · #Artificial intelligence #Biomimetic flight and propulsion mechanisms #Computer science #Computer vision #Ecology #Gait #Human Pose and Action Recognition #Human–computer interaction #Legged robot #Perception #Physical medicine and rehabilitation #Psychology #Robot #Robotic Locomotion and Control #Robotics #Robustness (evolution) #Simulation #Terrain #cs.RO

paper · pdf · doi:10.1126/scirobotics.abk2822

published in Science Robotics 7(62), eabk2822 (American Association for the Advancement of Science)

openalex publication_date 2022/01/19 · arxiv created 2022/01/20 · arxiv updated 2022/01/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Legged robots that can operate autonomously in remote and hazardous environments will greatly increase opportunities for exploration into underexplored areas. Exteroceptive perception is crucial for fast and energy-efficient locomotion: Perceiving the terrain before making contact with it enables planning and adaptation of the gait ahead of time to maintain speed and stability. However, using exteroceptive perception robustly for locomotion has remained a grand challenge in robotics. Snow, vegetation, and water visually appear as obstacles on which the robot cannot step or are missing altogether due to high reflectance. In addition, depth perception can degrade due to difficult lighting, dust, fog, reflective or transparent surfaces, sensor occlusion, and more. For this reason, the most robust and general solutions to legged locomotion to date rely solely on proprioception. This severely limits locomotion speed because the robot has to physically feel out the terrain before adapting its gait accordingly. Here, we present a robust and general solution to integrating exteroceptive and proprioceptive perception for legged locomotion. We leverage an attention-based recurrent encoder that integrates proprioceptive and exteroceptive input. The encoder is trained end to end and learns to seamlessly combine the different perception modalities without resorting to heuristics. The result is a legged locomotion controller with high robustness and speed. The controller was tested in a variety of challenging natural and urban environments over multiple seasons and completed an hour-long hike in the Alps in the time recommended for human hikers.

Citations

• Long Short-Term Memory
• Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
• DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
• Gaze and the Control of Foot Placement When Walking in Natural Terrain
• Learning agile and dynamic motor skills for legged robots

Cited by

• PIN-WM: Learning Physics-INformed World Models for Non-Prehensile Manipulation
• Dynamic Legged Ball Manipulation on Rugged Terrains with Hierarchical Reinforcement Learning
• Crossing the Human-Robot Embodiment Gap with Sim-to-Real RL using One Human Demonstration
• RESPLE: Recursive Spline Estimation for LiDAR-Based Odometry
• Co-optimizing Physical Reconfiguration Parameters and Controllers for an Origami-inspired Reconfigurable Manipulator
• PPF: Pre-training and Preservative Fine-tuning of Humanoid Locomotion via Model-Assumption-based Regularization
• GenTe: Generative Real-world Terrains for General Legged Robot Locomotion Control
• Neural Fidelity Calibration for Informative Sim-to-Real Adaptation
• Collision avoidance from monocular vision trained with novel view synthesis
• RobustDexGrasp: Robust Dexterous Grasping of General Objects