Sim-to-Real Learning of All Common Bipedal Gaits via Periodic Reward Composition

2020/11/02 by Jonah Siekmann, Siekmann, Jonah, Yesh Godse +5 · 29 citations
Computer Science · Engineering · #FOS: Computer and information sciences #Muscle activation and electromyography studies #Reinforcement Learning in Robotics #Robotic Locomotion and Control #Robotics (cs.RO) #cs.RO

paper · pdf · doi:10.48550/arxiv.2011.01387

Accepted for presentation at ICRA 2021. The first two authors contributed equally to this work

openalex publication_date 2020/11/02 · arxiv created 2021/03/11 · arxiv updated 2021/03/12 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28

Abstract

We study the problem of realizing the full spectrum of bipedal locomotion on a real robot with sim-to-real reinforcement learning (RL). A key challenge of learning legged locomotion is describing different gaits, via reward functions, in a way that is intuitive for the designer and specific enough to reliably learn the gait across different initial random seeds or hyperparameters. A common approach is to use reference motions (e.g. trajectories of joint positions) to guide learning. However, finding high-quality reference motions can be difficult and the trajectories themselves narrowly constrain the space of learned motion. At the other extreme, reference-free reward functions are often underspecified (e.g. move forward) leading to massive variance in policy behavior, or are the product of significant reward-shaping via trial-and-error, making them exclusive to specific gaits. In this work, we propose a reward-specification framework based on composing simple probabilistic periodic costs on basic forces and velocities. We instantiate this framework to define a parametric reward function with intuitive settings for all common bipedal gaits - standing, walking, hopping, running, and skipping. Using this function we demonstrate successful sim-to-real transfer of the learned gaits to the bipedal robot Cassie, as well as a generic policy that can transition between all of the two-beat gaits.

Cited by

• PvP: Data-Efficient Humanoid Robot Learning with Proprioceptive-Privileged Contrastive Representations
• Learning Terrain Aware Bipedal Locomotion via Reduced Dimensional Perceptual Representations
• Toward Seamless Physical Human-Humanoid Interaction: Insights from Control, Intent, and Modeling with a Vision for What Comes Next
• Sim-to-Real Transfer in Deep Reinforcement Learning for Bipedal Locomotion
• Periodic Skill Discovery
• Heuristic Step Planning for Learning Dynamic Bipedal Locomotion: A Comparative Study of Model-Based and Model-Free Approaches
• NaviGait: Navigating Dynamically Feasible Gait Libraries using Deep Reinforcement Learning
• Towards Dynamic Quadrupedal Gaits: A Symmetry-Guided RL Hierarchy Enables Free Gait Transitions at Varying Speeds
• DexNDM: Closing the Reality Gap for Dexterous In-Hand Rotation via Joint-Wise Neural Dynamics Model
• LocoFormer: Generalist Locomotion via Long-context Adaptation
• RobotDancing: Residual-Action Reinforcement Learning Enables Robust Long-Horizon Humanoid Motion Tracking
• HuMam: Humanoid Motion Control via End-to-End Deep Reinforcement Learning with Mamba
• Contrastive Representation Learning for Robust Sim-to-Real Transfer of Adaptive Humanoid Locomotion
• Non-conflicting Energy Minimization in Reinforcement Learning based Robot Control
• MASH: Cooperative-Heterogeneous Multi-Agent Reinforcement Learning for Single Humanoid Robot Locomotion
• Achieving Precise and Reliable Locomotion with Differentiable Simulation-Based System Identification
• Optimizing Bipedal Locomotion for The 100m Dash With Comparison to Human Running
• In-between Motion Generation Based Multi-Style Quadruped Robot Locomotion
• A Gait Driven Reinforcement Learning Framework for Humanoid Robots
• Integrating Diffusion-based Multi-task Learning with Online Reinforcement Learning for Robust Quadruped Robot Control
• Evaluating Robots Like Human Infants: A Case Study of Learned Bipedal Locomotion
• Versatile Loco-Manipulation through Flexible Interlimb Coordination
• Booster Gym: An End-to-End Reinforcement Learning Framework for Humanoid Robot Locomotion
• Gait-Conditioned Reinforcement Learning with Multi-Phase Curriculum for Humanoid Locomotion
• Sampling-Based System Identification with Active Exploration for Legged Robot Sim2Real Learning
• Unleashing Humanoid Reaching Potential via Real-world-Ready Skill Space
• PPF: Pre-training and Preservative Fine-tuning of Humanoid Locomotion via Model-Assumption-based Regularization
• Spectral Normalization for Lipschitz-Constrained Policies on Learning Humanoid Locomotion
• Sim-to-Real of Humanoid Locomotion Policies via Joint Torque Space Perturbation Injection

Related

• Dynamic Bipedal Maneuvers through Sim-to-Real Reinforcement Learning
• Optimizing Bipedal Maneuvers of Single Rigid-Body Models for Reinforcement Learning
• Sim-to-Real Learning of Footstep-Constrained Bipedal Dynamic Walking
• Infer and Adapt: Bipedal Locomotion Reward Learning from Demonstrations via Inverse Reinforcement Learning
• Reinforcement Learning for Versatile, Dynamic, and Robust Bipedal Locomotion Control