Sim-to-Real Learning of All Common Bipedal Gaits via Periodic Reward\n Composition

2020/11/02 by Jonah Siekmann, Siekmann, Jonah, Yesh Godse +5 · 18 citations
Computer Science · Engineering · #FOS: Computer and information sciences #Muscle activation and electromyography studies #Reinforcement Learning in Robotics #Robotic Locomotion and Control #Robotics (cs.RO)

paper · pdf · doi:10.48550/arxiv.2011.01387

openalex publication_date 2020/11/02 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28

Abstract

We study the problem of realizing the full spectrum of bipedal locomotion on\na real robot with sim-to-real reinforcement learning (RL). A key challenge of\nlearning legged locomotion is describing different gaits, via reward functions,\nin a way that is intuitive for the designer and specific enough to reliably\nlearn the gait across different initial random seeds or hyperparameters. A\ncommon approach is to use reference motions (e.g. trajectories of joint\npositions) to guide learning. However, finding high-quality reference motions\ncan be difficult and the trajectories themselves narrowly constrain the space\nof learned motion. At the other extreme, reference-free reward functions are\noften underspecified (e.g. move forward) leading to massive variance in policy\nbehavior, or are the product of significant reward-shaping via trial-and-error,\nmaking them exclusive to specific gaits. In this work, we propose a\nreward-specification framework based on composing simple probabilistic periodic\ncosts on basic forces and velocities. We instantiate this framework to define a\nparametric reward function with intuitive settings for all common bipedal gaits\n- standing, walking, hopping, running, and skipping. Using this function we\ndemonstrate successful sim-to-real transfer of the learned gaits to the bipedal\nrobot Cassie, as well as a generic policy that can transition between all of\nthe two-beat gaits.\n

Cited by

• PvP: Data-Efficient Humanoid Robot Learning with Proprioceptive-Privileged Contrastive Representations
• Learning Terrain Aware Bipedal Locomotion via Reduced Dimensional Perceptual Representations
• Toward Seamless Physical Human-Humanoid Interaction: Insights from Control, Intent, and Modeling with a Vision for What Comes Next
• Sim-to-Real Transfer in Deep Reinforcement Learning for Bipedal Locomotion
• Periodic Skill Discovery
• Heuristic Step Planning for Learning Dynamic Bipedal Locomotion: A Comparative Study of Model-Based and Model-Free Approaches
• NaviGait: Navigating Dynamically Feasible Gait Libraries using Deep Reinforcement Learning
• Towards Dynamic Quadrupedal Gaits: A Symmetry-Guided RL Hierarchy Enables Free Gait Transitions at Varying Speeds
• DexNDM: Closing the Reality Gap for Dexterous In-Hand Rotation via Joint-Wise Neural Dynamics Model
• LocoFormer: Generalist Locomotion via Long-context Adaptation
• RobotDancing: Residual-Action Reinforcement Learning Enables Robust Long-Horizon Humanoid Motion Tracking
• HuMam: Humanoid Motion Control via End-to-End Deep Reinforcement Learning with Mamba
• Contrastive Representation Learning for Robust Sim-to-Real Transfer of Adaptive Humanoid Locomotion
• Non-conflicting Energy Minimization in Reinforcement Learning based Robot Control
• MASH: Cooperative-Heterogeneous Multi-Agent Reinforcement Learning for Single Humanoid Robot Locomotion
• Achieving Precise and Reliable Locomotion with Differentiable Simulation-Based System Identification
• Optimizing Bipedal Locomotion for The 100m Dash With Comparison to Human Running
• In-between Motion Generation Based Multi-Style Quadruped Robot Locomotion

Related

• Dynamic Bipedal Maneuvers through Sim-to-Real Reinforcement Learning
• Optimizing Bipedal Maneuvers of Single Rigid-Body Models for Reinforcement Learning
• Sim-to-Real Learning of Footstep-Constrained Bipedal Dynamic Walking
• Infer and Adapt: Bipedal Locomotion Reward Learning from Demonstrations via Inverse Reinforcement Learning
• Reinforcement Learning for Versatile, Dynamic, and Robust Bipedal Locomotion Control