Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning
2021/08/23 by Viktor Makoviychuk, Makoviychuk, Viktor, Lukasz Wawrzyniak +19 · 193 citations
Computer Science · #Advanced Neural Network Applications #Distributed and Parallel Computing Systems #FOS: Computer and information sciences #Machine Learning (cs.LG) #Parallel Computing and Optimization Techniques #Reinforcement Learning in Robotics #Robotics (cs.RO)
paper · pdf · doi:10.48550/arxiv.2108.10470
openalex publication_date 2021/08/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Abstract
Isaac Gym offers a high performance learning platform to train policies for wide variety of robotics tasks directly on GPU. Both physics simulation and the neural network policy training reside on GPU and communicate by directly passing data from physics buffers to PyTorch tensors without ever going through any CPU bottlenecks. This leads to blazing fast training times for complex robotics tasks on a single GPU with 2-3 orders of magnitude improvements compared to conventional RL training that uses a CPU based simulator and GPU for neural networks. We host the results and videos at \urlhttps://sites.google.com/view/isaacgym-nvidia and isaac gym can be downloaded at \urlhttps://developer.nvidia.com/isaac-gym.
Cited by
- Envision: Embodied Visual Planning via Goal-Imagery Video Diffusion
- TongSIM: A General Platform for Simulating Intelligent Machines
- Learning Generalizable Hand-Object Tracking from Synthetic Demonstrations
- EGM: Efficiently Learning General Motion Tracking Policy for High Dynamic Humanoid Whole-Body Control
- Learning to Plan, Planning to Learn: Adaptive Hierarchical RL-MPC for Sample-Efficient Decision Making
- Lang2Manip: A Tool for LLM-Based Symbolic-to-Geometric Planning for Manipulation
- CRISP: Contact-Guided Real2Sim from Monocular Video with Planar Scene Primitives
- MoLingo: Motion-Language Alignment for Text-to-Human Motion Generation
- Universal Dexterous Functional Grasping via Demonstration-Editing Reinforcement Learning
- START: Traversing Sparse Footholds with Terrain Reconstruction
- PvP: Data-Efficient Humanoid Robot Learning with Proprioceptive-Privileged Contrastive Representations
- UniBYD: A Unified Framework for Learning Robotic Manipulation Across Embodiments Beyond Imitation of Human Demonstrations
- Cross-Entropy Optimization of Physically Grounded Task and Motion Plans
- Visionary: The World Model Carrier Built on WebGPU-Powered Gaussian Splatting Platform
- Entropy-Controlled Intrinsic Motivation Reinforcement Learning for Quadruped Robot Locomotion in Complex Terrains
- Embodied Co-Design for Rapidly Evolving Agents: Taxonomy, Frontiers, and Challenges
- SMP: Reusable Score-Matching Motion Priors for Physics-Based Character Control
- Diffusion-Augmented Markov Decision Processes for Maximum Entropy Reinforcement Learning
- Learning Dexterous Manipulation Skills from Imperfect Simulations
- Learning Sim-to-Real Humanoid Locomotion in 15 Minutes
- SPARK: Sim-ready Part-level Articulated Reconstruction with VLM Knowledge
- Discovering Self-Protective Falling Policy for Humanoid Robot via Deep Reinforcement Learning
- H-Zero: Cross-Humanoid Locomotion Pretraining Enables Few-shot Novel Embodiment Transfer
- Beyond Topology: A Morphological Symmetry Graph Representation for Locomotion Policy Learning
- RealAppliance: Let High-fidelity Appliance Assets Controllable and Workable as Aligned Real Manuals
- Physics-Informed Spiking Neural Networks via Conservative Flux Quantization
- Staggered Environment Resets Improve Massively Parallel On-Policy Reinforcement Learning
- BRIC: Bridging Kinematic Plans and Physical Control at Test Time
- HAFO: A Force-Adaptive Control Framework for Humanoid Robots in Intense Interaction Environments
- Collaborate sim and real: Robot Bin Packing Learning in Real-world and Physical Engine
- Discover, Learn, and Reinforce: Scaling Vision-Language-Action Pretraining with Diverse RL-Generated Trajectories
- ArtiWorld: LLM-Driven Articulation of 3D Objects in Scenes
- Agility Meets Stability: Versatile Humanoid Control with Heterogeneous Data
- MagBotSim: Physics-Based Simulation and Reinforcement Learning Environments for Magnetic Robotics
- DiffuDepGrasp: Diffusion-based Depth Noise Modeling Empowers Sim2Real Robotic Grasping
- VIRAL: Visual Sim-to-Real at Scale for Humanoid Loco-Manipulation
- Extending Test-Time Scaling: A 3D Perspective with Context, Batch, and Turn
- Force-Aware 3D Contact Modeling for Stable Grasp Generation
- Humanoid Whole-Body Badminton via Multi-Stage Reinforcement Learning
- Unveiling the Impact of Data and Model Scaling on High-Level Control for Humanoid Robots
- EquiMus: Energy-Equivalent Dynamic Modeling and Simulation of Musculoskeletal Robots Driven by Linear Elastic Actuators
- Contact Map Transfer with Conditional Diffusion Model for Generalizable Dexterous Grasp Generation
- Unified Humanoid Fall-Safety Policy from a Few Demonstrations
- Rapidly Learning Soft Robot Control via Implicit Time-Stepping
- Sim-to-Real Transfer in Deep Reinforcement Learning for Bipedal Locomotion
- Towards Adaptive Humanoid Control via Multi-Behavior Distillation and Reinforced Fine-Tuning
- Adversarial Game-Theoretic Algorithm for Dexterous Grasp Synthesis
- Decomposed Object Manipulation via Dual-Actor Policy
- ReGen: Generative Robot Simulation via Inverse Design
- Temporal Action Selection for Action Chunking
- LEGO-Eval: Towards Fine-Grained Evaluation on Synthesizing 3D Embodied Environments with Tool Augmentation
- Heuristic Adaptation of Potentially Misspecified Domain Support for Likelihood-Free Inference in Stochastic Dynamical Systems
- Towards Reinforcement Learning Based Log Loading Automation
- Embracing Evolution: A Call for Body-Control Co-Design in Embodied Humanoid Robot
- To Distill or Decide? Understanding the Algorithmic Trade-off in Partially Observable Reinforcement Learning
- From Passive Video to Editable Experience: Physically Grounded Experience Synthesis for Embodied Intelligence
- Nautilus: From One Prompt to Plug-and-Play Robot Learning
- BuildArena: A Physics-Aligned Interactive Benchmark of LLMs for Engineering Construction
- VOCALoco: Viability-Optimized Cost-aware Adaptive Locomotion
- Endowing GPT-4 with a Humanoid Body: Building the Bridge Between Off-the-Shelf VLMs and the Physical World
- DeGrip: A Compact Cable-driven Robotic Gripper for Desktop Disassembly
- FlowCritic: Bridging Value Estimation with Flow Matching in Reinforcement Learning
- Real-Time Gait Adaptation for Quadrupeds using Model Predictive Control and Reinforcement Learning
- The Reality Gap in Robotics: Challenges, Solutions, and Best Practices
- Seed3D 1.0: From Images to High-Fidelity Simulation-Ready 3D Assets
- OffSim: Offline Simulator for Model-based Offline Inverse Reinforcement Learning
- GRASPLAT: Enabling dexterous grasping through novel view synthesis
- Efficient Model-Based Reinforcement Learning for Robot Control via Online Optimization
- SoftMimic: Learning Compliant Whole-body Control from Examples
- Closing the Sim2Real Performance Gap in RL
- GaussGym: An open-source real-to-sim framework for learning locomotion from pixels
- A Comprehensive Survey on World Models for Embodied AI
- DexCanvas: Bridging Human Demonstrations and Robot Learning for Dexterous Manipulation
- VT-Refine: Learning Bimanual Assembly with Visuo-Tactile Feedback via Simulation Fine-Tuning
- PhysHMR: Learning Humanoid Control Policies from Vision for Physically Plausible Human Motion Reconstruction
- Restoring Noisy Demonstration for Imitation Learning With Diffusion Models
- Towards Adaptable Humanoid Control via Adaptive Motion Tracking
- MimicKit: A Reinforcement Learning Framework for Motion Imitation and Control
- Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agents
- Rethinking the Simulation vs. Rendering Dichotomy: No Free Lunch in Spatial World Modelling
- InternVLA-M1: A Spatially Guided Vision-Language-Action Framework for Generalist Robot Policy
- Autonomous Legged Mobile Manipulation for Lunar Surface Operations via Constrained Reinforcement Learning
- Fast Visuomotor Policy for Robotic Manipulation
- SCOOP'D: Learning Mixed-Liquid-Solid Scooping via Sim2Real Generative Policy
- DemoHLM: From One Demonstration to Generalizable Humanoid Loco-Manipulation
- PhysHSI: Towards a Real-World Generalizable and Natural Humanoid-Scene Interaction System
- Preference-Conditioned Multi-Objective RL for Integrated Command Tracking and Force Compliance in Humanoid Locomotion
- Population-Coded Spiking Neural Networks for High-Dimensional Robotic Control
- Towards Dynamic Quadrupedal Gaits: A Symmetry-Guided RL Hierarchy Enables Free Gait Transitions at Varying Speeds
- It Takes Two: Learning Interactive Whole-Body Control Between Humanoid Robots
- PolySim: Bridging the Sim-to-Real Gap for Humanoid Control via Multi-Simulator Dynamics Randomization
- Model-Based Lookahead Reinforcement Learning for in-hand manipulation
- DexNDM: Closing the Reality Gap for Dexterous In-Hand Rotation via Joint-Wise Neural Dynamics Model
- Towards Proprioception-Aware Embodied Planning for Dual-Arm Humanoid Robots
- DexMan: Learning Bimanual Dexterous Manipulation from Human and Generated Videos
- AVO: Amortized Value Optimization for Contact Mode Switching in Multi-Finger Manipulation
- DPL: Depth-only Perceptive Humanoid Locomotion via Realistic Depth Synthesis and Cross-Attention Terrain Reconstruction
- Diffusing Trajectory Optimization Problems for Recovery During Multi-Finger Manipulation
- Reference Grounded Skill Discovery
- Octax: Accelerated CHIP-8 Arcade Environments for Reinforcement Learning in JAX
- Performance-guided Task-specific Optimization for Multirotor Design
- Reliable and Scalable Robot Policy Evaluation with Imperfect Simulators
- Flexible Locomotion Learning with Diffusion Model Predictive Control
- Vision-Guided Quadrupedal Locomotion in the Wild with Multi-Modal Delay Randomization
- The Trajectory Bundle Method: Unifying Sequential-Convex Programming and Sampling-Based Trajectory Optimization
- Evolutionary Continuous Adaptive RL-Powered Co-Design for Humanoid Chin-Up Performance
- Best of Sim and Real: Decoupled Visuomotor Manipulation via Learning Control in Simulation and Perception in Real
- ISyHand: A Dexterous Multi-finger Robot Hand with an Articulated Palm
- JuggleRL: Mastering Ball Juggling with a Quadrotor via Deep Reinforcement Learning
- Unifying Agent Interaction and World Information for Multi-agent Coordination
- GES-UniGrasp: A Two-Stage Dexterous Grasping Strategy With Geometry-Based Expert Selection
- SAC-Loco: Safe and Adjustable Compliant Quadrupedal Locomotion
- Agile perceptive multiskill locomotion for quadrupedal robots in the wild
- Learning to Ball: Composing Policies for Long-Horizon Basketball Moves
- RoboView-Bias: Benchmarking Visual Bias in Embodied Agents for Robotic Manipulation
- DHAGrasp: Synthesizing Affordance-Aware Dual-Hand Grasps with Text Instructions
- DemoGrasp: Universal Dexterous Grasping from a Single Demonstration
- DAGDiff: Guiding Dual-Arm Grasp Diffusion to Stable and Collision-Free Grasps
- MARG: MAstering Risky Gap Terrains for Legged Robots with Elevation Mapping
- D3Grasp: Diverse and Deformable Dexterous Grasping for General Objects
- RoMoCo: Robotic Motion Control Toolbox for Reduced-Order Model-Based Locomotion on Bipedal and Humanoid Robots
- Pure Vision Language Action (VLA) Models: A Comprehensive Survey
- SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer
- Do You Need Proprioceptive States in Visuomotor Policies?
- BiGraspFormer: End-to-End Bimanual Grasp Transformer
- Query-Centric Diffusion Policy for Generalizable Robotic Assembly
- Learning Geometry-Aware Nonprehensile Pushing and Pulling with Dexterous Hands
- RoboManipBaselines: A Unified Framework for Imitation Learning in Robotic Manipulation across Real and Simulated Environments
- KungfuBot2: Learning Versatile Motion Skills for Humanoid Whole-Body Control
- LodeStar: Long-horizon Dexterity via Synthetic Data Augmentation from Human Demonstrations
- End-to-end RL Improves Dexterous Grasping Policies
- ExT: Towards Scalable Autonomous Excavation via Large-Scale Multi-Task Pretraining and Fine-Tuning
- Variational Shape Inference for Grasp Diffusion on SE(3)
- The Role of Touch: Towards Optimal Tactile Sensing Distribution in Anthropomorphic Hands for Dexterous In-Hand Manipulation
- Dynamic Adaptive Legged Locomotion Policy via Decoupling Reaction Force Control and Gait Control
- Behavior Foundation Model for Humanoid Robots
- Contrastive Representation Learning for Robust Sim-to-Real Transfer of Adaptive Humanoid Locomotion
- Gen2Real: Towards Demo-Free Dexterous Manipulation by Harnessing Generated Video
- Robust Online Residual Refinement via Koopman-Guided Dynamics Modeling
- Integrating Trajectory Optimization and Reinforcement Learning for Quadrupedal Jumping with Terrain-Adaptive Landing
- Learning to Generate Pointing Gestures in Situated Embodied Conversational Agents
- Geometric Red-Teaming for Robotic Manipulation
- FR-Net: Learning Robust Quadrupedal Fall Recovery on Challenging Terrains through Mass-Contact Prediction
- DiffAero: A GPU-Accelerated Differentiable Simulation Framework for Efficient Quadrotor Policy Learning
- Dexplore: Scalable Neural Control for Dexterous Manipulation from Reference-Scoped Exploration
- InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation
- Learning to Walk with Less: a Dyna-Style Approach to Quadrupedal Locomotion
- Grasp-MPC: Closed-Loop Visual Grasping via Value-Guided Model Predictive Control
- Learning to Walk in Costume: Adversarial Motion Priors for Aesthetically Constrained Humanoids
- Solving Robotics Tasks with Prior Demonstration via Exploration-Efficient Deep Reinforcement Learning
- CTBC: Contact-Triggered Blind Climbing for Wheeled Bipedal Robots with Instruction Learning and Reinforcement Learning
- Acrobotics: A Generalist Approach to Quadrupedal Robots' Parkour
- Fail2Progress: Learning from Real-World Robot Failures with Stein Variational Inference
- Disentangled Multi-Context Meta-Learning: Unlocking robust and Generalized Task Learning
- First Order Model-Based RL through Decoupled Backpropagation
- Poke and Strike: Learning Task-Informed Exploration Policies
- Deep Sensorimotor Control by Imitating Predictive Models of Human Motion
- Neural Robot Dynamics
- Virtual Community: An Open World for Humans, Robots, and Society
- Sim-to-Real Dynamic Object Manipulation on Conveyor Systems via Optimization Path Shaping
- Simultaneous Contact Sequence and Patch Planning for Dynamic Locomotion
- Improving Pre-Trained Vision-Language-Action Policies with Model-Based Search
- Fully Spiking Actor-Critic Neural Network for Robotic Manipulation
- Control of Legged Robots using Model Predictive Optimized Path Integral
- MASH: Cooperative-Heterogeneous Multi-Agent Reinforcement Learning for Single Humanoid Robot Locomotion
- Towards Affordance-Aware Robotic Dexterous Grasping with Human-like Priors
- Multimodal Spiking Neural Network for Space Robotic Manipulation
- Whole-Body Coordination for Dynamic Object Grasping with Legged Manipulators
- ADPro: a Test-time Adaptive Diffusion Policy via Manifold-constrained Denoising and Task-aware Initialization for Robotic Manipulation
- REBot: Reflexive Evasion Robot for Instantaneous Dynamic Obstacle Avoidance
- Reparameterization Proximal Policy Optimization
- Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulation
- Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategies
- Safety-Aware Imitation Learning via MPC-Guided Disturbance Injection
- GACL: Grounded Adaptive Curriculum Learning with Active Task and Performance Monitoring
- Hand-Eye Autonomous Delivery: Learning Humanoid Navigation, Locomotion and Reaching
- Towards Immersive Human-X Interaction: A Real-Time Framework for Physically Plausible Motion Synthesis
- DexReMoE:In-hand Reorientation of General Object via Mixtures of Experts
- Coordinated Humanoid Robot Locomotion with Symmetry Equivariant Reinforcement Learning Policy
- BarlowWalk: Self-supervised Representation Learning for Legged Robot Terrain-adaptive Locomotion
- Learning to Drift with Individual Wheel Drive: Maneuvering Autonomous Vehicle at the Handling Limits
- Benchmarking Massively Parallelized Multi-Task Reinforcement Learning for Robotics Tasks
- Viser: Imperative, Web-based 3D Visualization in Python
- A Two-Stage Lightweight Framework for Efficient Land-Air Bimodal Robot Autonomous Navigation
- Improving Generalization Ability of Robotic Imitation Learning by Resolving Causal Confusion in Observations
- FLORES: A Reconfigured Wheel-Legged Robot for Enhanced Steering and Adaptability
- Flow Matching Policy Gradients
- Reconstructing 4D Spatial Intelligence: A Survey
- Bipedalism for Quadrupedal Robots: Versatile Loco-Manipulation through Risk-Adaptive Reinforcement Learning
- RESCUE: Crowd Evacuation Simulation via Controlling SDM-United Characters
- Extending Group Relative Policy Optimization to Continuous Control: A Theoretical Framework for Robotic Reinforcement Learning
- Towards Scalable Spatial Intelligence via 2D-to-3D Data Lifting
- Adaptive Articulated Object Manipulation On The Fly with Foundation Model Reasoning and Part Grounding
Related