Sample-efficient Cross-Entropy Method for Real-time Planning

2020/08/14 by Cristina Pinneri, Shambhuraj Sawant, Pinneri, Cristina +12 · 18 citations
Computer Science · Engineering · Mathematics · #Advanced Control Systems Optimization #Reinforcement Learning in Robotics #Robotic Path Planning Algorithms #cs.LG #cs.RO #stat.ML

paper · pdf · doi:10.48550/arxiv.2008.06389

arxiv created 2020/08/14 · arxiv updated 2020/08/17

Abstract

Trajectory optimizers for model-based reinforcement learning, such as the Cross-Entropy Method (CEM), can yield compelling results even in high-dimensional control tasks and sparse-reward environments. However, their sampling inefficiency prevents them from being used for real-time planning and control. We propose an improved version of the CEM algorithm for fast planning, with novel additions including temporally-correlated actions and memory, requiring 2.7-22x less samples and yielding a performance increase of 1.2-10x in high-dimensional control problems.

Cited by

• Action from Adjacent Set in Physical Space Outperforms the Best Prediction in World Models
• Temporal-Distance JEPA: Plan-Aware Representation Learning for Latent World Model Predictive Control
• Variational Neural Belief Parameterizations for Robust Dexterous Grasping under Multimodal Uncertainty
• SOMBRL: Scalable and Optimistic Model-Based RL
• Learning Soft Robotic Dynamics with Active Exploration
• Sample-efficient and Scalable Exploration in Continuous-Time RL
• Sample-Efficient and Smooth Cross-Entropy Method Model Predictive Control Using Deterministic Samples
• Empowering Multi-Robot Cooperation via Sequential World Models
• Learning Contact Dynamics through Touching: Action-conditional Graph Neural Networks for Robotic Peg Insertion
• Why and How Auxiliary Tasks Improve JEPA Representations
• Control of Legged Robots using Model Predictive Optimized Path Integral
• Beyond Constant Parameters: Hyper Prediction Models and HyperMPC
• Test-time Offline Reinforcement Learning on Goal-related Experience
• Continual Reinforcement Learning by Planning with Online World Models
• Differentiable Simulation of Hard Contacts with Soft Gradients for Learning and Control
• Bregman Centroid Guided Cross-Entropy Method
• Model Tensor Planning
• Smooth Operator: A Real-Time Sampling-Based Algorithm for Kinematic Hand Retargeting
• Learned Perceptive Forward Dynamics Model for Safe and Platform-aware Robotic Navigation

Related

• CEM-GD: Cross-Entropy Method with Gradient Descent Planner for Model-Based Reinforcement Learning
• Model-Predictive Control via Cross-Entropy and Gradient-Based Optimization
• Sample-efficient Real-time Planning with Curiosity Cross-Entropy Method and Contrastive Learning
• A Simple Decentralized Cross-Entropy Method
• Bregman Centroid Guided Cross-Entropy Method