Challenges of Real-World Reinforcement Learning

2019/04/29 by Gabriel Dulac-Arnold, Daniel J. Mankowitz, Dulac-Arnold, Gabriel +3 · 29 citations
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #Artificial Intelligence (cs.AI) #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics #Robotics (cs.RO)

paper · pdf · doi:10.48550/arxiv.1904.12901

openalex publication_date 2019/04/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Reinforcement learning (RL) has proven its worth in a series of artificial domains, and is beginning to show some successes in real-world scenarios. However, much of the research advances in RL are often hard to leverage in real-world systems due to a series of assumptions that are rarely satisfied in practice. We present a set of nine unique challenges that must be addressed to productionize RL to real world problems. For each of these challenges, we specify the exact meaning of the challenge, present some approaches from the literature, and specify some metrics for evaluating that challenge. An approach that addresses all nine challenges would be applicable to a large number of real world problems. We also present an example domain that has been modified to present these challenges as a testbed for practical RL research.

Citations

• Constrained Markov Decision Processes
• Doubly Robust Policy Evaluation and Learning
• Robust Dynamic Programming
• Policy Gradient for Coherent Risk Measures
• Continuous control with deep reinforcement learning
• Methods for Interpreting and Understanding Deep Neural Networks
• DeepMind Control Suite
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Reward Constrained Policy Optimization
• Learning Latent Dynamics for Planning from Pixels
• A Reduction of Imitation Learning and Structured Prediction to No-Regret\n Online Learning
• Learning dexterous in-hand manipulation

Cited by

• Trustworthy and Explainable Deep Reinforcement Learning for Safe and Energy-Efficient Process Control: A Use Case in Industrial Compressed Air Systems
• Reinforcement Learning in Financial Decision Making: A Systematic Review of Performance, Challenges, and Implementation Strategies
• Latent Action World Models for Control with Unlabeled Trajectories
• RoboScape-R: Unified Reward-Observation World Models for Generalizable Robotics Training via RL
• Shielded Controller Units for RL with Operational Constraints Applied to Remote Microgrids
• Harnessing Bounded-Support Evolution Strategies for Policy Refinement
• PrefPoE: Advantage-Guided Preference Fusion for Learning Where to Explore
• Tensor-Efficient High-Dimensional Q-learning
• A Step Toward World Models: A Survey on Robotic Manipulation
• RESample: A Robust Data Augmentation Framework via Exploratory Sampling for Robotic Manipulation
• RM-RL: Role-Model Reinforcement Learning for Precise Robot Manipulation
• Don't Run with Scissors: Pruning Breaks VLA Models but They Can Be Recovered
• Latent Activation Editing: Inference-Time Refinement of Learned Policies for Safer Multirobot Navigation
• CAQL: Continuous Action Q-Learning
• Acting in Delayed Environments with Non-Stationary Markov Policies
• Evaluation-Aware Reinforcement Learning
• Real-Time Reinforcement Learning for Dynamic Tasks with a Parallel Soft Robot
• Robust Reinforcement Learning for Continuous Control with Model Misspecification
• Mutual Information Tracks Policy Coherence in Reinforcement Learning
• Self-Supervised Goal-Reaching Results in Multi-Agent Cooperation and Exploration
• Robust Multi-Modal Policies for Industrial Assembly via Reinforcement Learning and Demonstrations: A Large-Scale Study
• Learning hierarchical relationships for object-goal navigation
• RT-HCP: Dealing with Inference Delays and Sample Efficiency to Learn Directly on Robotic Platforms
• Critic Regularized Regression
• Diverse Auto-Curriculum is Critical for Successful Real-World Multiagent\n Learning Systems
• QPLEX: Duplex Dueling Multi-Agent Q-Learning
• Integrating Reinforcement Learning with Visual Generative Models: Foundations and Advances
• RecSim: A Configurable Simulation Platform for Recommender Systems
• Quick Learner Automated Vehicle Adapting its Roadmanship to Varying Traffic Cultures with Meta Reinforcement Learning

Related

• An empirical investigation of the challenges of real-world reinforcement learning
• Gym4ReaL: A Suite for Benchmarking Real-World Reinforcement Learning
• A Comprehensive Survey of Reinforcement Learning: From Algorithms to Practical Challenges
• Reinforcement Learning in Practice: Opportunities and Challenges
• Evaluating the progress of Deep Reinforcement Learning in the real world: aligning domain-agnostic and domain-specific research