Challenges of Real-World Reinforcement Learning

2019/04/29 by Gabriel Dulac-Arnold, Daniel Mankowitz, Dulac-Arnold, Gabriel +4 · 1 voice · 256 citations
Computer Science · Decision Sciences · Mathematics · #Advanced Bandit Algorithms Research #Artificial Intelligence (cs.AI) #Artificial intelligence #Computer science #Domain (mathematical analysis) #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #Leverage (statistics) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Mathematics #Programming language #Reinforcement Learning in Robotics #Reinforcement learning #Robotics (cs.RO) #Set (abstract data type) #Testbed #World Wide Web #cs.AI #cs.LG #cs.RO #stat.ML

paper · pdf · doi:10.48550/arxiv.1904.12901

published in arXiv (Cornell University) (Cornell University)

arxiv created 2019/04/29 · openalex publication_date 2019/04/29 · arxiv published 2019/04/29 · arxiv updated 2019/05/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Reinforcement learning (RL) has proven its worth in a series of artificial domains, and is beginning to show some successes in real-world scenarios. However, much of the research advances in RL are often hard to leverage in real-world systems due to a series of assumptions that are rarely satisfied in practice. We present a set of nine unique challenges that must be addressed to productionize RL to real world problems. For each of these challenges, we specify the exact meaning of the challenge, present some approaches from the literature, and specify some metrics for evaluating that challenge. An approach that addresses all nine challenges would be applicable to a large number of real world problems. We also present an example domain that has been modified to present these challenges as a testbed for practical RL research.

Citations

• Constrained Markov Decision Processes
• Doubly Robust Policy Evaluation and Learning
• Robust Dynamic Programming
• Policy Gradient for Coherent Risk Measures
• Continuous control with deep reinforcement learning
• Methods for Interpreting and Understanding Deep Neural Networks
• DeepMind Control Suite
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Reward Constrained Policy Optimization
• Learning Latent Dynamics for Planning from Pixels
• A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
• Learning dexterous in-hand manipulation

Cited by

• Trustworthy and Explainable Deep Reinforcement Learning for Safe and Energy-Efficient Process Control: A Use Case in Industrial Compressed Air Systems
• Reinforcement Learning in Financial Decision Making: A Systematic Review of Performance, Challenges, and Implementation Strategies
• Latent Action World Models for Control with Unlabeled Trajectories
• RoboScape-R: Unified Reward-Observation World Models for Generalizable Robotics Training via RL
• Shielded Controller Units for RL with Operational Constraints Applied to Remote Microgrids
• Harnessing Bounded-Support Evolution Strategies for Policy Refinement
• PrefPoE: Advantage-Guided Preference Fusion for Learning Where to Explore
• Tensor-Efficient High-Dimensional Q-learning
• A Step Toward World Models: A Survey on Robotic Manipulation
• RESample: A Robust Data Augmentation Framework via Exploratory Sampling for Robotic Manipulation
• RM-RL: Role-Model Reinforcement Learning for Precise Robot Manipulation
• Don't Run with Scissors: Pruning Breaks VLA Models but They Can Be Recovered
• Latent Activation Editing: Inference-Time Refinement of Learned Policies for Safer Multirobot Navigation
• CAQL: Continuous Action Q-Learning
• Acting in Delayed Environments with Non-Stationary Markov Policies
• Evaluation-Aware Reinforcement Learning
• Real-Time Reinforcement Learning for Dynamic Tasks with a Parallel Soft Robot
• Robust Reinforcement Learning for Continuous Control with Model Misspecification
• Mutual Information Tracks Policy Coherence in Reinforcement Learning
• Self-Supervised Goal-Reaching Results in Multi-Agent Cooperation and Exploration
• Robust Multi-Modal Policies for Industrial Assembly via Reinforcement Learning and Demonstrations: A Large-Scale Study
• Learning hierarchical relationships for object-goal navigation
• RT-HCP: Dealing with Inference Delays and Sample Efficiency to Learn Directly on Robotic Platforms
• Critic Regularized Regression
• Diverse Auto-Curriculum is Critical for Successful Real-World Multiagent Learning Systems
• QPLEX: Duplex Dueling Multi-Agent Q-Learning
• Integrating Reinforcement Learning with Visual Generative Models: Foundations and Advances
• RecSim: A Configurable Simulation Platform for Recommender Systems
• Quick Learner Automated Vehicle Adapting its Roadmanship to Varying Traffic Cultures with Meta Reinforcement Learning
• Average-Reward Off-Policy Policy Evaluation with Function Approximation
• Deep reinforcement learning for inventory control: A roadmap
• Truncated Emphatic Temporal Difference Methods for Prediction and Control
• Challenges of Applying Deep Reinforcement Learning in Dynamic Dispatching
• The Benchmark Lottery
• Self-Imitation Advantage Learning
• Can Reinforcement Learning for Continuous Control Generalize Across Physics Engines?
• Policy-Based Trajectory Clustering in Offline Reinforcement Learning
• Detecting Proxy Gaming in RL and LLM Alignment via Evaluator Stress Tests
• Energy and Thermal-aware Resource Management of Cloud Data Centres: A Taxonomy and Future Directions
• Breaking the Deadly Triad with a Target Network
• Improving Learning from Demonstrations by Learning from Experience
• Probabilistic Active Meta-Learning
• Learning Task Belief Similarity with Latent Dynamics for Meta-Reinforcement Learning
• Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environment
• Challenges in Deploying Machine Learning: a Survey of Case Studies
• VRAIL: Vectorized Reward-based Attribution for Interpretable Learning
• How Do You Act? An Empirical Study to Understand Behavior of Deep Reinforcement Learning Agents
• Federated Neuroevolution O-RAN: Enhancing the Robustness of Deep Reinforcement Learning xApps
• Continuous Doubly Constrained Batch Reinforcement Learning
• Aligning an optical interferometer with beam divergence control and continuous action space
• Risk-Averse Offline Reinforcement Learning
• Learning robust driving policies without online exploration
• Reinforcement Learning with Combinatorial Actions: An Application to Vehicle Routing
• ACNMP: Skill Transfer and Task Extrapolation through Learning from Demonstration and Reinforcement Learning via Representation Sharing
• WorldGym: World Model as An Environment for Policy Evaluation
• An empirical investigation of the challenges of real-world reinforcement learning
• Balancing Constraints and Rewards with Meta-Gradient D4PG
• A Snapshot of Influence: A Local Data Attribution Framework for Online Reinforcement Learning
• Boosting Offline Reinforcement Learning with Residual Generative Modeling
• Multi-parameter Control for the (1+(λ,λ))-GA on OneMax via Deep Reinforcement Learning
• RecSim NG: Toward Principled Uncertainty Modeling for Recommender Ecosystems
• AI’s Role in Fighting Corruption
• Reducing Conservativeness Oriented Offline Reinforcement Learning
• I'm sorry Dave, I'm afraid I can't do that, Deep Q-learning from forbidden action
• Integration of Multi-Mode Preference into Home Energy Management System Using Deep Reinforcement Learning
• Designing Agent-Ready Websites for AI Web Agents: A Framework for Machine Readability, Actionability, and Decision Reliability
• PPMC RL Training Algorithm: Rough Terrain Intelligent Robots through Reinforcement Learning
• Evolution Strategies for Deep RL pretraining
• RoboVerse: Towards a Unified Platform, Dataset and Benchmark for Scalable and Generalizable Robot Learning
• The Big Three: A Methodology to Increase Data Science ROI by Answering the Questions Companies Care About
• Dynamic Contrastive Skill Learning with State-Transition Based Skill Clustering and Dynamic Length Adjustment
• A Survey of Reinforcement Learning Algorithms for Dynamically Varying Environments
• ProDVI: Programmatic Dynamics Priors for Value Network Initialization

Discussions

• Challenges of Real-World Reinforcement Learning [hn, 1 points, 0 comments]

Related

• An empirical investigation of the challenges of real-world reinforcement learning
• Gym4ReaL: A Suite for Benchmarking Real-World Reinforcement Learning
• A Comprehensive Survey of Reinforcement Learning: From Algorithms to Practical Challenges
• Reinforcement Learning in Practice: Opportunities and Challenges
• Evaluating the progress of Deep Reinforcement Learning in the real world: aligning domain-agnostic and domain-specific research