Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs

2020/06/29 by Jianzhun Du, Joseph Futoma, Du, Jianzhun +3 · 1 citation
Computer Science · Engineering · #Data Stream Mining Techniques #Elevator Systems and Control #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2006.16210

openalex publication_date 2020/06/29 · openalex created_date 2020/07/02 · openalex updated_date 2026/07/28

Abstract

We present two elegant solutions for modeling continuous-time dynamics, in a novel model-based reinforcement learning (RL) framework for semi-Markov decision processes (SMDPs), using neural ordinary differential equations (ODEs). Our models accurately characterize continuous-time dynamics and enable us to develop high-performing policies using a small amount of data. We also develop a model-based approach for optimizing time schedules to reduce interaction rates with the environment while maintaining the near-optimal performance, which is not possible for model-free methods. We experimentally demonstrate the efficacy of our methods across various continuous-time domains.

Citations

• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Stochastic Backpropagation and Approximate Inference in Deep Generative\n Models
• Constrained model predictive control: Stability and optimality
• Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
• Sequence to Sequence Learning with Neural Networks
• Continuous control with deep reinforcement learning
• On Learning to Think: Algorithmic Information Theory for Novel\n Combinations of Reinforcement Learning Controllers and Recurrent Neural World\n Models
• Prioritized Experience Replay
• Recurrent Environment Simulators
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Model-Ensemble Trust-Region Policy Optimization
• Addressing Function Approximation Error in Actor-Critic Methods
• Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control
• Exploring Model-based Planning with Policy Networks
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Benchmarking Model-Based Reinforcement Learning
• HMSPC: A Hybrid Mechanistic-Stochastic Physical-Continuous Model for Battery Dynamics
• Model-Augmented Actor-Critic: Backpropagating through Paths
• MOReL : Model-Based Offline Reinforcement Learning
• MOPO: Model-based Offline Policy Optimization
• Learning Differential Equations that are Easy to Solve

Cited by

• SoC-DT: Standard-of-Care Aligned Digital Twins for Patient-Specific Tumor Dynamics

Related

• ODE-based Recurrent Model-free Reinforcement Learning for POMDPs
• Continuous-Time Model-Based Reinforcement Learning
• Sample-efficient and Scalable Exploration in Continuous-Time RL
• Efficient Exploration in Continuous-time Model-based Reinforcement Learning
• Model-Based Reinforcement Learning in Discrete-Action Non-Markovian Reward Decision Processes