Operator Splitting Value Iteration

2022/11/25 by Amin Rakhsha, Andrew Wang, Rakhsha, Amin +5 · 1 citation
Computer Science · Decision Sciences · #Artificial Intelligence (cs.AI) #Auction Theory and Applications #FOS: Computer and information sciences #FOS: Electrical engineering #FOS: Mathematics #Formal Methods in Verification #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Optimization and Control (math.OC) #Reinforcement Learning in Robotics #Systems and Control (eess.SY) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2211.13937

openalex publication_date 2022/11/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/29

Abstract

We introduce new planning and reinforcement learning algorithms for discounted MDPs that utilize an approximate model of the environment to accelerate the convergence of the value function. Inspired by the splitting approach in numerical linear algebra, we introduce Operator Splitting Value Iteration (OS-VI) for both Policy Evaluation and Control problems. OS-VI achieves a much faster convergence rate when the model is accurate enough. We also introduce a sample-based version of the algorithm called OS-Dyna. Unlike the traditional Dyna architecture, OS-Dyna still converges to the correct value function in presence of model approximation error.

Cited by

• Calibrated Value-Aware Model Learning with Probabilistic Environment Models

Related

• Deflated Dynamics Value Iteration
• Policy Iteration for Factored MDPs
• Reinforcement Learning with Unbiased Policy Evaluation and Linear Function Approximation
• A Computationally Efficient Algorithm for Infinite-Horizon Average-Reward Linear MDPs
• Acceleration Operators in the Value Iteration Algorithms for Markov Decision Processes