A Tensor Network Approach to Finite Markov Decision Processes

2020/02/12 by Edward Gillman, Gillman, Edward, D. C. Rose +4
Computer Science · Mathematics · Physics and Astronomy · #Advanced Thermodynamics and Statistical Mechanics #Artificial intelligence #Computer science #Context (archaeology) #Exploit #FOS: Computer and information sciences #FOS: Physical sciences #Machine Learning (cs.LG) #Machine learning #Markov chain #Markov decision process #Markov process #Mathematical optimization #Mathematics #Quantum Physics (quant-ph) #Quantum many-body systems #Reinforcement learning #Simple (philosophy) #Statistical Mechanics (cond-mat.stat-mech) #Tensor (intrinsic definition) #Tensor decomposition and applications #Theoretical computer science #cond-mat.stat-mech #cs.LG #quant-ph

paper · pdf · doi:10.48550/arxiv.2002.05185

10 pages, 2 figures

arxiv created 2020/02/12 · openalex publication_date 2020/02/12 · arxiv updated 2020/02/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Tensor network (TN) techniques - often used in the context of quantum many-body physics - have shown promise as a tool for tackling machine learning (ML) problems. The application of TNs to ML, however, has mostly focused on supervised and unsupervised learning. Yet, with their direct connection to hidden Markov chains, TNs are also naturally suited to Markov decision processes (MDPs) which provide the foundation for reinforcement learning (RL). Here we introduce a general TN formulation of finite, episodic and discrete MDPs. We show how this formulation allows us to exploit algorithms developed for TNs for policy optimisation, the key aim of RL. As an application we consider the issue - formulated as an RL problem - of finding a stochastic evolution that satisfies specific dynamical conditions, using the simple example of random walk excursions as an illustration.

Citations

• Trust Region Policy Optimization
• Colloquium: Area laws for the entanglement entropy
• Exponential Decay of Correlations Implies Area Law
• Density-matrix algorithms for quantum renormalization groups
• Finite automata for caching in matrix product algorithms
• TRANSITIONPATHSAMPLING: Throwing Ropes Over Rough Mountain Passes, in the Dark
• Human-level control through deep reinforcement learning
• Mastering the game of Go with deep neural networks and tree search
• Classical stochastic dynamics and continuous matrix product states: gauge transformations, conditioned and driven processes, and equivalence of trajectory ensembles
• Unsupervised Generative Modeling Using Matrix Product States
• Soft Actor-Critic Algorithms and Applications
• Neural-Network Approach to Dissipative Quantum Many-Body Dynamics
• Constructing neural stationary states for open quantum many-body systems
• TensorNetwork for Machine Learning
• Variational Quantum Monte Carlo Method with a Neural-Network Ansatz for Open Quantum Systems
• Benchmarking Model-Based Reinforcement Learning

Related

• A Tensor Network Implementation of Multi Agent Reinforcement Learning
• Combining Reinforcement Learning and Tensor Networks, with an Application to Dynamical Large Deviations
• Learning Good State and Action Representations via Tensor Decomposition
• Solving Finite-Horizon MDPs via Low-Rank Tensors
• Tensor Low-rank Approximation of Finite-horizon Value Functions