Tensor and Matrix Low-Rank Value-Function Approximation in Reinforcement Learning

2022/01/21 by Sergio Rozada, Rozada, Sergio, Paternain, Santiago +1 · 4 citations
Engineering · Mathematics · #Advanced MIMO Systems Optimization #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Sparse and Compressive Sensing Techniques #Tensor decomposition and applications

paper · pdf · doi:10.48550/arxiv.2201.09736

openalex publication_date 2022/01/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Value-function (VF) approximation is a central problem in Reinforcement Learning (RL). Classical non-parametric VF estimation suffers from the curse of dimensionality. As a result, parsimonious parametric models have been adopted to approximate VFs in high-dimensional spaces, with most efforts being focused on linear and neural-network-based approaches. Differently, this paper puts forth a a parsimonious non-parametric approach, where we use stochastic low-rank algorithms to estimate the VF matrix in an online and model-free fashion. Furthermore, as VFs tend to be multi-dimensional, we propose replacing the classical VF matrix representation with a tensor (multi-way array) representation and, then, use the PARAFAC decomposition to design an online model-free tensor low-rank algorithm. Different versions of the algorithms are proposed, their complexity is analyzed, and their performance is assessed numerically using standardized RL environments.

Cited by

• Tensor-Efficient High-Dimensional Q-learning
• ADARL: Adaptive Low-Rank Structures for Robust Policy Learning under Uncertainty
• Unrolling Dynamic Programming via Graph Filters
• Identifiability of Deep Polynomial Neural Networks

Related

• Tensor Low-rank Approximation of Finite-horizon Value Functions
• Low-rank State-action Value-function Approximation
• Multilinear Tensor Low-Rank Approximation for Policy-Gradient Methods in Reinforcement Learning
• A Tensor Low-Rank Approximation for Value Functions in Multi-Task Reinforcement Learning
• Biologically inspired architectures for sample-efficient deep reinforcement learning