Reinforcement Learning with History-Dependent Dynamic Contexts

2023/02/04 by Tennenholtz, Guy, Merlis, Nadav, Shani, Lior +2 · 4 citations
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Systems and Control (eess.SY) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2302.02061

Abstract

We introduce Dynamic Contextual Markov Decision Processes (DCMDPs), a novel reinforcement learning framework for history-dependent environments that generalizes the contextual MDP framework to handle non-Markov environments, where contexts change over time. We consider special cases of the model, with a focus on logistic DCMDPs, which break the exponential dependence on history length by leveraging aggregation functions to determine context transitions. This special structure allows us to derive an upper-confidence-bound style algorithm for which we establish regret bounds. Motivated by our theoretical results, we introduce a practical model-based algorithm for logistic DCMDPs that plans in a latent space and uses optimism over history-dependent features. We demonstrate the efficacy of our approach on a recommendation task (using MovieLens data) where user behavior dynamics evolve in response to recommendations.

Cited by

• Adaptive Context Length Optimization with Low-Frequency Truncation for Multi-Agent Reinforcement Learning
• Reinforcement Learning and Consumption-Savings Behavior
• Learning to Decide with Just Enough: Information-Theoretic Context Summarization for CMDPs
• Wavelet Predictive Representations for Non-Stationary Reinforcement Learning

Related

• Reinforcement Learning in Presence of Discrete Markovian Context Evolution
• No-regret Exploration in Contextual Reinforcement Learning
• Block Contextual MDPs for Continual Learning
• On learning history based policies for controlling Markov decision processes
• Markov Decision Processes under External Temporal Processes