Batch-Constrained Distributional Reinforcement Learning for Session-based Recommendation

2020/12/16 by Diksha Garg, D. Garg, Garg, Diksha +8
Computer Science · Decision Sciences · Engineering · #Advanced Bandit Algorithms Research #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Machine Learning (cs.LG) #Recommender Systems and Techniques #Smart Grid Energy Management #cs.IR #cs.LG

paper · pdf · doi:10.48550/arxiv.2012.08984

Presented at Offline Reinforcement Learning Workshop at Neural Information Processing Systems, 2020

arxiv created 2020/12/16 · openalex publication_date 2020/12/16 · arxiv updated 2020/12/17 · openalex created_date 2020/12/21 · openalex updated_date 2026/07/28

Abstract

Most of the existing deep reinforcement learning (RL) approaches for session-based recommendations either rely on costly online interactions with real users, or rely on potentially biased rule-based or data-driven user-behavior models for learning. In this work, we instead focus on learning recommendation policies in the pure batch or offline setting, i.e. learning policies solely from offline historical interaction logs or batch data generated from an unknown and sub-optimal behavior policy, without further access to data from the real-world or user-behavior models. We propose BCD4Rec: Batch-Constrained Distributional RL for Session-based Recommendations. BCD4Rec builds upon the recent advances in batch (offline) RL and distributional RL to learn from offline logs while dealing with the intrinsically stochastic nature of rewards from the users due to varied latent interest preferences (environments). We demonstrate that BCD4Rec significantly improves upon the behavior policy as well as strong RL and non-RL baselines in the batch setting in terms of standard performance metrics like Click Through Rates or Buy Rates. Other useful properties of BCD4Rec include: i. recommending items from the correct latent categories indicating better value estimates despite large action space (of the order of number of items), and ii. overcoming popularity bias in clicked or bought items typically present in the offline logs.

Citations

• Playing Atari with Deep Reinforcement Learning
• Robust Estimation of a Location Parameter
• word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method
• Implicit Quantile Networks for Distributional Reinforcement Learning
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• Distributional Reinforcement Learning with Quantile Regression
• RecSim: A Configurable Simulation Platform for Recommender Systems
• Benchmarking Batch Deep Reinforcement Learning Algorithms
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• MOReL : Model-Based Offline Reinforcement Learning
• Conservative Q-Learning for Offline Reinforcement Learning
• Critic Regularized Regression

Related

• Rethinking Reinforcement Learning for Recommendation: A Prompt Perspective
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• Energy-Guided Diffusion Sampling for Long-Term User Behavior Prediction in Reinforcement Learning-based Recommendation
• A General Offline Reinforcement Learning Framework for Interactive Recommendation
• Intrinsically Motivated Reinforcement Learning based Recommendation with Counterfactual Data Augmentation