Automatic Representation for Lifetime Value Recommender Systems

2017/02/23 by Assaf Hallak, Yishay Mansour, Hallak, Assaf +3 · 1 citation
Computer Science · Decision Sciences · Engineering · #Advanced Bandit Algorithms Research #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Recommender Systems and Techniques #Smart Grid Energy Management

paper · pdf · doi:10.48550/arxiv.1702.07125

openalex publication_date 2017/02/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Many modern commercial sites employ recommender systems to propose relevant content to users. While most systems are focused on maximizing the immediate gain (clicks, purchases or ratings), a better notion of success would be the lifetime value (LTV) of the user-system interaction. The LTV approach considers the future implications of the item recommendation, and seeks to maximize the cumulative gain over time. The Reinforcement Learning (RL) framework is the standard formulation for optimizing cumulative successes over time. However, RL is rarely used in practice due to its associated representation, optimization and validation techniques which can be complex. In this paper we propose a new architecture for combining RL with recommendation systems which obviates the need for hand-tuned features, thus automating the state-space representation construction process. We analyze the practical difficulties in this formulation and test our solutions on batch off-line real-world recommendation data.

Citations

• Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
• word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method
• BPR: Bayesian Personalized Ranking from Implicit Feedback
• Benchmarking Deep Reinforcement Learning for Continuous Control

Cited by

• Reinforcement Learning for Slate-based Recommender Systems: A Tractable\n Decomposition and Practical Methodology

Related

• Reinforcement learning based recommender systems: A survey
• Reinforcement Learning-based Recommender Systems with Large Language Models for State Reward and Action Modeling
• On Modeling Long-Term User Engagement from Stochastic Feedback
• Rethinking Reinforcement Learning for Recommendation: A Prompt Perspective
• ResAct: Reinforcing Long-term Engagement in Sequential Recommendation with Residual Actor