An MDP-based Recommender System

2012/12/12 by Guy Shani, Shani, Guy, Ronen I. Brafman +3 · 101 citations
Computer Science · Mathematics · #Advanced Graph Neural Networks #Artificial Intelligence (cs.AI) #Artificial intelligence #Computer science #Data mining #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Machine Learning (cs.LG) #Machine learning #Markov decision process #Markov process #Mathematics #Predictive value #Process (computing) #Recommender Systems and Techniques #Recommender system #Topic Modeling #cs.AI #cs.IR #cs.LG

paper · pdf · doi:10.48550/arxiv.1301.0600

published in arXiv (Cornell University) 6(43), 1265-1295 (Cornell University) · Appears in Proceedings of the Eighteenth Conference on Uncertainty in Artificial Intelligence (UAI2002)

openalex publication_date 2012/12/12 · arxiv created 2015/05/16 · arxiv updated 2015/05/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/08

Abstract

Typical Recommender systems adopt a static view of the recommendation process and treat it as a prediction problem. We argue that it is more appropriate to view the problem of generating recommendations as a sequential decision problem and, consequently, that Markov decision processes (MDP) provide a more appropriate model for Recommender systems. MDPs introduce two benefits: they take into account the long-term effects of each recommendation, and they take into account the expected value of each recommendation. To succeed in practice, an MDP-based Recommender system must employ a strong initial model; and the bulk of this paper is concerned with the generation of such a model. In particular, we suggest the use of an n-gram predictive model for generating the initial MDP. Our n-gram model induces a Markov-chain model of user behavior whose predictive accuracy is greater than that of existing predictive models. We describe our predictive model in detail and evaluate its performance on real data. In addition, we show how the model can be used in an MDP-based Recommender system.

Citations

• Anytime State-Based Solution Methods for Decision Processes with non-Markovian Rewards
• Empirical Analysis of Predictive Algorithms for Collaborative Filtering

Cited by

• BiCoRec: Bias-Mitigated Context-Aware Sequential Recommendation Model
• An Interdisciplinary and Cross-Task Review on Missing Data Imputation
• CausalRec: A CausalBoost Attention Model for Sequential Recommendation
• Hierarchical Semantic RL: Tackling the Problem of Dynamic Action Space for RL-based Recommendations
• Beyond Static Evaluation: Rethinking the Assessment of Personalized Agent Adaptability in Information Retrieval
• Integrating LLM-Derived Multi-Semantic Intent into Graph Model for Session-based Recommendation
• Reinforce Lifelong Interaction Value of User-Author Pairs for Large-Scale Recommendation Systems
• When Transformers Meet Recommenders: Integrating Self-Attentive Sequential Recommendation with Fine-Tuned LLMs
• Hierarchical Intent-guided Optimization with Pluggable LLM-Driven Semantics for Session-based Recommendation
• Churn-Aware Recommendation Planning under Aggregated Preference Feedback
• Listwise Preference Alignment Optimization for Tail Item Recommendation
• Alleviating User-Sensitive bias with Fair Generative Sequential Recommendation Model
• LLM-Enhanced Multimodal Fusion for Cross-Domain Sequential Recommendation
• Heterogeneous Sequel-Aware Graph Neural Networks for Sequential Learning
• Bidirectional Knowledge Distillation for Enhancing Sequential Recommendation with Large Language Models
• A Survey on Side Information-driven Session-based Recommendation: From a Data-centric Perspective
• Adaptive Resolving Methods for Reinforcement Learning with Function Approximations
• Context-aware short-term interest first model for session-based recommendation
• Exact Unlearning in Reinforcement Learning
• Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions
• Cognitive-Inspired Hierarchical Attention Fusion With Visual and Textual for Cross-Domain Sequential Recommendation

Related

• Partially Observable Markov Decision Process for Recommender Systems
• Recommendation systems in the scope of opinion formation: a model
• Optimal Recommendation to Users that React: Online Learning for a Class of POMDPs
• Reinforcement learning based recommender systems: A survey
• Modeling the Dynamics of User Preferences for Sequence-Aware Recommendation Using Hidden Markov Models