Candidate Generation with Binary Codes for Large-Scale Top-N Recommendation

2019/09/12 by Wang-Cheng Kang, Julian McAuley, Kang, Wang-Cheng +1 · 8 citations
Computer Science · Mathematics · #Advanced Image and Video Retrieval Techniques #Artificial intelligence #Binary number #Computer science #Data mining #Embedding #FOS: Computer and information sciences #Hash function #Hash table #Information Retrieval (cs.IR) #Information retrieval #Machine learning #Mathematics #Mobile Crowdsensing and Crowdsourcing #Rank (graph theory) #Ranking (information retrieval) #Recommender Systems and Techniques #Recommender system #Scale (ratio) #cs.IR

paper · pdf · doi:10.48550/arxiv.1909.05475

published in arXiv (Cornell University) (Cornell University) · accepted to CIKM'19 as long paper

arxiv created 2019/09/12 · openalex publication_date 2019/09/12 · arxiv updated 2019/09/13 · openalex created_date 2019/09/19 · openalex updated_date 2026/08/08

Abstract

Generating the Top-N recommendations from a large corpus is computationally expensive to perform at scale. Candidate generation and re-ranking based approaches are often adopted in industrial settings to alleviate efficiency problems. However it remains to be fully studied how well such schemes approximate complete rankings (or how many candidates are required to achieve a good approximation), or to develop systematic approaches to generate high-quality candidates efficiently. In this paper, we seek to investigate these questions via proposing a candidate generation and re-ranking based framework (CIGAR), which first learns a preference-preserving binary embedding for building a hash table to retrieve candidates, and then learns to re-rank the candidates using real-valued ranking models with a candidate-oriented objective. We perform a comprehensive study on several large-scale real-world datasets consisting of millions of users/items and hundreds of millions of interactions. Our results show that CIGAR significantly boosts the Top-N accuracy against state-of-the-art recommendation models, while reducing the query time by orders of magnitude. We hope that this work could draw more attention to the candidate generation problem in recommender systems.

Citations

• Distilling the Knowledge in a Neural Network
• BPR: Bayesian Personalized Ranking from Implicit Feedback
• Item-based top-Nrecommendation algorithms
• Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS)
• Neural Collaborative Filtering

Cited by

• OneShot: Index-in-Ranking with Neural Scoring for Large-Scale Retrieval
• CoVE: Compressed Vocabulary Expansion Makes Better LLM-based Recommender Systems
• Learning Binarized Representations with Pseudo-positive Sample Enhancement for Efficient Graph Collaborative Filtering
• Killing Two Birds with One Stone: Unifying Retrieval and Ranking with a Single Generative Recommendation Model

Related

• Deep Retrieval: Learning A Retrievable Structure for Large-Scale Recommendations
• GAPfm: Optimal Top-N Recommendations for Graded Relevance Domains
• Large Language Models are Zero-Shot Rankers for Recommender Systems
• Top-N-Rank: A Scalable List-wise Ranking Method for Recommender Systems
• A Novel User Representation Paradigm for Making Personalized Candidate Retrieval