Wukong: Towards a Scaling Law for Large-Scale Recommendation

2024/03/04 by Buyun Zhang, Zhang, Buyun, Liang Luo +27 · 22 citations
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning in Healthcare

paper · pdf · doi:10.48550/arxiv.2403.02545

openalex publication_date 2024/03/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Scaling laws play an instrumental role in the sustainable improvement in model quality. Unfortunately, recommendation models to date do not exhibit such laws similar to those observed in the domain of large language models, due to the inefficiencies of their upscaling mechanisms. This limitation poses significant challenges in adapting these models to increasingly more complex real-world datasets. In this paper, we propose an effective network architecture based purely on stacked factorization machines, and a synergistic upscaling strategy, collectively dubbed Wukong, to establish a scaling law in the domain of recommendation. Wukong's unique design makes it possible to capture diverse, any-order of interactions simply through taller and wider layers. We conducted extensive evaluations on six public datasets, and our results demonstrate that Wukong consistently outperforms state-of-the-art models quality-wise. Further, we assessed Wukong's scalability on an internal, large-scale dataset. The results show that Wukong retains its superiority in quality over state-of-the-art models, while holding the scaling law across two orders of magnitude in model complexity, extending beyond 100 GFLOP/example, where prior arts fall short.

Cited by

• Mosaic: A Fleet of User Embedding Specialists for Recommendation at Meta
• Unifying Generative Recall and Multi-Objective Ranking in a Single Decoder-Only Sequence
• Bumblebee: Interleaved Mixed-Layer Building Blocks for Large-Scale Recommendation Systems
• Exploring Test-time Scaling via Prediction Merging on Large-Scale Recommendation
• HHFT: Hierarchical Heterogeneous Feature Transformer for Recommendation Systems
• DynamiX: Dynamic Resource eXploration for Personalized Ad-Recommendations
• SilverTorch: A Unified Model-based System to Democratize Large-Scale Recommendation on GPUs
• From Scaling to Structured Expressivity: Rethinking Transformers for CTR Prediction
• LEMUR: Large scale End-to-end MUltimodal Recommendation
• GPR: Towards a Generative Pre-trained One-Model Paradigm for Large-Scale Advertising Recommendation
• Adaptive Regularization for Large-Scale Sparse Feature Embedding Models
• OneTrans: Unified Feature Interaction and Sequence Modeling with One Transformer in Industrial Recommender
• WhisperRec: Latent Reasoning for Efficient Foundation Recommendation Models
• Revisiting scalable sequential recommendation with Multi-Embedding Approach and Mixture-of-Experts
• MiniOneRec: An Open-Source Framework for Scaling Generative Recommendation
• MTmixAtt: Integrating Mixture-of-Experts with Multi-Mix Attention for Large-Scale Recommendation
• RecIS: Sparse to Dense, A Unified Training Framework for Recommendation Models
• Understanding Embedding Scaling in Collaborative Filtering
• FLAME: A Serving System Optimized for Large-Scale Generative Recommendation with Efficiency
• asLLR: LLM based Leads Ranking in Auto Sales
• Exploring Scaling Laws of CTR Model for Online Performance Improvement
• Realizing Scaling Laws in Recommender Systems: A Foundation-Expert Paradigm for Hyperscale Model Deployment

Related

• Scaling New Frontiers: Insights into Large Recommendation Models
• Climber: Toward Efficient Scaling Laws for Large Recommendation Models
• Understanding Scaling Laws for Recommendation Models
• WHALE: A Scalable Unified Model for Recommendation with Wukong-HSTU Architecture
• Generative Large Recommendation Models: Emerging Trends in LLMs for Recommendation