Ultra-Sparse Memory Network

2024/11/19 by Zihao Huang, Huang, Zihao, Qiyang Min +11 · 2 voices · 6 citations
Computer Science · Engineering · #Advanced Memory and Neural Computing #Machine Learning and ELM #Neural Networks and Reservoir Computing #cs.LG

paper · pdf · doi:10.48550/arxiv.2411.12364

openalex publication_date 2024/11/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

It is widely acknowledged that the performance of Transformer models is logarithmically related to their number of parameters and computational complexity. While approaches like Mixture of Experts (MoE) decouple parameter count from computational complexity, they still face challenges in inference due to high memory access costs. This work introduces UltraMem, incorporating large-scale, ultra-sparse memory layer to address these limitations. Our approach significantly reduces inference latency while maintaining model performance. We also investigate the scaling laws of this new architecture, demonstrating that it not only exhibits favorable scaling properties but outperforms MoE. In experiments, the largest UltraMem we train has 20 million memory slots. The results show that our method achieves state-of-the-art inference speed and model performance within a given computational budget, paving the way for billions of slots or experts.

Cited by

• SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations
• MIDUS: Memory-Infused Depth Up-Scaling
• Mixture of Lookup Key-Value Experts
• Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
• Pretraining with hierarchical memories: separating long-tail and common knowledge
• UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior Long-Context Learning

Discussions

• UltraMem, from Bytedance, incorporates large-scale, ultra-sparse memory layers to enhance computational efficiency and reduce inference latency in language models 🙌 shows stronger scaling ability tha [bsky, 2 points, 0 comments]
• Ultra-Sparse Memory Network Introduces ultra-sparse architectures that reduce latency and improve memory efficiency, rivaling larger dense models. arxiv.org/abs/2411.12364 [bsky, 0 points, 1 comments]

Related

• UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior Long-Context Learning
• Easy and Efficient Transformer : Scalable Inference Solution For large NLP model
• DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
• Efficiently Scaling Transformer Inference
• Mixture of A Million Experts