Mixture of Lookup Key-Value Experts

2025/12/10 by Wang, Zongcheng
Computer Science · #Context-Aware Activity Recognition Systems #FOS: Computer and information sciences #IoT and Edge/Fog Computing #Machine Learning (cs.LG) #Mobile Crowdsensing and Crowdsourcing

paper · doi:10.48550/arxiv.2512.09723

openalex publication_date 2025/12/10 · openalex created_date 2025/12/12 · openalex updated_date 2026/07/28

Abstract

Recent research has developed several LLM architectures suitable for inference on end-user devices, such as the Mixture of Lookup Experts (MoLE)~\parencitejie_mixture₂025. A key feature of MoLE is that each token id is associated with a dedicated group of experts. For a given input, only the experts corresponding to the input token id will be activated. Since the communication overhead of loading this small number of activated experts into RAM during inference is negligible, expert parameters can be offloaded to storage, making MoLE suitable for resource-constrained devices. However, MoLE's context-independent expert selection mechanism, based solely on input ids, may limit model performance. To address this, we propose the Mixture of Lookup Key-Value Experts (MoLKV) model. In MoLKV, each expert is structured as a key-value pair. For a given input, the input-derived query interacts with the cached key-value experts from the current sequence, generating a context-aware expert output. This context-aware mechanism alleviates the limitation of MoLE, and experimental results demonstrate that MoLKV achieves significantly lower validation loss in small-scale evaluations.

Citations

• DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
• UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior Long-Context Learning
• Cartridges: Lightweight and general-purpose long context representations via self-study
• MemOS: An Operating System for Memory-Augmented Generation (MAG) in Large Language Models
• ContextAgent: Context-Aware Proactive LLM Agents with Open-World Sensory Perceptions
• Qwen3 Technical Report
• Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
• Mixture of Lookup Experts
• MoBA: Mixture of Block Attention for Long-Context LLMs
• Scaling Embedding Layers in Language Models
• Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling
• Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models
• Memory Layers at Scale
• Densing Law of LLMs
• Ultra-Sparse Memory Network
• Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance
• Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts
• Mixture of A Million Experts
• The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
• PowerInfer-2: Fast Large Language Model Inference on a Smartphone
• Scaling Laws for Fine-Grained Mixture of Experts
• DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• GLU Variants Improve Transformer
• Root Mean Square Layer Normalization
• Large Memory Layers with Product Keys
• Attention Is All You Need
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs

Related

• Mixture of Lookup Experts
• ExpertRAG: Efficient RAG with Mixture of Experts -- Optimizing Context Retrieval for Adaptive LLM Responses
• Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading
• EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Models
• Mixture Compressor for Mixture-of-Experts LLMs Gains More