TransMLA: Multi-Head Latent Attention Is All You Need

2025/02/11 by Fanxu Meng, Pingzhi Tang, Meng, Fanxu +9 · 10 voices · 12 citations
Computer Science · Psychology · #Cognitive psychology #Computer science #Geology #Head (geology) #Machine Learning and Data Classification #Natural Language Processing Techniques #Psychology #Topic Modeling #cs.AI #cs.LG

paper · pdf · doi:10.48550/arxiv.2502.07864

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/02/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we present TransMLA, a framework that seamlessly converts any GQA-based pre-trained model into an MLA-based model. Our approach enables direct compatibility with DeepSeek's codebase, allowing these models to fully leverage DeepSeek-specific optimizations such as vLLM and SGlang. By compressing 93% of the KV cache in LLaMA-2-7B, TransMLA achieves a 10.6x inference speedup at an 8K context length while preserving meaningful output quality. Additionally, the model requires only 6 billion tokens for fine-tuning to regain performance on par with the original across multiple benchmarks. TransMLA offers a practical solution for migrating GQA-based models to the MLA structure. When combined with DeepSeek's advanced features, such as FP8 quantization and Multi-Token Prediction, even greater inference acceleration can be realized.

Citations

• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Tensor Product Attention Is All You Need
• Language Models are Few-Shot Learners
• Attention Is All You Need

Cited by

• Latent Multi-Head Attention for Small Language Models
• Can Transformer Memory Be Corrupted? Investigating Cache-Side Vulnerabilities in Large Language Models
• xLLM Technical Report
• SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass
• Combating the Memory Walls: Optimization Pathways for Long-Context Agentic LLM Inference
• TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference
• A Random Matrix Theory Perspective on the Learning Dynamics of Multi-head Latent Attention
• BLaST: High Performance Inference and Pretraining using BLock Sparse Transformers
• MoE-GPS: Guidlines for Prediction Strategy for Dynamic Expert Duplication in MoE Load Balancing
• Hardware-Centric Analysis of DeepSeek's Multi-Head Latent Attention
• MorphServe: Efficient and Workload-Aware LLM Serving via Runtime Quantized Layer Swapping and KV Cache Resizing
• VideoMLA: Low-Rank Latent KV Cache for Minute-Scale Autoregressive Video Diffusion

Discussions

• TransMLA: Multi-head latent attention is all you need [hn, 123 points, 32 comments]
• TransMLA: Multi-head latent attention is all you need https://arxiv.org/abs/2502.07864 https://news.ycombinator.com/item?id=43969442 [bsky, 2 points, 1 comments]
• ⚡ Hackernews Top story: TransMLA: Multi-head latent attention is all you need [bsky, 1 points, 0 comments]
• TransMLA: Multi-head latent attention is all you need [bsky, 0 points, 0 comments]
• TransMLA: Multi-head latent attention is all you need #HackerNews https://arxiv.org/abs/2502.07864 [bsky, 0 points, 0 comments]
• TransMLA: Multi-head latent attention is all you need https://arxiv.org/abs/2502.07864 (https://news.ycombinator.com/item?id=43969442) [bsky, 0 points, 0 comments]
• TransMLA: Multi-head latent attention is all you need https://arxiv.org/abs/2502.07864 [bsky, 0 points, 0 comments]
• TransMLA: Multi-head latent attention is all you need View Article | Join the HN Conversation Summary of HN discussion 🧵👇 #hacker-news [bsky, 0 points, 1 comments]
• https://arxiv.org/abs/2502.07864 #cs.LG #cs.AI Event Attributes [bsky, 0 points, 0 comments]
• TransMLA: Multi-head latent attention is all you need https://arxiv.org/abs/2502.07864 (https://news.ycombinator.com/item?id=43969442) [bsky, 0 points, 0 comments]

Related

• GQLA: Group-Query Latent Attention for Hardware-Adaptive Large Language Model Decoding
• Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs
• Beyond KV Reconstruction: Functional Reconstruction for MLA Draft Models in Speculative Decoding
• Inference with Reference: Lossless Acceleration of Large Language Models
• DeepSeek-V3 Technical Report