TransMLA: Multi-Head Latent Attention Is All You Need

2025/02/11 by Fanxu Meng, Pingzhi Tang, Meng, Fanxu +9 · 10 voices · 5 citations
Computer Science · #Machine Learning and Data Classification #Natural Language Processing Techniques #Topic Modeling #cs.AI #cs.LG

paper · pdf · doi:10.48550/arxiv.2502.07864

openalex publication_date 2025/02/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we present TransMLA, a framework that seamlessly converts any GQA-based pre-trained model into an MLA-based model. Our approach enables direct compatibility with DeepSeek's codebase, allowing these models to fully leverage DeepSeek-specific optimizations such as vLLM and SGlang. By compressing 93% of the KV cache in LLaMA-2-7B, TransMLA achieves a 10.6x inference speedup at an 8K context length while preserving meaningful output quality. Additionally, the model requires only 6 billion tokens for fine-tuning to regain performance on par with the original across multiple benchmarks. TransMLA offers a practical solution for migrating GQA-based models to the MLA structure. When combined with DeepSeek's advanced features, such as FP8 quantization and Multi-Token Prediction, even greater inference acceleration can be realized.

Citations

• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Tensor Product Attention Is All You Need
• Language Models are Few-Shot Learners
• Attention Is All You Need

Cited by

• Can Transformer Memory Be Corrupted? Investigating Cache-Side Vulnerabilities in Large Language Models
• xLLM Technical Report
• SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass
• Combating the Memory Walls: Optimization Pathways for Long-Context Agentic LLM Inference
• TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference

Discussions

• TransMLA: Multi-head latent attention is all you need [hn, 123 points, 32 comments]
• TransMLA: Multi-head latent attention is all you need https://arxiv.org/abs/2502.07864 https://news.ycombinator.com/item?id=43969442 [bsky, 2 points, 1 comments]
• ⚡ Hackernews Top story: TransMLA: Multi-head latent attention is all you need [bsky, 1 points, 0 comments]
• TransMLA: Multi-head latent attention is all you need [bsky, 0 points, 0 comments]
• TransMLA: Multi-head latent attention is all you need #HackerNews https://arxiv.org/abs/2502.07864 [bsky, 0 points, 0 comments]
• TransMLA: Multi-head latent attention is all you need https://arxiv.org/abs/2502.07864 (https://news.ycombinator.com/item?id=43969442) [bsky, 0 points, 0 comments]
• TransMLA: Multi-head latent attention is all you need https://arxiv.org/abs/2502.07864 [bsky, 0 points, 0 comments]
• TransMLA: Multi-head latent attention is all you need View Article | Join the HN Conversation Summary of HN discussion 🧵👇 #hacker-news [bsky, 0 points, 1 comments]
• https://arxiv.org/abs/2502.07864 #cs.LG #cs.AI Event Attributes [bsky, 0 points, 0 comments]
• TransMLA: Multi-head latent attention is all you need https://arxiv.org/abs/2502.07864 (https://news.ycombinator.com/item?id=43969442) [bsky, 0 points, 0 comments]

Related

• GQLA: Group-Query Latent Attention for Hardware-Adaptive Large Language Model Decoding
• Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs
• Beyond KV Reconstruction: Functional Reconstruction for MLA Draft Models in Speculative Decoding
• Inference with Reference: Lossless Acceleration of Large Language Models
• Zebra-Llama: Towards Extremely Efficient Hybrid Models