Efficiently Scaling Transformer Inference

2022/11/09 by Reiner Pope, Pope, Reiner, Sholto Douglas +17 · 76 citations
Computer Science · #Advanced Neural Network Applications #Computation and Language (cs.CL) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2211.05102

openalex publication_date 2022/11/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We study the problem of efficient generative inference for Transformer models, in one of its most challenging settings: large deep models, with tight latency targets and long sequence lengths. Better understanding of the engineering tradeoffs for inference for large Transformer-based models is important as use cases of these models are growing rapidly throughout application areas. We develop a simple analytical model for inference efficiency to select the best multi-dimensional partitioning techniques optimized for TPU v4 slices based on the application requirements. We combine these with a suite of low-level optimizations to achieve a new Pareto frontier on the latency and model FLOPS utilization (MFU) tradeoffs on 500B+ parameter models that outperforms the FasterTransformer suite of benchmarks. We further show that with appropriate partitioning, the lower memory requirements of multiquery attention (i.e. multiple query heads share single key/value head) enables scaling up to 32x larger context lengths. Finally, we achieve a low-batch-size latency of 29ms per token during generation (using int8 weight quantization) and a 76% MFU during large-batch-size processing of input tokens, while supporting a long 2048-token context length on the PaLM 540B parameter model.

Cited by

• Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Serving
• Attention Residuals
• RayRoPE: Projective Ray Positional Encoding for Multi-view Attention
• KAP: Bridging the Knowledge Selection-Runtime Consumption Gap in LLM Systems
• Rethinking CD: A Reproducibility Study and Extension on the Ineffectiveness of Contrastive Decoding at Mitigating Object Hallucinations in MLLMs
• PTStore (Prefix Tensor Store): Distributed Prefix Caching and Replication for High Throughput Inference Serving
• Hybrid Combinatorial Multi-armed Bandits with Probabilistically Triggered Arms
• LinkedOut: Linking World Knowledge Representation Out of Video LLM for Next-Generation Video Recommendation
• QuadSentinel: Sequent Safety for Machine-Checkable Control in Multi-agent Systems
• Transform Trained Transformer: Accelerating Naive 4K Video Generation Over 10×
• Unlocking the Address Book: Dissecting the Sparse Semantic Structure of LLM Key-Value Caches via Sparse Autoencoders
• RACAM: Enhancing DRAM with Reuse-Aware Computation and Automated Mapping for ML Inference
• LAWS: Learning from Actual Workloads Symbolically -- A Self-Certifying Parametrized Cache Architecture for Neural Inference, Robotics, and Edge Deployment
• CacheTrap: Injecting Trojans in LLMs without Leaving any Traces in Inputs or Weights
• Block Cascading: Training Free Acceleration of Block-Causal Video Models
• Orchestrating Dual-Boundaries: An Arithmetic Intensity Inspired Acceleration Framework for Diffusion Language Models
• Comparative Analysis of Large Language Model Inference Serving Systems: A Performance Study of vLLM and HuggingFace TGI
• Generative Caching for Structurally Similar Prompts and Responses
• Moirai 2.0: When Less Is More for Time Series Forecasting
• When, What, and How: Rethinking Retrieval-Enhanced Speculative Decoding
• You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations
• The Future of Fully Homomorphic Encryption System: from a Storage I/O Perspective
• BudgetMem: Learning Selective Memory Policies for Cost-Efficient Long-Context Processing in Language Models
• FlashEVA: Accelerating LLM inference via Efficient Attention
• Loquetier: A Virtualized Multi-LoRA Framework for Unified LLM Fine-tuning and Serving
• Serve Programs, Not Prompts
• RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention
• Keyless Attention: Value-Space Routing and Value-Only Caching for Efficient Transformers
• Autoregressive Boltzmann Generators
• Rethinking Inference Placement for Deep Learning across Edge and Cloud Platforms: A Multi-Objective Optimization Perspective and Future Directions
• Stateful KV Cache Management for LLMs: Balancing Space, Time, Accuracy, and Positional Fidelity
• Planned Diffusion
• Improving Model Representation and Reducing KV Cache via Skip Connections with First Value Heads
• Attention Is All You Need for KV Cache in Diffusion LLMs
• Tawa: Automatic Warp Specialization for Modern GPUs with Asynchronous References
• First Attentions Last: Better Exploiting First Attentions for Efficient Transformer Training
• Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference
• Neural Weight Compression for Language Models
• xLSTM Scaling Laws: Competitive Performance with Linear Time-Complexity
• Direct Multi-Token Decoding
• Sparse Query Attention (SQA): A Computationally Efficient Attention Mechanism with Query Heads Reduction
• The Price Reversal Phenomenon: When Cheaper Reasoning Models Cost More
• Stability of Transformers under Layer Normalization
• Online Video Depth Anything: Temporally-Consistent Depth Prediction with Low Memory Consumption
• LightCache: Memory-Efficient, Training-Free Acceleration for Video Generation
• Beyond Next-Token Prediction: A Performance Characterization of Diffusion versus Autoregressive Language Models
• Exact Causal Attention with 10% Fewer Operations
• TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Scheduling
• Color2Struct: efficient and accurate deep-learning inverse design of structural color with controllable inference
• The Pitfalls of KV Cache Compression
• Entropy After ⟨ /Think ⟩ for reasoning model early exiting
• LocoFormer: Generalist Locomotion via Long-context Adaptation
• Are We Scaling the Right Thing? A System Perspective on Test-Time Scaling
• Confidential LLM Inference: Performance and Cost Across CPU and GPU TEEs
• AHA -- Predicting What Matters Next: Online Highlight Detection Without Looking Ahead
• HERO: Rethinking Visual Token Early Dropping in High-Resolution Large Vision-Language Models
• Beyond Data Privacy: New Privacy Risks for Large Language Models
• Characterizing the Efficiency of Distributed Training: A Power, Performance, and Thermal Perspective
• VoltanaLLM: Feedback-Driven Frequency Control and State-Space Routing for Energy-Efficient LLM Serving
• TREE:Token-Responsive Energy Efficiency Framework For Green AI-Integrated 6G Networks
• AI Compute Architecture and Evolution Trends
• Provable Benefits of In-Tool Learning for Large Language Models
• APT-LLM: Exploiting Arbitrary-Precision Tensor Core Computing for LLM Acceleration
• Exploring Scaling Laws of CTR Model for Online Performance Improvement
• Transduction is All You Need for Structured Data Workflows
• MoEcho: Exploiting Side-Channel Attacks to Compromise User Privacy in Mixture-of-Experts LLMs
• Is GPT-OSS Good? A Comprehensive Evaluation of OpenAI's Latest Open Source Models
• Dynamic Quality-Latency Aware Routing for LLM Inference in Wireless Edge-Device Networks
• Meta-Metrics and Best Practices for System-Level Inference Performance Benchmarking
• Shadow in the Cache: Unveiling and Mitigating Privacy Risks of KV-cache in LLM Inference
• Follow-Your-Shape: Shape-Aware Image Editing via Trajectory-Guided Region Control
• Semantic Caching for Low-Cost LLM Serving: From Offline Learning to Online Adaptation
• Meta-Learning for Speeding Up Large Model Inference in Decentralized Environments
• Mamba-X: An End-to-End Vision Mamba Accelerator for Edge Computing Devices
• FlashSVD: Memory-Efficient Inference with Streaming for Low-Rank Models
• BAR Conjecture: the Feasibility of Inference Budget-Constrained LLM Services with Authenticity and Reasoning

Related

• Easy and Efficient Transformer : Scalable Inference Solution For large NLP model
• Exploiting Sparsity for Long Context Inference: Million Token Contexts on Commodity GPUs
• DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
• Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers
• Optimizing Inference in Transformer-Based Models: A Multi-Method Benchmark