SPAD: Specialized Prefill and Decode Hardware for Disaggregated LLM Inference

2025/10/09 by Hengrui Zhang, Zhang, Hengrui, Pratyush Patel +5 · 2 citations
Computer Science · #Advanced Graph Neural Networks #Advanced Neural Network Applications #Distributed #FOS: Computer and information sciences #Hardware Architecture (cs.AR) #Machine Learning (cs.LG) #Parallel #Privacy-Preserving Technologies in Data #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.2510.08544

openalex publication_date 2025/10/09 · openalex created_date 2025/10/11 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) have gained popularity in recent years, driving up the demand for inference. LLM inference is composed of two phases with distinct characteristics: a compute-bound prefill phase followed by a memory-bound decode phase. To efficiently serve LLMs, prior work proposes prefill-decode disaggregation to run each phase on separate hardware. However, existing hardware poorly matches the different requirements of each phase. Current datacenter GPUs and TPUs follow a more-is-better design philosophy that maximizes compute and memory resources, causing memory bandwidth underutilization in the prefill phase and compute underutilization in the decode phase. Such underutilization directly translates into increased serving costs. This paper proposes SPAD (Specialized Prefill and Decode hardware), adopting a less-is-more methodology to design specialized chips tailored to the distinct characteristics of prefill and decode phases. The proposed Prefill Chips have larger systolic arrays and use cost-effective GDDR memory, whereas the proposed Decode Chips retain high memory bandwidth but reduce compute capacity. Compared to modeled H100s, simulations show that the proposed Prefill Chips deliver 8% higher prefill performance on average at 52% lower hardware cost, while the proposed Decode Chips achieve 97% of the decode performance with 28% lower TDP. End-to-end simulations on production traces show that SPAD reduces hardware cost by 19%-41% and TDP by 2%-17% compared to modeled baseline clusters while offering the same performance. Even when models and workloads change, SPAD can reallocate either type of chip to run either phase and still achieve 11%-43% lower hardware costs, demonstrating the longevity of the SPAD design.

Citations

• Jenga: Effective Memory Management for Serving LLM with Heterogeneity
• ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments
• FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving
• NanoFlow: Towards Optimal Large Language Model Serving Throughput
• The Llama 3 Herd of Models
• Mirage: A Multi-Level Superoptimizer for Tensor Programs
• Preble: Efficient Distributed Prompt Scheduling for LLM Serving
• Vidur: A Large-Scale Simulation Framework For LLM Inference
• LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism
• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
• SGLang: Efficient Execution of Structured Language Model Programs
• Splitwise: Efficient generative LLM inference using phase splitting
• Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
• Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
• TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings
• GPT-4 Technical Report
• FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
• DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• Forecasting at Scale
• Attention Is All You Need
• In-Datacenter Performance Analysis of a Tensor Processing Unit

Cited by

• Efficient MoE Serving in the Memory-Bound Regime: Balance Activated Experts, Not Tokens
• NELSSA: A GPU-PNM Heterogeneous System for Mixed-Length LLM Serving via Length-based Request Placement

Related

• SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
• From Attention to Disaggregation: Tracing the Evolution of LLM Inference
• Make LLM Inference Affordable to Everyone: Augmenting GPU Memory with NDP-DIMM
• PD-Swap: Prefill-Decode Logic Swapping for End-to-End LLM Inference on Edge FPGAs via Dynamic Partial Reconfiguration
• Prefill vs. Decode Bottlenecks: SRAM-Frequency Tradeoffs and the Memory-Bandwidth Ceiling