Energy-Efficient LLM Serving via Disaggregated Attention--FFN and Flexible Frequency Scaling

2026/08/03 by Cunchen Hu, Liangliang Xu, Tian Liu +9
Computer Science · #cs.DC #cs.AI

paper · pdf

arxiv created 2026/08/03 · arxiv updated 2026/08/04

Abstract

Large language model (LLM) serving spans diverse applications with stringent service-level objectives (SLOs), often requiring GPUs to run at maximum frequencies and increasing energy consumption. Existing energy-management approaches adapt GPU frequencies only at the request or inference-phase level, overlooking operator-level differences in frequency sensitivity between Attention and feed-forward networks (FFNs). We find that the energy-optimal frequencies of Attention and FFN (A/F) differ and vary with the inference phase, workload, and system configurations. However, runtime variability and independent A/F frequency control create a large search space and high communication overhead. To address these challenges, we present AFlex, a framework that jointly optimizes resource provisioning and GPU frequency scaling for disaggregated A/F serving. AFlex introduces a global scheduler and a local operator-level dynamic voltage and frequency scaling (DVFS) controller to determine A/F resource allocations and frequencies. It further introduces an interleaved A/F pipeline with dynamic microbatch depth and adaptive request batching to reduce pipeline bubbles. We implement AFlex in SGLang and evaluate it on NVIDIA A800 GPUs using Qwen3-32B and Mixtral-8×7B under production Conversation and Coding traces. \AFlex reduces energy per token by up to 49% over state-of-the-art disaggregated serving and 48% over frequency-scaling systems while satisfying TTFT and TPOT SLOs.

Citations

• LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
• GreenLLM: SLO-Aware Dynamic Frequency Scaling for Energy-Efficient LLM Serving
• Qwen3 Technical Report
• Accelerating Retrieval-Augmented Generation
• SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Serving
• MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool
• Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving
• Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inference
• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
• Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
• Splitwise: Efficient generative LLM inference using phase splitting
• FlashDecoding++: Faster Large Language Model Inference on GPUs
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
• Fast Distributed Inference Serving for Large Language Models
• FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness