AGFT: An Adaptive GPU Frequency Tuner for Real-Time LLM Inference Optimization

2025/08/03 by Z. Ye, Ye, Zicong, Kunming Zhang +3
Computer Science · Engineering · #Big Data and Digital Economy #Cloud Computing and Resource Management #FOS: Computer and information sciences #Green IT and Sustainability #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2508.01744

openalex publication_date 2025/08/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The explosive growth of interactive Large Language Models (LLMs) has placed unprecedented demands for low latency on cloud GPUs, forcing them into high-power modes and causing escalating energy costs. Real-time inference workloads exhibit significant dynamic volatility, presenting substantial energy-saving opportunities. However, traditional static or rule-based power management strategies struggle to exploit these opportunities without compromising peak performance. To address this challenge, we propose AGFT (An Adaptive GPU Frequency Tuner), a framework that employs online reinforcement learning to autonomously learn an optimal frequency tuning policy. By monitoring real-time features like request load and latency, AGFT utilizes fine-grained frequency control for precise adjustments and intelligent action space pruning for stable, efficient decision-making. This creates a robust, automated energy management solution. We comprehensively evaluated AGFT in an environment simulating realistic, fluctuating inference requests. The experimental results demonstrate that AGFT successfully saves 44.3% of GPU energy consumption while introducing a minimal performance latency overhead of under 10%. This achievement translates into a comprehensive Energy-Delay Product (EDP) optimization of up to 40.3%, clearly showing that our framework can significantly enhance the energy efficiency and economic benefits of existing LLM inference clusters without compromising service quality.

Citations

• On Evaluating Performance of LLM Inference Serving Systems
• Online Multi-LLM Selection via Contextual Bandits under Unstructured Context Evolution
• Optimizing LLM Inference Throughput via Memory-aware and SLA-constrained Dynamic Batching
• LNUCB-TA: Linear-nonlinear Hybrid Bandit Learning with Temporal Attention
• Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scaling
• KunServe: Parameter-centric Memory Management for Efficient Memory Overloading Handling in LLM Serving
• Hierarchical Upper Confidence Bounds for Constrained Online Learning
• Online GPU Energy Optimization with Switching-Aware Bandits
• SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Serving
• Llumnix: Dynamic Scheduling for Large Language Model Serving
• BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems
• Splitwise: Efficient generative LLM inference using phase splitting
• Pure Exploration in Asynchronous Federated Bandits
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
• DRAWN APART : A Device Identification Technique based on Remote GPU Fingerprinting
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

Related

• Camel: Energy-Aware LLM Inference on Resource-Constrained Devices
• SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Serving
• DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency
• GreenLLM: SLO-Aware Dynamic Frequency Scaling for Energy-Efficient LLM Serving
• AQUA: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains