A Systematic Characterization of LLM Inference on GPUs

2025/12/01 by Haonan Wang, Wang, Haonan, Xiao, Xuxin +17 · 2 citations
Computer Science · Materials Science · #Advanced Neural Network Applications #Big Data and Digital Economy #FOS: Computer and information sciences #Hardware Architecture (cs.AR) #Machine Learning in Materials Science

paper · pdf · doi:10.48550/arxiv.2512.01644

openalex publication_date 2025/12/01 · openalex created_date 2025/12/03 · openalex updated_date 2026/07/28

Abstract

This work presents a systematic characterization of Large Language Model (LLM) inference to address fragmented understanding. Through comprehensive experiments, we establish a four-dimensional analytical framework: (1) Two-Phase Heterogeneity Observation; (2) Microarchitectural Root Cause Analysis; (3) System Scaling Principles; and (4) Emerging Paradigm Boundaries. Our investigation progresses systematically from observation to foresight: identifying performance phenomena, revealing hardware causes, validating system behavior, and exploring new paradigms. This study not only consolidates a reliable empirical foundation for existing research but also provides new discoveries and practical optimization guidance for LLM inference.

Citations

• EdgeReasoning: Characterizing Reasoning LLM Deployment on Edge GPUs
• Systematic Characterization of LLM Quantization: A Performance, Energy, and Quality Perspective
• Profiling Large Language Model Inference on Apple Silicon: A Quantization Perspective
• Characterizing Communication Patterns in Distributed Large Language Model Inference
• KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider
• CLONE: Customizing LLMs for Efficient Latency-Aware Inference at the Edge
• Qwen3 Technical Report
• PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applications
• A Survey on Inference Engines for Large Language Models: Perspectives on Optimization and Efficiency
• Taming the Titans: A Survey of Efficient LLM Inference Serving
• The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs
• Energy Considerations of Large Language Model Inference and Efficiency Optimizations
• Characterizing and Optimizing LLM Inference Workloads on CPU-GPU Coupled Architectures
• Characterizing Mobile SoC for Accelerating Heterogeneous LLM Inference
• Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective
• LLM-Pilot: Characterize and Optimize Performance of your LLM Inference Services
• A Review on Edge Large Language Models: Design, Execution, and Applications
• DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency
• The Llama 3 Herd of Models
• InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management
• A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models
• Characterization of Large Language Model Development in the Datacenter
• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
• LLM Inference Unveiled: Survey and Roofline Model Insights
• KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
• Splitwise: Efficient generative LLM inference using phase splitting
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
• Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline
• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• LaMDA: Language Models for Dialog Applications
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

Cited by

• Selecting Language Models for Social Science: Start Small, Start Open, and Validate
• Kernel Forge: An Agent Harness for LLM-based Generation and Optimization of CUDA Kernels
• LUT-LLM: Efficient Large Language Model Inference with Memory-based Computations on FPGAs

Related

• LLM Inference Unveiled: Survey and Roofline Model Insights
• LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Accelerators
• A Hardware Evaluation Framework for Large Language Model Inference
• LLM-Pilot: Characterize and Optimize Performance of your LLM Inference Services
• Efficient LLM inference solution on Intel GPU