From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference
2023/10/04 by Siddharth Samsi, Samsi, Siddharth, Dan Zhao +18 · 10 voices · 63 citations
Computer Science · #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling #cs.CL #cs.DC
paper · pdf · doi:10.48550/arxiv.2310.03003
openalex publication_date 2023/10/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Abstract
Large language models (LLMs) have exploded in popularity due to their new generative capabilities that go far beyond prior state-of-the-art. These technologies are increasingly being leveraged in various domains such as law, finance, and medicine. However, these models carry significant computational challenges, especially the compute and energy costs required for inference. Inference energy costs already receive less attention than the energy costs of training LLMs -- despite how often these large models are called on to conduct inference in reality (e.g., ChatGPT). As these state-of-the-art LLMs see increasing usage and deployment in various domains, a better understanding of their resource utilization is crucial for cost-savings, scaling performance, efficient hardware usage, and optimal inference strategies. In this paper, we describe experiments conducted to study the computational and energy utilization of inference with LLMs. We benchmark and conduct a preliminary analysis of the inference performance and inference energy costs of different sizes of LLaMA -- a recent state-of-the-art LLM -- developed by Meta AI on two generations of popular GPUs (NVIDIA V100 & A100) and two datasets (Alpaca and GSM8K) to reflect the diverse set of tasks/benchmarks for LLMs in research and practice. We present the results of multi-node, multi-GPU inference using model sharding across up to 32 GPUs. To our knowledge, our work is the one of the first to study LLM inference performance from the perspective of computational and energy resources at this scale.
Cited by
- Efficient Clustering with Provable Guardrails for LLM Inference at Scale
- Attention to Detail: Evaluating Energy, Performance, and Accuracy Trade-offs Across vLLM Configurations
- Non-Markovianity and memory enhancement in quantum reservoir computing
- Understanding Efficiency: Quantization, Batching, and Serving Strategies in LLM Energy Use
- Energy use of AI inference, efficiency pathways, and test-time scaling
- Selecting Language Models for Social Science: Start Small, Start Open, and Validate
- Large Language Models as a (Bad) Security Norm in the Context of Regulation and Compliance
- ELANA: A Simple Energy and Latency Analyzer for LLMs
- Exclusive Control of Quantum Memory Erasure
- TokenPowerBench: Benchmarking the Power Consumption of LLM Inference
- Copyright Detection in Large Language Models: An Ethical Approach to Generative AI Development
- Toward Sustainable Generative AI: A Scoping Review of Carbon Footprint and Environmental Impacts Across Training and Inference Stages
- Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
- LoopLLM: Transferable Energy-Latency Attacks in LLMs via Repetitive Generation
- Smart but Costly? Benchmarking LLMs on Functional Accuracy and Energy Efficiency
- Characterizing and Understanding Energy Footprint and Efficiency of Small Language Model on Edges
- Promoting Sustainable Web Agents: Benchmarking and Estimating Energy Consumption through Empirical and Theoretical Analysis
- From Prompts to Power: Measuring the Energy Footprint of LLM Inference
- EvalCards: A Framework for Standardized Evaluation Reporting
- FREESH: Fair, Resource- and Energy-Efficient Scheduling for LLM Serving on Heterogeneous GPUs
- Advancing AI Challenges for the United States Department of the Air Force
- From Tokens to Watt-hours: Analytical Energy Estimation for LLM Inference on Modern GPUs
- The environmental impact of ICT in the era of data and artificial intelligence
- Key and Value Weights Are Probably All You Need: On the Necessity of the Query, Key, Value weight Triplet in Decoder-Only Transformers
- FastVLM: Self-Speculative Decoding for Fast Vision-Language Model Inference
- Fluidity Index: Next-Generation Super-intelligence Benchmarks
- An Evaluation of LLMs Inference on Popular Single-board Computers
- Zero-Shot Performance Prediction for Probabilistic Scaling Laws
- Auditing Pay-Per-Token in Large Language Models
- Energy Efficiency in Cloud-Based Big Data Processing for Earth Observation: Gap Analysis and Future Directions
- Rearchitecting Datacenter Lifecycle for AI: A TCO-Driven Framework
- Artificial intelligence in dermatology: Clinical promise and environmental impact
- Beyond Greedy Exits: Improved Early Exit Decisions for Risk Control and Reliability
- A Computational Perspective on NeuroAI and Synthetic Biological Intelligence
- You Only Measure Once: On Designing Single-Shot Quantum Machine Learning Models
- LightCode: Compiling LLM Inference for Photonic-Electronic Systems
- CompAir: Synergizing Complementary PIMs and In-Transit NoC Computation for Efficient LLM Acceleration
- A brain-inspired paradigm for scalable quantum vision
- Less is More Tokens: Efficient Math Reasoning via Difficulty-Aware Chain-of-Thought Distillation
- Systolic Array-based Architecture for Low-Bit Integerized Vision Transformers
- Measuring the environmental impact of delivering AI at Google Scale
- Comparing energy consumption and accuracy in text classification inference
- ToxiFrench: Benchmarking and Enhancing Language Models via CoT Fine-Tuning for French Toxicity Detection
- Meta-Metrics and Best Practices for System-Level Inference Performance Benchmarking
- Semantic Caching for Low-Cost LLM Serving: From Offline Learning to Online Adaptation
- LOST: Low-rank and Sparse Pre-training for Large Language Models
- Agentic Satellite-Augmented Low-Altitude Economy and Terrestrial Networks: A Survey on Generative Approaches
- The Generative Energy Arena (GEA): Incorporating Energy Awareness in Large Language Model (LLM) Human Evaluations
- Quantifying the Energy Consumption and Carbon Emissions of LLM Inference via Simulations
- System-performance and cost modeling of Large Language Model training and inference
- Accurate and Energy Efficient: Local Retrieval-Augmented Generation Models Outperform Commercial Large Language Models in Medical Tasks
- Is There a Case for Conversation Optimized Tokenizers in Large Language Models?
- An entropy-optimal path to humble AI
- CKD-EHR:Clinical Knowledge Distillation for Electronic Health Records
- Empirically-Calibrated H100 Node Power Models for Reducing Uncertainty in AI Training Energy Estimation
- Tokenization Multiplicity Leads to Arbitrary Price Variation in LLM-as-a-service
- Perturbative Gradient Training: A novel training paradigm for bridging the gap between deep neural networks and physical reservoir computing
- Cascadia: An Efficient Cascade Serving System for Large Language Models
- Learning to Insert [PAUSE] Tokens for Better Reasoning
- SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decoupling
- A Platform for Investigating Public Health Content with Efficient Concern Classification
- Energy Considerations for Large Pretrained Neural Networks
- ReTern: Exploiting Natural Redundancy and Sign Transformations for Enhanced Fault Tolerance in Compute-in-Memory based Ternary LLMs
Discussions
- Most of the talk around AI and energy use refer to an older 2020 estimate of GPT-3 energy consumption, but a more recent paper directly measures energy use of Llama 65B as 3-4 joules per decoded token [bsky, 183 points, 9 comments]
- Very interesting paper - running a 65B parameter large language model with the minimum hardware setting takes the same power as a microwave oven (600W-1000W) and gives you 200 words per second. Scalin [bsky, 4 points, 0 comments]
- Benchmarking the Energy Costs of Large Language Model Inference (2023) [hn, 2 points, 0 comments]
- From Words to Watts: Benchmarking the Energy Costs of LLM Inference (2023) [hn, 1 points, 0 comments]
- Hard to find super specific numbers, but 4j per token seems to be a common and reasonable eating, which would mean most chats are a few Wh or less. This paper goes into it a bit more - arxiv.org/pdf/2 [bsky, 1 points, 0 comments]
- If you haven't seen any numbers, it's because you haven't looked. This was the top result in a search for "LLM inference energy consumption." arxiv.org/abs/2310.03003 The notion that multiplying a bil [bsky, 1 points, 1 comments]
- 340 TWh (~1% de la producción mundial) consumidos por datacenters al año, la training run de Gpt4 son ~7200 MWh. 0.4Wh por inferencia arxiv.org/pdf/2310.03003 (en los modelos que no son de chain of t [bsky, 1 points, 1 comments]
- From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference arxiv.org/abs/2310.03003 [bsky, 0 points, 0 comments]
- Agents hallucinate. The GPU screams. You pay for the fantasy. That 4 Joules per token? It's not a bug, it's the gig economy. We burn electricity to generate confidence in lies. The task fails, the bil [bsky, 0 points, 0 comments]
- "From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference" Samsi & Zhao et al. arxiv.org/pdf/2310.03003 tldr; 6 J par token (Ws) [bsky, 0 points, 0 comments]
Related