Reducing Energy Bloat in Large Model Training

2023/12/12 by Jae-Won Chung, Yile Gu, Insu Jang +3 · 1 voice
Computer Science · #Advanced Neural Network Applications #Cloud Computing and Resource Management #Parallel Computing and Optimization Techniques #cs.DC #cs.LG

paper · pdf · doi:10.1145/3694715.3695970

arxiv published 2023/12/12 · arxiv updated 2024/09/23 · openalex publication_date 2024/11/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/02

Abstract

Training large AI models on numerous GPUs consumes a massive amount of energy, making power delivery one of the largest limiting factors in building and operating datacenters for AI workloads. However, we observe that not all energy consumed during training directly contributes to end-to-end throughput; a significant portion can be removed without slowing down training. We call this portion energy bloat.

Citations

• Theoretical Improvements in Algorithmic Efficiency for Network Flow Problems
• Training Deep Nets with Sublinear Memory Cost
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp\n Minima
• Efficient Processing of Deep Neural Networks: A Tutorial and Survey
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Transformers: State-of-the-Art Natural Language Processing
• Memory-Efficient Pipeline-Parallel DNN Training
• DAPPLE: A Pipelined Data Parallel Approach for Training Large Models
• Green AI
• Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
• Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs
• Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Training
• Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model
• PipeFisher: Efficient Training of Large Language Models Using Pipelining and Fisher Information Matrices
• Chasing Low-Carbon Electricity for Practical and Sustainable DNN Training
• MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
• To better understand AI’s growing energy use, analysts need a data revolution

Cited by

• Active Inference AI Systems for Scientific Discovery

Discussions

• Perseus: Removing Energy Bloat from Large Model Training [hn, 2 points, 1 comments]