Energy and Policy Considerations for Deep Learning in NLP
2019/06/05 by Emma Strubell, Ananya Ganesh, Strubell, Emma +3 · 10 voices · 104 citations
#cs.CL
paper · pdf · doi:10.48550/arxiv.1906.02243
Abstract
Recent progress in hardware and methodology for training neural networks has ushered in a new generation of large networks trained on abundant data. These models have obtained notable gains in accuracy across many NLP tasks. However, these accuracy improvements depend on the availability of exceptionally large computational resources that necessitate similarly substantial energy consumption. As a result these models are costly to train and develop, both financially, due to the cost of hardware and electricity or cloud compute time, and environmentally, due to the carbon footprint required to fuel modern tensor processing hardware. In this paper we bring this issue to the attention of NLP researchers by quantifying the approximate financial and environmental costs of training a variety of recently successful neural network models for NLP. Based on these findings, we propose actionable recommendations to reduce costs and improve equity in NLP research and practice.
Cited by
- Lightweight Person-Place Relation Extraction from Historical Newspapers with Dependency Graphs and Proximity Features
- The Cognitive Kardashev Scale: Quantifying the Material Envelope of Civilisational Computation
- Parameter-Efficient Continual Fine-Tuning: A Survey
- A Predict-then-Schedule framework for Power Distribution Networks with AI Data Centers
- Exposure is not manifestation: measurement target and output resolution jointly determine which behavioural-faithfulness evaluator wins
- Advancing bioinformatics with language models: components, applications, and perspectives
- Democratizing AI with Small Language Models: Structured Benchmarking and Parameter-Efficient Fine-Tuning for Local Deployment
- Understanding Efficiency: Quantization, Batching, and Serving Strategies in LLM Energy Use
- DatBench: Discriminative, Faithful, and Efficient VLM Evaluations
- Continuous Autoregressive Language Models
- AI, Digital Platforms, and the New Systemic Risk
- More than Carbon: Cradle-to-Grave environmental impacts of GenAI training on the Nvidia A100 GPU
- Position: We Need An Algorithmic Understanding of Generative AI
- Misinformation by Omission: The Need for More Environmental Transparency in AI
- What Is Artificial General Intelligence?
- The Cake that is Intelligence and Who Gets to Bake it: An AI Analogy and its Implications for Participation
- From Efficiency Gains to Rebound Effects: The Problem of Jevons' Paradox in AI's Polarized Environmental Debate
- Life-Cycle Emissions of AI Hardware: A Cradle-To-Grave Approach and Generational Trends
- A robust methodology for long-term sustainability evaluation of Machine Learning models
- Taxing Artificial Intelligence
- Alignment Is Not Enough: A Relational Framework for Moral Standing in Human-AI Interaction
- Generative AI for Requirements Engineering: A Systematic Literature Review
- Exploring Budgeted Image Classification with Content-Sensitive Resource Allocation
- SPRKD: Effective Knowledge Distillation for Deep Neural Networks via Saddle Region Approximation
- Towards Nexus-Score: Metadata Gaps Limit Scholarly AI Attribution
- Opti-Q: A Constraint-Based Optimization Framework for Multi-LLM Question Planning
- Keyword Matters: Unveiling the Energy Sensitivity of On-Device LLM Prompting
- Efficient Jailbreak Mitigation Using Semantic Linear Classification in a Multi-Staged Pipeline
- CienaLLM: Generative Climate-Impact Extraction from News Articles with Autoregressive LLMs
- Few-Shot Learning of a Graph-Based Neural Network Model Without Backpropagation
- Multiscale Aggregated Hierarchical Attention (MAHA): A Game Theoretic and Optimization Driven Approach to Efficient Contextual Modeling in Large Language Models
- Fine-Grained Energy Prediction For Parallellized LLM Inference With PIE-P
- Should AI Become an Intergenerational Civil Right?
- LayerPipe2: Multistage Pipelining and Weight Recompute via Improved Exponential Moving Average for Training Neural Networks
- Efficient Text Classification with Conformal In-Context Learning
- A Theoretical Framework for Auxiliary-Loss-Free Load Balancing of Sparse Mixture-of-Experts in Large-Scale AI Models
- StructuredDNA: A Bio-Physical Framework for Energy-Aware Transformer Routing
- The Hidden AI Race: Tracking Environmental Costs of Innovation
- CACARA: Cross-Modal Alignment Leveraging a Text-Centric Approach for Cost-Effective Multimodal and Multilingual Learning
- Diagram-to-Circuit QNLP for Financial Sentiment Analysis
- In Search of Goodness: Large Scale Benchmarking of Goodness Functions for the Forward-Forward Algorithm
- Toward Sustainable Generative AI: A Scoping Review of Carbon Footprint and Environmental Impacts Across Training and Inference Stages
- Energy Scaling Laws for Diffusion Models: Quantifying Compute and Carbon Emissions in Image Generation
- Sex and age determination in European lobsters using AI-Enhanced bioacoustics
- Bytes of a Feather: Personality and Opinion Alignment Effects in Human-AI Interaction
- Chain of Summaries: Summarization Through Iterative Questioning
- Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
- Agentic AI Sustainability Assessment for Supply Chain Document Insights
- Green AI: A systematic review and meta-analysis of its definitions, lifecycle models, hardware and measurement attempts
- Bayesian Coreset Optimization for Personalized Federated Learning
- AI Progress Should Be Measured by Capability-Per-Resource, Not Scale Alone: A Framework for Gradient-Guided Resource Allocation in LLMs
- Diluting Restricted Boltzmann Machines
- Enhancing Sentiment Classification with Machine Learning and Combinatorial Fusion
- Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluation
- From Tokens to Watt-hours: Analytical Energy Estimation for LLM Inference on Modern GPUs
- Toward Carbon-Neutral Human AI: Rethinking Data, Computation, and Learning Paradigms for Sustainable Intelligence
- Memory-based Language Models: An Efficient, Explainable, and Eco-friendly Approach to Large Language Modeling
- Energy-Efficient Domain-Specific Artificial Intelligence Models and Agents: Pathways and Paradigms
- TernaryCLIP: Efficiently Compressing Vision-Language Models with Ternary Weights and Distilled Knowledge
- Optimization of the quantization of dense neural networks from an exact QUBO formulation
- Spiking Neural Network Architecture Search: A Survey
- Sparse Subnetwork Enhancement for Underrepresented Languages in Large Language Models
- CauchyNet: Compact and Data-Efficient Learning using Holomorphic Activation Functions
- AI of the People, by the People, for the People: A Social Choice Approach to Collective Control of Artificial Intelligence
- TinyTorch: Building Machine Learning Systems from First Principles
- An Alternative Trajectory for Generative AI
- The Environmental Impacts of Machine Learning Training Keep Rising Evidencing Rebound Effect
- Quantum-enhanced Computer Vision: Going Beyond Classical Algorithms
- Activation-Informed Pareto-Guided Low-Rank Compression for Efficient LLM/VLM
- Beyond Static Knowledge Messengers: Towards Adaptive, Fair, and Scalable Federated Learning for Medical AI
- Wave-PDE Nets: Trainable Wave-Equation Layers as an Alternative to Attention
- ModernVBERT: Towards Smaller Visual Document Retrievers
- BladderFormer: A Streaming Transformer for Real-Time Urological State Monitoring
- Physics Priors Offer Useful Accuracy-Carbon Trade-Offs in Spatio-Temporal Forecasting
- Progressive Weight Loading: Accelerating Initial Inference and Gradually Boosting Performance on Resource-Constrained Environments
- Temporal Poisoning: Clean-Label Backdoors via Event Redistribution in SNNs
- Modeling Decisions in Blockchain Analytics: A Leakage-Aware Evaluation of Tree-Based vs. Sequential Models
- From moral panic to pragmatic governance: reframing AI’s societal impacts in employment, education, and ethics
- Video Killed the Energy Budget: Characterizing the Latency and Power Regimes of Open Text-to-Video Models
- Towards Open-Ended Discovery for Low-Resource NLP
- RMT-KD: Random Matrix Theoretic Causal Knowledge Distillation
- Who Wins the Race? (R Vs Python) - An Exploratory Study on Energy Consumption of Machine Learning Algorithms
- An Analysis of Optimizer Choice on Energy Efficiency and Performance in Neural Network Training
- Green Recommender Systems: Understanding and Minimizing the Carbon Footprint of AI-Powered Personalization
- MetaFed: Advancing Privacy, Performance, and Sustainability in Federated Metaverse Systems
- Natural Language Satisfiability: Exploring the Problem Distribution and Evaluating Transformer-based Language Models
- Neuromorphic Intelligence
- CIFNet: An Analytic Neural Learning Framework for Efficient and Calibrated Class-Incremental Learning
- Differentiable Entropy Regularization: A Complexity-Aware Approach for Neural Optimization
- Holographic Knowledge Manifolds: A Novel Pipeline for Continual Learning Without Catastrophic Forgetting in Large Language Models
- Towards 6G Intelligence: The Role of Generative AI in Future Wireless Networks
- SLM-Bench: A Comprehensive Benchmark of Small Language Models on Environmental Impacts--Extended Version
- Learning with springs and sticks
- Novel Approaches to Artificial Intelligence Development Based on the Nearest Neighbor Method
- Comparing energy consumption and accuracy in text classification inference
- Is GPT-OSS Good? A Comprehensive Evaluation of OpenAI's Latest Open Source Models
- A Comprehensive Review of AI Agents: Transforming Possibilities in Technology and Beyond
- EMLIO: Minimizing I/O Latency and Energy Consumption for Large-Scale AI Training
- READER: Retrieval-Assisted Drafter for Efficient LLM Inference
- Position: Ideas Should be the Center of Machine Learning Research
- On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification
- FairLangProc: A Python package for fairness in NLP
- SPARTA: Advancing Sparse Attention in Spiking Neural Networks via Spike-Timing-Based Prioritization
- Compression-Induced Communication-Efficient Large Model Training and Inferencing
Discussions
- See arxiv.org/abs/1906.02243 and www.buzzsprout.com/2126417/epis... [bsky, 16 points, 3 comments]
- yep it’s from almost 6 years ago and the data is presumably at least a year older
arxiv.org/abs/1906.02243 [bsky, 10 points, 1 comments]
- Energy and Policy Considerations for Deep Learning in NLP [hn, 2 points, 0 comments]
- Energy and Policy Considerations for Deep Learning in NLP [pdf] [hn, 2 points, 0 comments]
- Receipts: arxiv.org/abs/1906.02243 arxiv.org/pdf/1906.02243 [bsky, 2 points, 1 comments]
- "The International Energy Agency (IEA) estimates that the electricity [...] in 2022 was 240–340 TWh, or 1–1.3% of world demand (if cryptocurrency mining and data-transmission infrastructure are includ [bsky, 1 points, 1 comments]
- arxiv.org/pdf/1906.022... [bsky, 1 points, 0 comments]
- Energy and Policy Considerations for Deep Learning in NLP [hn, 1 points, 0 comments]
- Laughing at a discussion about whether when one runs a long NLP job it's better to plant a tree or forego a burger to offset the carbon emissions. https://arxiv.org/abs/1906.02243 [bsky, 0 points, 0 comments]
- Next time someone is talking your ear off about how much good genAI can do for the world or some nonsense, show em this arxiv.org/abs/1906.02243v1 [bsky, 0 points, 0 comments]
Related