Energy and Policy Considerations for Deep Learning in NLP
2019/06/05 by Emma Strubell, Ananya Ganesh, Strubell, Emma +3 · 10 voices · 182 citations
Computer Science · #cs.CL
paper · pdf · doi:10.48550/arxiv.1906.02243
In the 57th Annual Meeting of the Association for Computational Linguistics (ACL). Florence, Italy. July 2019
arxiv created 2019/06/05 · arxiv updated 2019/06/07
Abstract
Recent progress in hardware and methodology for training neural networks has ushered in a new generation of large networks trained on abundant data. These models have obtained notable gains in accuracy across many NLP tasks. However, these accuracy improvements depend on the availability of exceptionally large computational resources that necessitate similarly substantial energy consumption. As a result these models are costly to train and develop, both financially, due to the cost of hardware and electricity or cloud compute time, and environmentally, due to the carbon footprint required to fuel modern tensor processing hardware. In this paper we bring this issue to the attention of NLP researchers by quantifying the approximate financial and environmental costs of training a variety of recently successful neural network models for NLP. Based on these findings, we propose actionable recommendations to reduce costs and improve equity in NLP research and practice.
Cited by
- Lightweight Person-Place Relation Extraction from Historical Newspapers with Dependency Graphs and Proximity Features
- The Cognitive Kardashev Scale: Quantifying the Material Envelope of Civilisational Computation
- Parameter-Efficient Continual Fine-Tuning: A Survey
- A Predict-then-Schedule framework for Power Distribution Networks with AI Data Centers
- Exposure is not manifestation: measurement target and output resolution jointly determine which behavioural-faithfulness evaluator wins
- Advancing bioinformatics with language models: components, applications, and perspectives
- Democratizing AI with Small Language Models: Structured Benchmarking and Parameter-Efficient Fine-Tuning for Local Deployment
- Understanding Efficiency: Quantization, Batching, and Serving Strategies in LLM Energy Use
- DatBench: Discriminative, Faithful, and Efficient VLM Evaluations
- Continuous Autoregressive Language Models
- AI, Digital Platforms, and the New Systemic Risk
- More than Carbon: Cradle-to-Grave environmental impacts of GenAI training on the Nvidia A100 GPU
- Position: We Need An Algorithmic Understanding of Generative AI
- Misinformation by Omission: The Need for More Environmental Transparency in AI
- What Is Artificial General Intelligence?
- The Cake that is Intelligence and Who Gets to Bake it: An AI Analogy and its Implications for Participation
- From Efficiency Gains to Rebound Effects: The Problem of Jevons' Paradox in AI's Polarized Environmental Debate
- Life-Cycle Emissions of AI Hardware: A Cradle-To-Grave Approach and Generational Trends
- A robust methodology for long-term sustainability evaluation of Machine Learning models
- Taxing Artificial Intelligence
- Alignment Is Not Enough: A Relational Framework for Moral Standing in Human-AI Interaction
- Generative AI for Requirements Engineering: A Systematic Literature Review
- Exploring Budgeted Image Classification with Content-Sensitive Resource Allocation
- SPRKD: Effective Knowledge Distillation for Deep Neural Networks via Saddle Region Approximation
- Towards Nexus-Score: Metadata Gaps Limit Scholarly AI Attribution
- Opti-Q: A Constraint-Based Optimization Framework for Multi-LLM Question Planning
- Keyword Matters: Unveiling the Energy Sensitivity of On-Device LLM Prompting
- Efficient Jailbreak Mitigation Using Semantic Linear Classification in a Multi-Staged Pipeline
- CienaLLM: Generative Climate-Impact Extraction from News Articles with Autoregressive LLMs
- Few-Shot Learning of a Graph-Based Neural Network Model Without Backpropagation
- Multiscale Aggregated Hierarchical Attention (MAHA): A Game Theoretic and Optimization Driven Approach to Efficient Contextual Modeling in Large Language Models
- Fine-Grained Energy Prediction For Parallellized LLM Inference With PIE-P
- Should AI Become an Intergenerational Civil Right?
- LayerPipe2: Multistage Pipelining and Weight Recompute via Improved Exponential Moving Average for Training Neural Networks
- Efficient Text Classification with Conformal In-Context Learning
- A Theoretical Framework for Auxiliary-Loss-Free Load Balancing of Sparse Mixture-of-Experts in Large-Scale AI Models
- StructuredDNA: A Bio-Physical Framework for Energy-Aware Transformer Routing
- The Hidden AI Race: Tracking Environmental Costs of Innovation
- CACARA: Cross-Modal Alignment Leveraging a Text-Centric Approach for Cost-Effective Multimodal and Multilingual Learning
- Diagram-to-Circuit QNLP for Financial Sentiment Analysis
- In Search of Goodness: Large Scale Benchmarking of Goodness Functions for the Forward-Forward Algorithm
- Toward Sustainable Generative AI: A Scoping Review of Carbon Footprint and Environmental Impacts Across Training and Inference Stages
- Energy Scaling Laws for Diffusion Models: Quantifying Compute and Carbon Emissions in Image Generation
- Sex and age determination in European lobsters using AI-Enhanced bioacoustics
- Bytes of a Feather: Personality and Opinion Alignment Effects in Human-AI Interaction
- Chain of Summaries: Summarization Through Iterative Questioning
- Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
- Agentic AI Sustainability Assessment for Supply Chain Document Insights
- Green AI: A systematic review and meta-analysis of its definitions, lifecycle models, hardware and measurement attempts
- Bayesian Coreset Optimization for Personalized Federated Learning
- AI Progress Should Be Measured by Capability-Per-Resource, Not Scale Alone: A Framework for Gradient-Guided Resource Allocation in LLMs
- Diluting Restricted Boltzmann Machines
- Enhancing Sentiment Classification with Machine Learning and Combinatorial Fusion
- Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluation
- From Tokens to Watt-hours: Analytical Energy Estimation for LLM Inference on Modern GPUs
- Toward Carbon-Neutral Human AI: Rethinking Data, Computation, and Learning Paradigms for Sustainable Intelligence
- Memory-based Language Models: An Efficient, Explainable, and Eco-friendly Approach to Large Language Modeling
- Energy-Efficient Domain-Specific Artificial Intelligence Models and Agents: Pathways and Paradigms
- TernaryCLIP: Efficiently Compressing Vision-Language Models with Ternary Weights and Distilled Knowledge
- Optimization of the quantization of dense neural networks from an exact QUBO formulation
- Spiking Neural Network Architecture Search: A Survey
- Sparse Subnetwork Enhancement for Underrepresented Languages in Large Language Models
- CauchyNet: Compact and Data-Efficient Learning using Holomorphic Activation Functions
- AI of the People, by the People, for the People: A Social Choice Approach to Collective Control of Artificial Intelligence
- TinyTorch: Building Machine Learning Systems from First Principles
- An Alternative Trajectory for Generative AI
- The Environmental Impacts of Machine Learning Training Keep Rising Evidencing Rebound Effect
- Quantum-enhanced Computer Vision: Going Beyond Classical Algorithms
- Activation-Informed Pareto-Guided Low-Rank Compression for Efficient LLM/VLM
- Beyond Static Knowledge Messengers: Towards Adaptive, Fair, and Scalable Federated Learning for Medical AI
- Wave-PDE Nets: Trainable Wave-Equation Layers as an Alternative to Attention
- ModernVBERT: Towards Smaller Visual Document Retrievers
- BladderFormer: A Streaming Transformer for Real-Time Urological State Monitoring
- Physics Priors Offer Useful Accuracy-Carbon Trade-Offs in Spatio-Temporal Forecasting
- Progressive Weight Loading: Accelerating Initial Inference and Gradually Boosting Performance on Resource-Constrained Environments
- Temporal Poisoning: Clean-Label Backdoors via Event Redistribution in SNNs
- Modeling Decisions in Blockchain Analytics: A Leakage-Aware Evaluation of Tree-Based vs. Sequential Models
- From moral panic to pragmatic governance: reframing AI’s societal impacts in employment, education, and ethics
- Video Killed the Energy Budget: Characterizing the Latency and Power Regimes of Open Text-to-Video Models
- Towards Open-Ended Discovery for Low-Resource NLP
- RMT-KD: Random Matrix Theoretic Causal Knowledge Distillation
- Who Wins the Race? (R Vs Python) - An Exploratory Study on Energy Consumption of Machine Learning Algorithms
- An Analysis of Optimizer Choice on Energy Efficiency and Performance in Neural Network Training
- Green Recommender Systems: Understanding and Minimizing the Carbon Footprint of AI-Powered Personalization
- MetaFed: Advancing Privacy, Performance, and Sustainability in Federated Metaverse Systems
- Natural Language Satisfiability: Exploring the Problem Distribution and Evaluating Transformer-based Language Models
- Neuromorphic Intelligence
- CIFNet: An Analytic Neural Learning Framework for Efficient and Calibrated Class-Incremental Learning
- Differentiable Entropy Regularization: A Complexity-Aware Approach for Neural Optimization
- Holographic Knowledge Manifolds: A Novel Pipeline for Continual Learning Without Catastrophic Forgetting in Large Language Models
- Towards 6G Intelligence: The Role of Generative AI in Future Wireless Networks
- SLM-Bench: A Comprehensive Benchmark of Small Language Models on Environmental Impacts--Extended Version
- Learning with springs and sticks
- Novel Approaches to Artificial Intelligence Development Based on the Nearest Neighbor Method
- Comparing energy consumption and accuracy in text classification inference
- Is GPT-OSS Good? A Comprehensive Evaluation of OpenAI's Latest Open Source Models
- A Comprehensive Review of AI Agents: Transforming Possibilities in Technology and Beyond
- EMLIO: Minimizing I/O Latency and Energy Consumption for Large-Scale AI Training
- READER: Retrieval-Assisted Drafter for Efficient LLM Inference
- Position: Ideas Should be the Center of Machine Learning Research
- Sustainable Techniques to Improve Data Quality for Training Image-Based Explanatory Models for Recommender Systems
- On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification
- FairLangProc: A Python package for fairness in NLP
- SPARTA: Advancing Sparse Attention in Spiking Neural Networks via Spike-Timing-Based Prioritization
- A Parallel Alternative for Energy-Efficient Neural Network Training and Inferencing
- Sustainable AI Training via Hardware-Software Co-Design on NVIDIA, AMD, and Emerging GPU Architectures
- Advancing Dialectal Arabic to Modern Standard Arabic Machine Translation
- The Carbon Cost of Conversation, Sustainability in the Age of Language Models
- Knowledge Grafting: A Mechanism for Optimizing AI Model Deployment in Resource-Constrained Environments
- Towards Sustainability Model Cards
- A Generative Paradigm for Environmental Science: From Prediction to Immersive Scenarios for Participatory Stewardship
- Low-resource domain adaptation while minimizing energy and hardware resource consumption
- SystolicAttention: Fusing FlashAttention within a Single Systolic Array
- PRM-Free Security Alignment of Large Models via Red Teaming and Adversarial Training
- The Hidden Costs of AI: A Review of Energy, E-Waste, and Inequality in Model Development
- PEFT of SLM for Telecommunications Customer Support: A Comparative Study of LoRA Configurations with Energy Consumption Analysis
- A Mathematical Theory of Discursive Networks
- The AI Shadow War: SaaS vs. Edge Computing Architectures
- Evolution without Large Models: Training Language Model with Task Principles
- Activation Steering for Chain-of-Thought Compression
- The Generalist Brain Module: Module Repetition in Neural Networks in Light of the Minicolumn Hypothesis
- Leveraging the Potential of Prompt Engineering for Hate Speech Detection in Low-Resource Languages
- Residual Matrix Transformers: Scaling the Size of the Residual Stream
- LinguaSynth: Heterogeneous Linguistic Signals for News Classification
- Accurate and Energy Efficient: Local Retrieval-Augmented Generation Models Outperform Commercial Large Language Models in Medical Tasks
- RobustiPy: An efficient next generation multiversal library with model selection, averaging, resampling, and explainable artificial intelligence
- Empirically-Calibrated H100 Node Power Models for Reducing Uncertainty in AI Training Energy Estimation
- Dagstuhl Perspectives Workshop 24352 -- Conversational Agents: A Framework for Evaluation (CAFE): Manifesto
- Large Language Models for History, Philosophy, and Sociology of Science: Interpretive Uses, Methodological Challenges, and Critical Perspectives
- How do datasets, developers, and models affect biases in a low-resourced language?
- The World of AI: A Novel Approach to AI Literacy for First-year Engineering Students
- Energentic Intelligence: From Self-Sustaining Systems to Enduring Artificial Life
- Survey on the Evaluation of Generative Models in Music
- A Heuristic Perspective on Debiasing Language Models
- SysLLMatic: Large Language Models are Software System Optimizers
- Translate With Care: Addressing Gender Bias, Neutrality, and Reasoning in Large Language Model Translations
- Eye of Judgement: Dissecting the Evaluation of Russian-speaking LLMs with POLLUX
- Buffer-free Class-Incremental Learning with Out-of-Distribution Detection
- Risks of AI-driven product development and strategies for their mitigation
- Responsible Data Stewardship: Generative AI and the Digital Waste Problem
- Generating HomeAssistant Automations Using an LLM-based Chatbot
- Climate Implications of Diffusion-based Generative Visual AI Systems and their Mass Adoption
- AI for Regulatory Affairs: Balancing Accuracy, Interpretability, and Computational Cost in Medical Device Classification
- BanglaByT5: Byte-Level Modelling for Bangla
- Laplace Sample Information: Data Informativeness Through a Bayesian Lens
- Breaking Bad Tokens: Detoxification of LLMs Using Sparse Autoencoders
- Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training
- Security practices in AI development
- Multi-Dimensional Assessment for AI Cognition (MAAC): A Theoretical Framework for Process-Oriented Cognitive Evaluation of Text-Based AI Systems
- F-WANDA: Fisher-Reweighted Post-Training Pruning for Sustainable Deployment of Large Language Models
- Uncovering Magnetic Phases with Synthetic Data and Physics-Informed Training
- Toward a Public and Secure Generative AI: A Comparative Analysis of Open and Closed LLMs
- Demystifying AI Agents: The Final Generation of Intelligence
- Interpretability-Guided Soft Pruning of Attention Heads in Vision Transformers
- Request-Level Energy Attribution for Batched LLM Serving
- Certificates without Electrons? Theory and Evidence on Impacts from AI-Driven Power Demand
- Efficient Unstructured Pruning of Mamba State-Space Models for Resource-Constrained Environments
- Energy Efficiency of Locally Deployed LLMs: A Preliminary Quantitative GPU Power Benchmark on Consumer Hardware
- Exploring Anthropomorphism in Conversational Agents for Environmental Sustainability
- Bridging AI and Carbon Capture: A Dataset for LLMs in Ionic Liquids and CBE Research
- Life Cycle Assessment of Pre-training the Lucie 7B Open-Source Large Language Model on the Jean Zay Supercomputer
- Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities
- Don't be lazy: CompleteP enables compute-efficient deep transformers
- Sustainability Is Not Linear: Quantifying Performance, Energy, and Privacy Trade-offs in On-Device Intelligence
- Sustainable Hybrid Document-Routed Retrieval for Financial RAG: Resolving the Robustness-Precision Trade-off
- On the retraining frequency of global models in retail demand forecasting
- Long-Tail Knowledge in Large Language Models: Taxonomy, Mechanisms, Interventions and Implications
- BIRDS: Characterizing and Understanding Biodiversity Impact of Large Language Model Serving
- Energy per Successful Goal: Goal-Level Energy Accounting for Agentic AI Systems
- MADP: A Multi-Agent Pipeline for Sustainable Document Processing with Human-in-the-Loop
- Separating Intelligence from Inference: A Standard for Edge-Native AI Computing
- Transformer Scalability Crisis: The First Comprehensive Empirical Analysis of Performance Walls in Modern Language Models
- The Hidden Cost of Thinking: Energy Use and Environmental Impact of LMs Beyond Pretraining
- Photons = Tokens: The Physics of AI and the Economics of Knowledge
- How Hyper-Datafication Impacts the Sustainability Costs in Frontier AI
- Performance Smells in ML and Non-ML Python Projects: A Comparative Study
- DeepInsert: Early Layer Bypass for Efficient and Performant Multimodal Understanding
- Representation-Aware Unlearning via Activation Signatures: From Suppression to Entity-Signature Erasure
- Large Language Models and Social Media Information Integrity: Opportunities, Challenges, and Research Directions
- Multilingual Scholarly Publishing and Artificial Intelligence Translation Tools: Weighing Social Justice and Climate Justice
- W-PCA Based Gradient-Free Proxy for Efficient Search of Lightweight Language Models
- Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory Constraints
- Exponential Shift: Humans Adapt to AI Economies
Discussions
- See arxiv.org/abs/1906.02243 and www.buzzsprout.com/2126417/epis... [bsky, 16 points, 3 comments]
- yep it’s from almost 6 years ago and the data is presumably at least a year older
arxiv.org/abs/1906.02243 [bsky, 10 points, 1 comments]
- Energy and Policy Considerations for Deep Learning in NLP [hn, 2 points, 0 comments]
- Energy and Policy Considerations for Deep Learning in NLP [pdf] [hn, 2 points, 0 comments]
- Receipts: arxiv.org/abs/1906.02243 arxiv.org/pdf/1906.02243 [bsky, 2 points, 1 comments]
- "The International Energy Agency (IEA) estimates that the electricity [...] in 2022 was 240–340 TWh, or 1–1.3% of world demand (if cryptocurrency mining and data-transmission infrastructure are includ [bsky, 1 points, 1 comments]
- arxiv.org/pdf/1906.022... [bsky, 1 points, 0 comments]
- Energy and Policy Considerations for Deep Learning in NLP [hn, 1 points, 0 comments]
- Laughing at a discussion about whether when one runs a long NLP job it's better to plant a tree or forego a burger to offset the carbon emissions. https://arxiv.org/abs/1906.02243 [bsky, 0 points, 0 comments]
- Next time someone is talking your ear off about how much good genAI can do for the world or some nonsense, show em this arxiv.org/abs/1906.02243v1 [bsky, 0 points, 0 comments]
Related