The Computational Limits of Deep Learning

2020/07/10 by Neil Thompson, Neil C. Thompson, Thompson, Neil C. +6 · 6 voices · 315 citations
Computer Science · Mathematics · #Artificial intelligence #Computer science #Computer security #Data science #Deep learning #Dependency (UML) #Exploit #Machine Learning and Data Classification #Machine learning #Variety (cybernetics) #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2007.05558

published in arXiv (Cornell University) (Cornell University) · 33 pages, 8 figures

openalex publication_date 2020/07/10 · arxiv created 2022/07/27 · arxiv updated 2022/07/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Deep learning's recent history has been one of achievement: from triumphing over humans in the game of Go to world-leading performance in image classification, voice recognition, translation, and other tasks. But this progress has come with a voracious appetite for computing power. This article catalogs the extent of this dependency, showing that progress across a wide variety of applications is strongly reliant on increases in computing power. Extrapolating forward this reliance reveals that progress along current lines is rapidly becoming economically, technically, and environmentally unsustainable. Thus, continued progress in these applications will require dramatically more computationally-efficient methods, which will either have to come from changes to deep learning or from moving to other machine learning methods.

Cited by

• GREEN: A lightweight architecture using learnable wavelets and Riemannian geometry for biomarker exploration with EEG signals
• A Theoretical Framework for Auxiliary-Loss-Free Load Balancing of Sparse Mixture-of-Experts in Large-Scale AI Models
• Acoustic neural networks: Identifying design principles and exploring physical feasibility
• Quantum optical neural networks using atom-cavity interactions to provide all-optical nonlinearity
• The Environmental Impacts of Machine Learning Training Keep Rising Evidencing Rebound Effect
• Single layer tiny Co⁴ outpaces GPT-2 and GPT-BERT
• From MNIST to ImageNet: Understanding the Scalability Boundaries of Differentiable Logic Gate Networks
• PreLoRA: Hybrid Pre-training of Vision Transformers with Full Training and Low-Rank Adapters
• Workload-flexible data centers reduce renewable curtailment in Europe’s net-zero energy system
• Ethics for the majority world: AI and the question of violence at scale
• Confidence-gated training for efficient early-exit neural networks
• Bilateral Distribution Compression: Reducing Both Data Size and Dimensionality
• From Membership-Privacy Leakage to Quantum Machine Unlearning
• Progressive Depth Up-scaling via Optimal Transport
• Sparsity-Driven Plasticity in Multi-Task Reinforcement Learning
• CarbonScaling: Extending Neural Scaling Laws for Carbon Footprint in Large Language Models
• From Propagator to Oscillator: The Dual Role of Symmetric Differential Equations in Neural Systems
• GPU-Accelerated Interpretable Generalization for Rapid Cyberattack Detection and Forensics
• The Generalist Brain Module: Module Repetition in Neural Networks in Light of the Minicolumn Hypothesis
• Improve Underwater Object Detection through YOLOv12 Architecture and Physics-informed Augmentation
• Compute Trends Across Three Eras of Machine Learning
• A Layered Self-Supervised Knowledge Distillation Framework for Efficient Multimodal Learning on the Edge
• Beyond Low-rank Decomposition: A Shortcut Approach for Efficient On-Device Learning
• Training-free AI for Earth Observation Change Detection using Physics Aware Neuromorphic Networks
• The Tensor Track VII: From Quantum Gravity to Artificial Intelligence
• PC-MoE: Memory-Efficient and Privacy-Preserving Collaborative Training for Mixture-of-Experts LLMs
• Telco-oRAG: Optimizing Retrieval-augmented Generation for Telecom Queries via Hybrid Retrieval and Neural Routing
• Organoid intelligence (OI): the new frontier in biocomputing and intelligence-in-a-dish
• Diffused Responsibility: Analyzing the Energy Consumption of Generative Text-to-Audio Diffusion Models
• Continuous Thought Machines
• Long-Tail Knowledge in Large Language Models: Taxonomy, Mechanisms, Interventions and Implications
• All-Optical Doubly Resonant Cavities for ReLU Function in Nanophotonic Deep Learning
• Single-shot Star-convex Polygon-based Instance Segmentation for Spatially-correlated Biomedical Objects

Discussions

• The Computational Limits of Deep Learning [hn, 168 points, 73 comments]
• The Computational Limits of Deep Learning [hn, 16 points, 0 comments]
• The Computational Limits of Deep Learning [hn, 4 points, 0 comments]
• The Computational Limits of Deep Learning [hn, 2 points, 0 comments]
• The Computational Limits of Deep Learning [hn, 1 points, 0 comments]
• The Computational Limits of Deep Learning [hn, 1 points, 0 comments]

Related

• Deep Learning: A Critical Appraisal
• How deep the machine learning can be
• Computational Power and the Social Impact of Artificial Intelligence
• Deep Learning: Computational Aspects
• Towards energy-efficient Deep Learning: An overview of energy-efficient approaches along the Deep Learning Lifecycle