The Hardware Lottery

2020/09/14 by Sara Hooker, Hooker, Sara · 11 voices · 19 citations
#cs.CY #cs.AI #cs.AR #cs.LG

paper · pdf · doi:10.48550/arxiv.2009.06489

Abstract

Hardware, systems and algorithms research communities have historically had different incentive structures and fluctuating motivation to engage with each other explicitly. This historical treatment is odd given that hardware and software have frequently determined which research ideas succeed (and fail). This essay introduces the term hardware lottery to describe when a research idea wins because it is suited to the available software and hardware and not because the idea is superior to alternative research directions. Examples from early computer science history illustrate how hardware lotteries can delay research progress by casting successful ideas as failures. These lessons are particularly salient given the advent of domain specialized hardware which make it increasingly costly to stray off of the beaten path of research ideas. This essay posits that the gains from progress in computing are likely to become even more uneven, with certain research directions moving into the fast-lane while progress on others is further obstructed.

Cited by

• Pretraining Recurrent Networks without Recurrence
• Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks
• Exposure is not manifestation: measurement target and output resolution jointly determine which behavioural-faithfulness evaluator wins
• A Blueprint for Equilibrium-Based Differentiable Continuous-Variable Thermodynamic Computing
• Neuromorphic Computing: A Theoretical Framework for Time, Space, and Energy Scaling
• Stop treating `AGI' as the north-star goal of AI research
• Evolution and The Knightian Blindspot of Machine Learning
• Learning Nonlinear Heterogeneity in Physical Kolmogorov-Arnold Networks
• Agentic Operator Generation for ML ASICs
• HipKittens: Fast and Furious AMD Kernels
• An efficient probabilistic hardware architecture for diffusion-like models
• Scalable GPU-Accelerated Euler Characteristic Curves: Optimization and Differentiable Learning for PyTorch
• GaLLoP: Gradient-based Sparse Learning on Low-Magnitude Parameters
• The Anatomy of a Triton Attention Kernel
• LAPIS: A Performance Portable, High Productivity Compiler Framework
• Qu’est-ce que la science informatique non faite?
• Building an Open CGRA Ecosystem for Agile Innovation
• Cross-Layer Design of Vector-Symbolic Computing: Bridging Cognition and Brain-Inspired Hardware Acceleration
• Computational Economics in Large Language Models: Exploring Model Behavior and Incentive Design under Resource Constraints
• The Ubiquitous Sparse Matrix-Matrix Products

Discussions

• The Hardware Lottery [hn, 73 points, 16 comments]
• The Hardware Lottery [hn, 7 points, 0 comments]
• The Hardware Lottery [lobsters, 6 points, 0 comments]
• The Hardware Lottery [hn, 4 points, 0 comments]
• "This essay introduces the term hardware lottery to describe when a research idea wins because it is suited to the available software and hardware and not because the idea is superior to alternative r [bsky, 3 points, 1 comments]
• So many takeaways from this @sarahooker.bsky.social article on how the hardware lottery has shaped (and is shaping) our technological landscape arxiv.org/abs/2009.06489 [bsky, 3 points, 0 comments]
• The Hardware Lottery [hn, 2 points, 0 comments]
• The Hardware Lottery (2020) [hn, 1 points, 0 comments]
• Really enjoyed this 2020 article on how the parameters and constraints of computer hardware have unintended effects on the direction of computer science research, which algorithms get developed and de [bsky, 0 points, 0 comments]
• You don’t need direct access to future hardware and there are few serious hurdles. Think how pre-GPU neural net research still had a lasting influence once the hardware was available to make it scale. [bsky, 0 points, 1 comments]
• I think the hardware lottery is more fundamental than the bitter lesson in machine learning. The hardware lottery causes the bitter lesson. Based on conversations, it seems that not enough people ar [bsky, 0 points, 0 comments]

Related

• Beyond Moore's Law: Harnessing the Redshift of Generative AI with Effective Hardware-Software Co-Design
• Advancing Computing's Foundation of US Industry & Society
• Open Hardware in Science: The Benefits of Open Electronics
• Open-source Hardware: Opportunities and Challenges
• Computational Power and the Social Impact of Artificial Intelligence