The Cost of Training NLP Models: A Concise Overview

2020/04/19 by Or Sharir, Barak Peleg, Sharir, Or +3 · 4 voices · 7 citations
Computer Science · #Natural Language Processing Techniques #Software Engineering Research #Topic Modeling #cs.CL #cs.LG #cs.NE

paper · pdf · doi:10.48550/arxiv.2004.08900

openalex publication_date 2020/04/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We review the cost of training large-scale language models, and the drivers of these costs. The intended audience includes engineers and scientists budgeting their model-training experiments, as well as non-practitioners trying to make sense of the economics of modern-day Natural Language Processing (NLP).

Citations

• LLM-Supported Manufacturing Mapping Generation
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Scaling Laws for Neural Language Models

Cited by

• GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training
• Confidential LLM Inference: Performance and Cost Across CPU and GPU TEEs
• Artificial neurons based on antiferromagnetic auto-oscillators as a platform for neuromorphic computing
• When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset
• SLM-Bench: A Comprehensive Benchmark of Small Language Models on Environmental Impacts--Extended Version
• First-Generation Inference Accelerator Deployment at Facebook
• When a Paper Has 1000 Authors: Rethinking Citation Metrics in the Era of LLMs

Discussions

• The Cost of Training NLP Models: A Concise Overview [hn, 41 points, 5 comments]
• The Cost of Training NLP Models: A Concise Overview [hn, 5 points, 0 comments]
• The Cost of Training NLP Models: A Concise Overview [hn, 3 points, 0 comments]
• The Cost of Training NLP Models: A Concise Overview [hn, 2 points, 0 comments]

Related

• Costs to Consider in Adopting NLP for Your Business
• Position: The Most Expensive Part of an LLM should be its Training Data
• The Costly Dilemma: Generalization, Evaluation and Cost-Optimal Deployment of Large Language Models
• When Do You Need Billions of Words of Pretraining Data?
• Energy and Policy Considerations for Deep Learning in NLP