How to Train BERT with an Academic Budget

2021/04/15 by Izsak, Peter, Berchansky, Moshe, Levy, Omer · 5 citations
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2104.07705

Abstract

While large language models a la BERT are used ubiquitously in NLP, pretraining them is considered a luxury that only a few well-funded industry labs can afford. How can one train such models with a more modest budget? We present a recipe for pretraining a masked language model in 24 hours using a single low-end deep learning server. We demonstrate that through a combination of software optimizations, design choices, and hyperparameter tuning, it is possible to produce models that are competitive with BERT-base on GLUE tasks at a fraction of the original pretraining cost.

Cited by

• A Multi-fidelity Double-Delta Wing Dataset and Empirical Scaling Laws for GNN-based Aerodynamic Field Surrogate
• Sliced ReLU attention: Quasi-linear contextual expressivity via sorting
• What is the Best Sequence Length for BABYLM?
• Patent Language Model Pretraining with ModernBERT
• When a Paper Has 1000 Authors: Rethinking Citation Metrics in the Era of LLMs

Related

• Multi-node Bert-pretraining: Cost-efficient Approach
• EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets
• bert2BERT: Towards Reusable Pretrained Language Models
• Accelerated Large Batch Optimization of BERT Pretraining in 54 minutes
• RoBERTa: A Robustly Optimized BERT Pretraining Approach