Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averaging

2022/09/29 by Jean Kaddour, Kaddour, Jean · 6 citations
Computer Science · #Topic Modeling #Multimodal Machine Learning Applications #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2209.14981

Abstract

Training vision or language models on large datasets can take days, if not weeks. We show that averaging the weights of the k latest checkpoints, each collected at the end of an epoch, can speed up the training progression in terms of loss and accuracy by dozens of epochs, corresponding to time savings up to ~68 and ~30 GPU hours when training a ResNet50 on ImageNet and RoBERTa-Base model on WikiText-103, respectively. We also provide the code and model checkpoint trajectory to reproduce the results and facilitate research on reusing historical weights for faster convergence.

Cited by

• How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining
• MaP: A Unified Framework for Reliable Evaluation of Pre-training Dynamics
• Training Dynamics Impact Post-Training Quantization Robustness
• EMA Without the Lag: Bias-Corrected Iterate Averaging Schemes
• FedSWA: Improving Generalization in Federated Learning with Highly Heterogeneous Data via Momentum-Based Stochastic Controlled Weight Averaging
• WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training

Related

• FFCV: Accelerating Training by Removing Data Bottlenecks
• ImageNet Training in Minutes
• Stochastic Weight Averaging in Parallel: Large-Batch Training that Generalizes Well
• One Epoch Is All You Need
• Accelerating Large Batch Training via Gradient Signal to Noise Ratio (GSNR)