Improving Layer-wise Adaptive Rate Methods using Trust Ratio Clipping

2020/11/27 by Jeffrey Fong, Fong, Jeffrey, Si-Wei Chen +3
Computer Science · Engineering · #Cooperative Communication and Network Coding #Error Correcting Code Techniques #FOS: Computer and information sciences #Machine Learning (cs.LG) #Wireless Communication Security Techniques

paper · pdf · doi:10.48550/arxiv.2011.13584

openalex publication_date 2020/11/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Training neural networks with large batch is of fundamental significance to deep learning. Large batch training remarkably reduces the amount of training time but has difficulties in maintaining accuracy. Recent works have put forward optimization methods such as LARS and LAMB to tackle this issue through adaptive layer-wise optimization using trust ratios. Though prevailing, such methods are observed to still suffer from unstable and extreme trust ratios which degrades performance. In this paper, we propose a new variant of LAMB, called LAMBC, which employs trust ratio clipping to stabilize its magnitude and prevent extreme values. We conducted experiments on image classification tasks such as ImageNet and CIFAR-10 and our empirical results demonstrate promising improvements across different batch sizes.

Citations

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Scalability in Perception for Autonomous Driving: Waymo Open Dataset
• Learning Multiple Layers of Features from Tiny Images

Related

• Large Batch Training Does Not Need Warmup
• Revisiting LARS for Large Batch Training Generalization of Neural Networks
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• High-Performance Large-Scale Image Recognition Without Normalization
• A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes