A second-order-like optimizer with adaptive gradient scaling for deep learning

2024/10/08 by Bolte, Jérôme, Boustany, Ryan, Pauwels, Edouard +1
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Optimization and Control (math.OC)

paper · doi:10.48550/arxiv.2410.05871

Abstract

In this empirical article, we introduce INNAprop, an optimization algorithm that combines the INNA method with the RMSprop adaptive gradient scaling. It leverages second-order information and rescaling while keeping the memory requirements of standard DL methods as AdamW or SGD with momentum. After giving geometrical insights, we evaluate INNAprop on CIFAR-10, Food101, and ImageNet with ResNets, VGG, DenseNet, and ViT, and on GPT-2 (OpenWebText) train from scratch and with LoRA fine-tuning (E2E). INNAprop consistently matches or outperforms AdamW both in training speed and accuracy, with minimal hyperparameter tuning in large-scale settings. Our code is publicly available at \urlhttps://github.com/innaprop/innaprop.

Related

• Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noise
• Scalable Second Order Optimization for Deep Learning
• Read the Signs: Towards Invariance to Gradient Descent's Hyperparameter Initialization
• No More Adam: Learning Rate Scaling at Initialization is All You Need
• Optimization Methods in Deep Learning: A Comprehensive Overview