Effectiveness of Optimization Algorithms in Deep Image Classification

2021/10/04 by Zhaoyang Zhu, Haozhe Sun, Zhu, Zhaoyang +3
Computer Science · #Advanced Image and Video Retrieval Techniques #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Neural Networks and Applications #Optimization and Control (math.OC)

paper · pdf · doi:10.48550/arxiv.2110.01598

openalex publication_date 2021/10/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Adam is applied widely to train neural networks. Different kinds of Adam methods with different features pop out. Recently two new adam optimizers, AdaBelief and Padam are introduced among the community. We analyze these two adam optimizers and compare them with other conventional optimizers (Adam, SGD + Momentum) in the scenario of image classification. We evaluate the performance of these optimization algorithms on AlexNet and simplified versions of VGGNet, ResNet using the EMNIST dataset. (Benchmark algorithm is available at \hyperref[https://github.com/chuiyunjun/projectCSC413]https://github.com/chuiyunjun/projectCSC413).

Citations

• ImageNet classification with deep convolutional neural networks
• An overview of gradient descent optimization algorithms
• EMNIST: an extension of MNIST to handwritten letters
• PyTorch: An Imperative Style, High-Performance Deep Learning Library

Related

• Mixing ADAM and SGD: a Combined Optimization Method
• Normalized Direction-preserving Adam
• Optimization Methods in Deep Learning: A Comprehensive Overview
• AdaFamily: A family of Adam-like adaptive gradient methods
• On Suppressing Range of Adaptive Stepsizes of Adam to Improve Generalisation Performance