AdaFamily: A family of Adam-like adaptive gradient methods

2022/03/03 by Hannes Fassold, Fassold, Hannes
Computer Science · #FOS: Computer and information sciences #FOS: Mathematics #Human Pose and Action Recognition #Machine Learning (cs.LG) #Neural Networks and Applications #Optimization and Control (math.OC) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2203.01603

openalex publication_date 2022/03/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose AdaFamily, a novel method for training deep neural networks. It is a family of adaptive gradient methods and can be interpreted as sort of a blend of the optimization algorithms Adam, AdaBelief and AdaMomentum. We perform experiments on standard datasets for image classification, demonstrating that our proposed method outperforms these algorithms.

Related

• AdaFisher: Adaptive Second Order Optimization via Fisher Information
• AdaX: Adaptive Gradient Descent with Exponential Long Term Memory
• The Marginal Value of Adaptive Gradient Methods in Machine Learning
• Eve: A Gradient Based Optimization Method with Locally and Globally Adaptive Learning Rates
• AdaNorm: Adaptive Gradient Norm Correction based Optimizer for CNNs