AdamNX: An Adam improvement algorithm based on a novel exponential decay mechanism for the second-order moment estimate

2025/11/17 by Meng Zhu, Quan Xiao, Zhu, Meng +3
Computer Science · #Advanced Neural Network Applications #Machine Learning in Healthcare #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2511.13465

Abstract

This paper studies the exponential decay mechanism of the second-moment estimate in Adam. We propose AdamNX and a time-varying second-moment decay rate that gradually weakens the correction applied to the update scale. Under the assumptions used in our analysis, this mechanism makes the updates approach momentum-SGD-like behavior during the training plateau phase. We report results on the image-classification, object-detection, and semantic-segmentation tasks, configurations, and comparators included in this paper. These results do not establish multi-seed statistical effects, flatness, or generalization beyond the reported tasks. Our code is open-sourced at https://github.com/mengzhu0308/AdamNX.

Citations

• Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training
• Symbolic Discovery of Optimization Algorithms
• YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• ImageNet classification with deep convolutional neural networks

Related

• MC-SJD : Maximal Coupling Speculative Jacobi Decoding for Autoregressive Visual Generation Acceleration
• Speculative Jacobi-Denoising Decoding for Accelerating Autoregressive Text-to-image Generation
• GigaTok: Scaling Visual Tokenizers to 3 Billion Parameters for Autoregressive Image Generation
• MENTOR: Efficient Multimodal-Conditioned Tuning for Autoregressive Vision Generation Models
• Scalable Autoregressive Image Generation with Mamba