Tuning Learning Rates with the Cumulative-Learning Constant

2025/04/30 by Nathan Faraj, Faraj, Nathan · 1 citation
Computer Science · #Data Stream Mining Techniques #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Data Classification #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2505.13457

openalex publication_date 2025/04/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper introduces a novel method for optimizing learning rates in machine learning. A previously unrecognized proportionality between learning rates and dataset sizes is discovered, providing valuable insights into how dataset scale influences training dynamics. Additionally, a cumulative learning constant is identified, offering a framework for designing and optimizing advanced learning rate schedules. These findings have the potential to enhance training efficiency and performance across a wide range of machine learning applications.

Cited by

• Cumulative Learning Rate Adaptation: Revisiting Path-Based Schedules for SGD and Adam

Related

• Learning Rate Perturbation: A Generic Plugin of Learning Rate Schedule towards Flatter Local Minima
• Cumulative Learning Rate Adaptation: Revisiting Path-Based Schedules for SGD and Adam
• Cyclical Learning Rates for Training Neural Networks
• How to decay your learning rate
• Revisiting Learning Rate Control