Predicting Training Time Without Training

2020/08/28 by Luca Zancato, Alessandro Achille, Zancato, Luca +7 · 2 citations
Computer Science · #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2008.12478

openalex publication_date 2020/08/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We tackle the problem of predicting the number of optimization steps that a pre-trained deep network needs to converge to a given value of the loss function. To do so, we leverage the fact that the training dynamics of a deep network during fine-tuning are well approximated by those of a linearized model. This allows us to approximate the training loss and accuracy at any point during training by solving a low-dimensional Stochastic Differential Equation (SDE) in function space. Using this result, we are able to predict the time it takes for Stochastic Gradient Descent (SGD) to fine-tune a model to a given loss without having to perform any training. In our experiments, we are able to predict training time of a ResNet within a 20% error margin on a variety of datasets and hyper-parameters, at a 30 to 45-fold reduction in cost compared to actual training. We also discuss how to further reduce the computational and memory cost of our method, and in particular we show that by exploiting the spectral properties of the gradients' matrix it is possible predict training time on a large dataset while processing only a subset of the samples.

Citations

• Fine-Grained Visual Classification of Aircraft
• Database-friendly random projections: Johnson-Lindenstrauss with binary coins
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Convergence Analysis of Two-layer Neural Networks with ReLU Activation
• Deep Neural Networks as Gaussian Processes
• A Convergence Theory for Deep Learning via Over-Parameterization
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Toward a Holistic Approach to Continual Model Merging
• Efficient Fine-Grained GPU Performance Modeling for Distributed Deep Learning of LLM

Related

• LipschitzLR: Using theoretically computed adaptive learning rates for fast convergence
• Stochastic Gradient Descent: Going As Fast As Possible But Not Faster
• Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?
• A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minima
• Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks