A termination criterion for stochastic gradient descent for binary classification

2020/03/23 by Sina Baghal, Baghal, Sina, Courtney Paquette +3
Computer Science · #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (stat.ML) #Machine Learning and Algorithms #Optimization and Control (math.OC) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2003.10312

openalex publication_date 2020/03/23 · openalex created_date 2020/03/27 · openalex updated_date 2026/07/28

Abstract

We propose a new, simple, and computationally inexpensive termination test for constant step-size stochastic gradient descent (SGD) applied to binary classification on the logistic and hinge loss with homogeneous linear predictors. Our theoretical results support the effectiveness of our stopping criterion when the data is Gaussian distributed. This presence of noise allows for the possibility of non-separable data. We show that our test terminates in a finite number of iterations and when the noise in the data is not too large, the expected classifier at termination nearly minimizes the probability of misclassification. Finally, numerical experiments indicate for both real and synthetic data sets that our termination test exhibits a good degree of predictability on accuracy and running time.

Citations

• A Stochastic Approximation Method
• Gradient-based learning applied to document recognition
• Learning Multiple Layers of Features from Tiny Images

Related

• Stochastic Gradient Descent on Separable Data: Exact Convergence with a\n Fixed Learning Rate
• Stochastic Gradient Descent with Exponential Convergence Rates of Expected Classification Errors
• Exponential Convergence Rates of Classification Errors on Learning with\n SGD and Random Features
• On Convergence-Diagnostic based Step Sizes for Stochastic Gradient Descent
• Convergence diagnostics for stochastic gradient descent with constant step size