Generalization Properties and Implicit Regularization for Multiple Passes SGM

2016/05/26 by Junhong Lin, Raffaello Camoriano, Lin, Junhong +3
Computer Science · Engineering · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and ELM #Sparse and Compressive Sensing Techniques #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.1605.08375

openalex publication_date 2016/05/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We study the generalization properties of stochastic gradient methods for learning with convex loss functions and linearly parameterized functions. We show that, in the absence of penalizations or constraints, the stability and approximation properties of the algorithm can be controlled by tuning either the step-size or the number of passes over the data. In this view, these parameters can be seen to control a form of implicit regularization. Numerical results complement the theoretical findings.

Citations

• Stochastic Gradient Descent for Non-smooth Optimization: Convergence\n Results and Optimal Averaging Schemes

Related

• On the Origin of Implicit Regularization in Stochastic Gradient Descent
• Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensions
• Can Implicit Bias Explain Generalization? Stochastic Convex Optimization as a Case Study
• SGD: The Role of Implicit Regularization, Batch-size and Multiple-epochs
• Stochastic Gradient/Mirror Descent: Minimax Optimality and Implicit Regularization