Weighted Averaged Stochastic Gradient Descent: Asymptotic Normality and Optimality

2023/07/13 by Ziyang Wei, Wei, Ziyang, Wanrong Zhu +3 · 3 citations
Computer Science · Engineering · Mathematics · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Sparse and Compressive Sensing Techniques #Statistical Methods and Inference #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2307.06915

openalex publication_date 2023/07/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Stochastic Gradient Descent (SGD) is one of the most popular algorithms in statistical and machine learning due to its computational and memory efficiency. Various averaging schemes have been proposed to accelerate the convergence of SGD in different settings. In this paper, we explore a general averaging scheme for SGD. Specifically, we establish the asymptotic normality of a broad range of weighted averaged SGD solutions and provide asymptotically valid online inference approaches. Furthermore, we propose an adaptive averaging scheme that exhibits both optimal statistical rate and favorable non-asymptotic convergence, drawing insights from the optimal weight for the linear model in terms of non-asymptotic mean squared error (MSE).

Cited by

• Online Statistical Inference of Constrained Stochastic Optimization via Random Scaling
• Statistical Inference for Online Algorithms
• Sharp Gaussian approximations for Decentralized Federated Learning

Related

• Revisiting SGD with Increasingly Weighted Averaging: Optimization and Generalization Perspectives
• Stochastic Gradient Descent for Non-smooth Optimization: Convergence Results and Optimal Averaging Schemes
• Acceleration of stochastic gradient descent with momentum by averaging: finite-sample rates and asymptotic normality
• Optimized convergence of stochastic gradient descent by weighted averaging
• A Unified Analysis for Finite Weight Averaging