On Constructing Confidence Region for Model Parameters in Stochastic Gradient Descent via Batch Means

2019/11/04 by Yi Zhu, Jing Dong, Zhu, Yi +1 · 1 citation
Computer Science · Mathematics · #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Markov Chains and Monte Carlo Methods #Statistical Methods and Inference #Statistics Theory (math.ST) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.1911.01483

openalex publication_date 2019/11/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we study a simple algorithm to construct asymptotically valid confidence regions for model parameters using the batch means method. The main idea is to cancel out the covariance matrix which is hard/costly to estimate. In the process of developing the algorithm, we establish process-level functional central limit theorem for Polyak-Ruppert averaging based stochastic gradient descent estimators. We also extend the batch means method to accommodate more general batch size specifications.

Cited by

• Boosting Accelerated Proximal Gradient Method with Adaptive Sampling for Stochastic Composite Optimization

Related

• On the Utility of Equal Batch Sizes for Inference in Stochastic Gradient Descent
• Statistical Inference for Model Parameters in Stochastic Gradient Descent
• Statistical Inference for Polyak-Ruppert Averaged Zeroth-order Stochastic Gradient Algorithm
• Cheap Bootstrap for Fast Uncertainty Quantification of Stochastic Gradient Descent
• Cheap Bootstrap for Fast Uncertainty Quantification of Stochastic Gradient Descent