2015/05/10 by Panos Toulis, Dustin Tran, Toulis, Panos +3 · 3 citations
Computer Science · Decision Sciences · Engineering · #Advanced Bandit Algorithms Research #Computation (stat.CO) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Methodology (stat.ME) #Sparse and Compressive Sensing Techniques #Stochastic Gradient Optimization Techniques
paper · pdf · doi:10.48550/arxiv.1505.02417
openalex publication_date 2015/05/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Iterative procedures for parameter estimation based on stochastic gradient descent allow the estimation to scale to massive data sets. However, in both theory and practice, they suffer from numerical instability. Moreover, they are statistically inefficient as estimators of the true parameter value. To address these two issues, we propose a new iterative procedure termed averaged implicit SGD (AI-SGD). For statistical efficiency, AI-SGD employs averaging of the iterates, which achieves the optimal Cramér-Rao bound under strong convexity, i.e., it is an optimal unbiased estimator of the true parameter value. For numerical stability, AI-SGD employs an implicit update at each iteration, which is related to proximal operators in optimization. In practice, AI-SGD achieves competitive performance with other state-of-the-art procedures. Furthermore, it is more stable than averaging procedures that do not employ proximal updates, and is simple to implement as it requires fewer tunable hyperparameters than procedures that do employ proximal updates.