2011/05/23 by Farzad Yousefian, Angelia Nedić, Yousefian, Farzad +3 · 4 citations
Computer Science · Decision Sciences · Engineering · Mathematics · #FOS: Electrical engineering #FOS: Mathematics #Optimization and Control (math.OC) #Risk and Portfolio Optimization #Sparse and Compressive Sensing Techniques #Statistical Methods and Inference #Stochastic Gradient Optimization Techniques #Systems and Control (eess.SY) #electronic engineering #information engineering
paper · pdf · doi:10.48550/arxiv.1105.4549
openalex publication_date 2011/05/23 · openalex created_date 2022/09/19 · openalex updated_date 2026/07/28
The performance of standard stochastic approximation implementations can vary\nsignificantly based on the choice of the steplength sequence, and in general,\nlittle guidance is provided about good choices. Motivated by this gap, in the\nfirst part of the paper, we present two adaptive steplength schemes for\nstrongly convex differentiable stochastic optimization problems, equipped with\nconvergence theory. The first scheme, referred to as a recursive steplength\nstochastic approximation scheme, optimizes the error bounds to derive a rule\nthat expresses the steplength at a given iteration as a simple function of the\nsteplength at the previous iteration and certain problem parameters. This rule\nis seen to lead to the optimal steplength sequence over a prescribed set of\nchoices. The second scheme, termed as a cascading steplength stochastic\napproximation scheme, maintains the steplength sequence as a piecewise-constant\ndecreasing function with the reduction in the steplength occurring when a\nsuitable error threshold is met. In the second part of the paper, we allow for\nnondifferentiable objective and we propose a local smoothing technique that\nleads to a differentiable approximation of the function. Assuming a uniform\ndistribution on the local randomness, we establish a Lipschitzian property for\nthe gradient of the approximation and prove that the obtained Lipschitz bound\ngrows at a modest rate with problem size. This facilitates the development of\nan adaptive steplength stochastic approximation framework, which now requires\nsampling in the product space of the original measure and the artificially\nintroduced distribution. The resulting adaptive steplength schemes are applied\nto three stochastic optimization problems. We observe that both schemes perform\nwell in practice and display markedly less reliance on user-defined parameters.\n