Improved Optimization of Finite Sums with Minibatch Stochastic Variance Reduced Proximal Iterations

2017/06/21 by Jialei Wang, Tong Zhang, Wang, Jialei +1 · 2 citations
Computer Science · Decision Sciences · Engineering · #Stochastic Gradient Optimization Techniques #Advanced Bandit Algorithms Research #Sparse and Compressive Sensing Techniques

paper · pdf · doi:10.48550/arxiv.1706.07001

Abstract

We present novel minibatch stochastic optimization methods for empirical risk minimization problems, the methods efficiently leverage variance reduced first-order and sub-sampled higher-order information to accelerate the convergence speed. For quadratic objectives, we prove improved iteration complexity over state-of-the-art under reasonable assumptions. We also provide empirical evidence of the advantages of our method compared to existing approaches in the literature.

Citations

• First-order methods of smooth convex optimization with inexact oracle
• Optimization Methods for Large-Scale Machine Learning

Cited by

• Trajectory Normalized Gradients for Distributed Optimization
• Simultaneous Inference for Massive Data: Distributed Bootstrap

Related

• Stochastic Variance-Reduced Newton: Accelerating Finite-Sum Minimization with Large Batches
• Variance Reduction for Faster Non-Convex Optimization
• Un-regularizing: approximate proximal point and faster stochastic\n algorithms for empirical risk minimization
• Variance Reduction and Low Sample Complexity in Stochastic Optimization via Proximal Point Method
• Nonconvex Variance Reduced Optimization with Arbitrary Sampling