Riemannian stochastic recursive momentum method for non-convex optimization

2020/08/11 by Andi Han, Han, Andi, Junbin Gao +1 · 4 citations
Computer Science · Engineering · Mathematics · #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Markov Chains and Monte Carlo Methods #Optimization and Control (math.OC) #Sparse and Compressive Sensing Techniques #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2008.04555

openalex publication_date 2020/08/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose a stochastic recursive momentum method for Riemannian non-convex optimization that achieves a near-optimal complexity of O(ε^-3) to find ε-approximate solution with one sample. That is, our method requires O(1) gradient evaluations per iteration and does not require restarting with a large batch gradient, which is commonly used to obtain the faster rate. Extensive experiment results demonstrate the superiority of our proposed algorithm.

Citations

• Adam: A Method for Stochastic Optimization
• A Stochastic Approximation Method
• Gradient-based learning applied to document recognition
• Riemannian adaptive stochastic gradient algorithms on matrix manifolds
• An Alternative to EM for Gaussian Mixture Models: Batch and Stochastic\n Riemannian Optimization
• Variance reduction for Riemannian non-convex optimization with batch size adaptation
• Learning Multiple Layers of Features from Tiny Images

Cited by

• A Single-loop Stochastic Riemannian ADMM for Nonsmooth Optimization
• A Retraction-free Method for Nonsmooth Minimax Optimization over a Compact Manifold
• Efficient Optimization with Orthogonality Constraint: a Randomized Riemannian Submanifold Method
• Single-loop O(ε^-3) stochastic smoothing algorithms for nonsmooth Riemannian optimization

Related

• Single-loop O(ε^-3) stochastic smoothing algorithms for nonsmooth Riemannian optimization
• Decentralized projected Riemannian stochastic recursive momentum method for nonconvex optimization
• Convergence of a Stochastic Gradient Method with Momentum for Non-Smooth Non-Convex Optimization
• Constrained Stochastic Recursive Momentum Successive Convex Approximation
• A Single-loop Stochastic Riemannian ADMM for Nonsmooth Optimization