Accelerated Block Coordinate Proximal Gradients with Applications in High Dimensional Statistics

2017/10/15 by Tsz Kit Lau, Yuan Yao, Lau, Tsz Kit +1
Computer Science · Engineering · Mathematics · #Distributed Sensor Networks and Detection Algorithms #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Optimization and Control (math.OC) #Photoacoustic and Ultrasonic Imaging #Sparse and Compressive Sensing Techniques #Stochastic Gradient Optimization Techniques #cs.LG #math.OC #stat.ML

paper · pdf · doi:10.48550/arxiv.1710.05338

10th NIPS Workshop on Optimization for Machine Learning (NIPS 2017). 8 pages, 4 figures

openalex publication_date 2017/10/15 · arxiv created 2017/12/03 · arxiv updated 2017/12/05 · openalex created_date 2022/09/03 · openalex updated_date 2026/07/28

Abstract

Nonconvex optimization problems arise in different research fields and arouse lots of attention in signal processing, statistics and machine learning. In this work, we explore the accelerated proximal gradient method and some of its variants which have been shown to converge under nonconvex context recently. We show that a novel variant proposed here, which exploits adaptive momentum and block coordinate update with specific update rules, further improves the performance of a broad class of nonconvex problems. In applications to sparse linear regression with regularizations like Lasso, grouped Lasso, capped ℓ₁ and SCAP, the proposed scheme enjoys provable local linear convergence, with experimental justification.

Citations

• Nearly unbiased variable selection under minimax concave penalty
• Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward–backward splitting, and regularized Gauss–Seidel methods
• Proximal alternating linearized minimization for nonconvex and nonsmooth problems
• Proximal Alternating Minimization and Projection Methods for Nonconvex Problems: An Approach Based on the Kurdyka-Łojasiewicz Inequality
• A General Iterative Shrinkage and Thresholding Algorithm for Non-convex Regularized Optimization Problems

Related

• Convergence Analysis of Proximal Gradient with Momentum for Nonconvex Optimization
• An Accelerated Block Proximal Framework with Adaptive Momentum for Nonconvex and Nonsmooth Optimization
• Smoothing Proximal Gradient Methods for Nonsmooth Sparsity Constrained Optimization: Optimality Conditions and Global Convergence
• Accelerated Proximal Stochastic Dual Coordinate Ascent for Regularized Loss Minimization
• An Accelerated Proximal Coordinate Gradient Method and its Application to Regularized Empirical Risk Minimization