Efficient Over-parameterized Matrix Sensing from Noisy Measurements via Alternating Preconditioned Gradient Descent

2025/02/01 by Zhiyu Liu, Zhi Han, Liu, Zhiyu +7 · 3 citations
Earth and Planetary Sciences · Engineering · #Electrical and Bioimpedance Tomography #FOS: Computer and information sciences #FOS: Mathematics #Geophysical and Geoelectrical Methods #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Optimization and Control (math.OC) #Sparse and Compressive Sensing Techniques

paper · pdf · doi:10.48550/arxiv.2502.00463

openalex publication_date 2025/02/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We consider the noisy matrix sensing problem in the over-parameterization setting, where the estimated rank r is larger than the true rank r_⋆ of the target matrix X_⋆. Specifically, our main objective is to recover a matrix X_⋆ ∈ ℝ^{n₁ × n₂} with rank r_⋆ from noisy measurements using an over-parameterized factorization LR^\top , where L ∈ ℝ^{n₁ × r}, R ∈ ℝ^{n₂ × r} and min\n₁, n₂\ ≥ r > r_⋆ , with r_⋆ being unknown. Recently, preconditioning methods have been proposed to accelerate the convergence of matrix sensing problem compared to vanilla gradient descent, incorporating preconditioning terms (L^\top L + λI)^-1 and (R^\top R + λI)^-1 into the original gradient. However, these methods require careful tuning of the damping parameter λ and are sensitive to step size. To address these limitations, we propose the alternating preconditioned gradient descent (APGD) algorithm, which alternately updates the two factor matrices, eliminating the need for the damping parameter λ and enabling faster convergence with larger step sizes. We theoretically prove that APGD convergences to a near-optimal error at a linear rate. We further show that APGD can be extended to deal with other low-rank matrix estimation tasks, also with a theoretical guarantee of linear convergence. To validate the effectiveness and scalability of the proposed APGD, we conduct simulated and real-world experiments on a wide range of low-rank estimation problems, including noisy matrix sensing, weighted PCA, 1-bit matrix completion, and matrix completion. The extensive results demonstrate that APGD consistently achieves the fastest convergence and the lowest computation time compared to the existing alternatives.

Cited by

• RGNMR: A Gauss-Newton method for robust matrix completion with theoretical guarantees
• AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projections
• Low-Rank Adaptation Redux for Large Models

Related

• On the computational and statistical complexity of over-parameterized matrix sensing
• The Power of Preconditioning in Overparameterized Low-Rank Matrix Sensing
• How Over-Parameterization Slows Down Gradient Descent in Matrix Sensing: The Curses of Symmetry and Initialization
• Accelerating Ill-Conditioned Low-Rank Matrix Estimation via Scaled Gradient Descent
• Preconditioned Gradient Descent for Over-Parameterized Nonconvex Matrix Factorization