Distributed Retraction-Free and Communication-Efficient Optimization on the Stiefel Manifold

2025/06/03 by Peijin Li, Song, Yilong, Bin Gao +4
Computer Science · #FOS: Mathematics #Face and Expression Recognition #Optimization and Control (math.OC)

paper · pdf · doi:10.48550/arxiv.2506.02879

openalex publication_date 2025/06/03 · openalex created_date 2025/10/14 · openalex updated_date 2026/07/28

Abstract

Optimization problems on the Stiefel manifold, ranging from principal component analysis to enhancing neural network robustness, are ubiquitous in machine learning. The Landing algorithm avoids computationally expensive retraction operations on manifolds, making it highly competitive for large-scale problems. This paper extends this method to distributed settings, introducing *EF-Landing*, the first retraction-free and communication-efficient algorithm for distributed stochastic optimization on the Stiefel manifold. By incorporating communication compression and error feedback, EF-Landing ensures convergence and constraint feasibility while significantly reducing communication overhead. We provide sharp convergence guarantees, demonstrating that EF-Landing achieves the same asymptotic linear speedup convergence rate as existing methods without communication compression. Furthermore, our analysis is highly versatile, applying to both deterministic and stochastic settings and encompassing algorithms based on gradient descent or momentum-based gradient descent. We also generalize EF-Landing to operate on block-wise Stiefel manifolds, enabling greater flexibility for structured constraints. Extensive numerical experiments validate our theoretical results.

Citations

• An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architectures
• Improving the communication in decentralized manifold optimization through single-step consensus and compression
• Distributed Riemannian Stochastic Gradient Tracking Algorithm on the Stiefel Manifold
• Retraction-Free Decentralized Non-convex Optimization with Orthogonal Constraints
• Optimization without Retraction on the Random Generalized Stiefel Manifold
• Unbiased Compression Saves Communication in Distributed Optimization: When and How Much?
• Momentum Provably Improves Error Feedback!
• Infeasible Deterministic, Stochastic, and Variance-Reduction Algorithms for Optimization under Orthogonality Constraints
• Lower Bounds and Nearly Optimal Algorithms in Distributed Learning with Communication Compression
• EF21: A New, Simpler, Theoretically Better, and Practically Faster Error Feedback
• Skew Orthogonal Convolutions
• A Penalty-free Infeasible Approach for a Class of Nonsmooth Optimization Problems over the Stiefel Manifold
• Fast and accurate optimization on the orthogonal manifold without\n retraction
• Decentralized Riemannian Gradient Descent on the Stiefel Manifold
• From Nesterov's Estimate Sequence to Riemannian Acceleration
• Error Feedback Fixes SignSGD and other Gradient Compression Schemes
• Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group
• Faster First-Order Methods for Stochastic Non-Convex Optimization on Riemannian Manifolds
• Sparsified SGD with Memory
• Local SGD Converges Fast and Communicates Little
• Averaging Stochastic Gradient Descent on Riemannian Manifolds
• signSGD: Compressed Optimisation for Non-Convex Problems
• Gradient Sparsification for Communication-Efficient Distributed Optimization
• Achieving Geometric Convergence for Distributed Optimization over Time-Varying Graphs
• First-order Methods for Geodesically Convex Optimization
• Deep Residual Learning for Image Recognition
• Unitary Evolution Recurrent Neural Networks
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• The Geometry of Algorithms with Orthogonality Constraints
• Analysis of a complex of statistical variables into principal components.

Related

• Retraction-Free Optimization over the Stiefel Manifold for the LoRA Fine-Tuning
• Distributed Riemannian Stochastic Gradient Tracking Algorithm on the Stiefel Manifold
• Optimization without Retraction on the Random Generalized Stiefel Manifold
• Stochastic optimization over expectation-formulated generalized Stiefel manifold
• Decentralized Riemannian Gradient Descent on the Stiefel Manifold