2021/02/15 by Pierre Ablin, Gabriel Peyré, Ablin, Pierre +1 · 10 citations
Computer Science · Engineering · Mathematics · #Advanced Optimization Algorithms Research #Blind Source Separation Techniques #Computation (stat.CO) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Matrix Theory and Algorithms #Sparse and Compressive Sensing Techniques
paper · pdf · doi:10.48550/arxiv.2102.07432
openalex publication_date 2021/02/15 · openalex created_date 2021/03/01 · openalex updated_date 2026/07/28
We consider the problem of minimizing a function over the manifold of\northogonal matrices. The majority of algorithms for this problem compute a\ndirection in the tangent space, and then use a retraction to move in that\ndirection while staying on the manifold. Unfortunately, the numerical\ncomputation of retractions on the orthogonal manifold always involves some\nexpensive linear algebra operation, such as matrix inversion, exponential or\nsquare-root. These operations quickly become expensive as the dimension of the\nmatrices grows. To bypass this limitation, we propose the landing algorithm\nwhich does not use retractions. The algorithm is not constrained to stay on the\nmanifold but its evolution is driven by a potential energy which progressively\nattracts it towards the manifold. One iteration of the landing algorithm only\ninvolves matrix multiplications, which makes it cheap compared to its\nretraction counterparts. We provide an analysis of the convergence of the\nalgorithm, and demonstrate its promises on large-scale and deep learning\nproblems, where it is faster and less prone to numerical errors than\nretraction-based methods.\n