Understanding Deflation Process in Over-parametrized Tensor Decomposition

2021/06/11 by Rong Ge, Ge, Rong, Yunwei Ren +5 · 1 citation
Engineering · Mathematics · Medicine · #Advanced Neuroimaging Techniques and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Sparse and Compressive Sensing Techniques #Tensor decomposition and applications

paper · pdf · doi:10.48550/arxiv.2106.06573

openalex publication_date 2021/06/11 · openalex created_date 2021/06/22 · openalex updated_date 2026/07/28

Abstract

In this paper we study the training dynamics for gradient flow on over-parametrized tensor decomposition problems. Empirically, such training process often first fits larger components and then discovers smaller components, which is similar to a tensor deflation process that is commonly used in tensor decomposition algorithms. We prove that for orthogonally decomposable tensor, a slightly modified version of gradient flow would follow a tensor deflation process and recover all the tensor components. Our proof suggests that for orthogonal tensors, gradient flow dynamics works similarly as greedy low-rank learning in the matrix setting, which is a first step towards understanding the implicit regularization effect of over-parametrized models for low-rank tensors.

Citations

• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Learning One-hidden-layer Neural Networks with Landscape Design
• Learning Overparameterized Neural Networks via Stochastic Gradient\n Descent on Structured Data
• A Convergence Theory for Deep Learning via Over-Parameterization
• Fine-Grained Analysis of Optimization and Generalization for\n Overparameterized Two-Layer Neural Networks
• On Exact Computation with an Infinitely Wide Neural Net
• Implicit Regularization in Deep Matrix Factorization
• Gradient Descent Maximizes the Margin of Homogeneous Neural Networks
• Taylorized Training: Towards Better Approximation of Neural Network Training at Finite Width
• Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learning
• Implicit Regularization in Tensor Factorization

Cited by

• On the Benefits of Weight Normalization for Overparameterized Matrix Sensing

Related

• Implicit Regularization for Tubal Tensor Factorizations via Gradient Descent
• Beyond Lazy Training for Over-parameterized Tensor Decomposition
• Optimizing Orthogonalized Tensor Deflation via Random Tensor Theory
• Implicit Regularization in Tensor Factorization
• Smoothed Analysis of Tensor Decompositions