On learning Gaussian multi‐index models with gradient flow part I: General properties and two‐timescale learning

2025/07/15 by Alberto Bietti, Joan Bruna, Loucas Pillaud‐Vivien · 8 citations
Computer Science · Mathematics · #Neural Networks and Applications #Gaussian Processes and Bayesian Inference #Statistical Methods and Inference

paper · doi:10.1002/cpa.70006

Abstract

Abstract We study gradient flow on the multi‐index regression problem for high‐dimensional Gaussian data. Multi‐index functions consist of a composition of an unknown low‐rank linear projection and an arbitrary unknown, low‐dimensional link function. As such, they constitute a natural template for feature learning in neural networks. We consider a two‐timescale algorithm, whereby the low‐dimensional link function is learnt with a non‐parametric model infinitely faster than the subspace parametrizing the low‐rank projection. By appropriately exploiting the matrix semigroup structure arising over the subspace correlation matrices, we establish global convergence of the resulting Grassmannian gradient flow dynamics, and provide a quantitative description of its associated “saddle‐to‐saddle” dynamics. Notably, the timescales associated with each saddle can be explicitly characterized in terms of an appropriate Hermite decomposition of the target link function.

Citations

• Envelope Theorems for Arbitrary Choice Sets

Cited by

• Sharp convergence rates for Spectral methods via the feature space decomposition method
• Universality of high-dimensional scaling limits of stochastic gradient descent
• Fast Escape, Slow Convergence: Learning Dynamics of Phase Retrieval under Power-Law Data
• Spectral Gradient Descent Mitigates Anisotropy-Driven Misalignment: A Case Study in Phase Retrieval
• Learning Multi-Index Models with Hyper-Kernel Ridge Regression
• Kernel ridge regression under power-law data: spectrum and generalization
• Closed-Form Last Layer Optimization
• Inductive Bias and Spectral Properties of Single-Head Attention in High Dimensions

Related

• On Learning Gaussian Multi-index Models with Gradient Flow
• Learning Gaussian Multi-Index Models with Gradient Flow: Time Complexity and Directional Convergence
• Neural Networks Learn Generic Multi-Index Models Near Information-Theoretic Limit
• Gradient flow in the gaussian covariate model: exact solution of learning curves and multiple descent structures
• Repetita Iuvant: Data Repetition Allows SGD to Learn High-Dimensional Multi-Index Functions