High-Dimensional Non-Convex Landscapes and Gradient Descent Dynamics

2023/08/07 by Bonnaire, Tony, Ghio, Davide, Krishnamurthy, Kamesh +3 · 2 citations
#Disordered Systems and Neural Networks (cond-mat.dis-nn) #FOS: Physical sciences #Statistical Mechanics (cond-mat.stat-mech)

paper · doi:10.48550/arxiv.2308.03754

Abstract

In these lecture notes we present different methods and concepts developed in statistical physics to analyze gradient descent dynamics in high-dimensional non-convex landscapes. Our aim is to show how approaches developed in physics, mainly statistical physics of disordered systems, can be used to tackle open questions on high-dimensional dynamics in Machine Learning.

Cited by

• Learnability Window in Gated Recurrent Neural Networks
• Dynamics of an outlier in the Gaussian Unitary Ensemble

Related

• High-dimensional random landscapes: from typical to large deviations
• Explorations on high dimensional landscapes
• Escaping Saddles with Stochastic Gradients
• Analytical Study of Momentum-Based Acceleration Methods in Paradigmatic High-Dimensional Non-Convex Problems
• Stochastic Gradient Descent outperforms Gradient Descent in recovering a high-dimensional signal in a glassy energy landscape