2023/11/26 by Noboru Isobe, Isobe, Noboru · 1 citation
#cs.LG #math.AP #math.FA #math.PR
paper · pdf · doi:10.48550/arxiv.2311.15365
We study an idealized training process for deep neural networks in a continuous-depth, mean-field model in which each layer is parameterized by a probability measure on a Euclidean parameter space. The training dynamics are formulated as a Wasserstein-type gradient flow of an objective with a fixed L2-regularization. Under suitable analyticity and growth assumptions, together with a coercivity assumption and sufficient regularity of the initial data, we prove that every curve of maximal slope converges to a single critical point of the objective as the training time tends to infinity. The proof combines compactness of the curve with a Łojasiewicz--Simon inequality for the metric slope. To establish the inequality, we lift the objective to a Hilbert space of random variables and use the analyticity of the lifted gradient in a stronger L^∞ topology to overcome its lack of continuous differentiability in the Hilbert-space topology. Our convergence result does not require global displacement convexity, a Polyak--Łojasiewicz-type condition, or initialization near a minimizer; the objective may remain genuinely nonconvex.