2025/02/13 by Beatrice Achilli, Achilli, Beatrice, Luca Ambrogioni +7 · 3 citations
Engineering · #Industrial Technology and Control Systems
paper · pdf · doi:10.48550/arxiv.2502.09578
We study the memorization and generalization capabilities of Diffusion Models (DMs) when data lies on a structured latent manifold. Specifically, we consider a set of P data points in N dimensions confined to a latent subspace of dimension D = αD N, following the Hidden Manifold Model (HMM). We analyze the reverse diffusion process using the empirical score function as a proxy, and characterize it in the high-dimensional limit P = exp(αN), N ≫ 1, by exploiting a connection with the Random Energy Model (REM). We show that a characteristic time to marks the emergence of traps in the time-dependent potential, which however do not affect typical trajectories. The size of their basins of attraction is computed at all times. We derive the collapse time tc < to, at which trajectories fall into the basin of a training point, signaling memorization. An explicit formula for tc as a function of P and αD shows that the curse of dimensionality is avoided for structured data (αD ≪ 1), even with nonlinear manifolds. We also prove that collapse corresponds to the condensation transition in the REM. Generalization is quantified via the Kullback-Leibler divergence between the exact distribution and the reverse one at time t. We find a distinct time tg < tc < to minimizing this divergence. Surprisingly, the best generalization occurs inside the memorization phase. We conclude that generalization in DMs improves with data structure, as tg → 0 faster than tc when αD → 0.