Side Effects of Learning from Low-dimensional Data Embedded in a Euclidean Space

2022/03/01 by Juncai He, He, Juncai, Richard Tzong‐Han Tsai +3
Computer Science · #FOS: Computer and information sciences #FOS: Mathematics #Gaussian Processes and Bayesian Inference #Image and Signal Denoising Methods #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Applications #Numerical Analysis (math.NA)

paper · pdf · doi:10.48550/arxiv.2203.00614

openalex publication_date 2022/03/01 · openalex created_date 2022/04/03 · openalex updated_date 2026/07/28

Abstract

The low-dimensional manifold hypothesis posits that the data found in many applications, such as those involving natural images, lie (approximately) on low-dimensional manifolds embedded in a high-dimensional Euclidean space. In this setting, a typical neural network defines a function that takes a finite number of vectors in the embedding space as input. However, one often needs to consider evaluating the optimized network at points outside the training distribution. This paper considers the case in which the training data is distributed in a linear subspace of \mathbb R^d. We derive estimates on the variation of the learning function, defined by a neural network, in the direction transversal to the subspace. We study the potential regularization effects associated with the network's depth and noise in the codimension of the data manifold. We also present additional side effects in training due to the presence of noise.

Related

• Effects of Data Geometry in Early Deep Learning
• Hardness of Learning Neural Networks under the Manifold Hypothesis
• Construction of neural networks for realization of localized deep learning
• An Analytical Characterization of Sloppiness in Neural Networks: Insights from Linear Models
• Dimensionality compression and expansion in Deep Neural Networks