Efficient Representation of Low-Dimensional Manifolds using Deep Networks

2016/02/15 by Ronen Basri, David Jacobs, Basri, Ronen +2 · 5 citations
Computer Science · Engineering · Mathematics · #3D Shape Modeling and Analysis #FOS: Computer and information sciences #Face and Expression Recognition #Human Pose and Action Recognition #Image Processing and 3D Reconstruction #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Morphological variations and asymmetry #Neural and Evolutionary Computing (cs.NE) #cs.LG #cs.NE #stat.ML

paper · pdf · doi:10.48550/arxiv.1602.04723

arxiv created 2016/02/15 · openalex publication_date 2016/02/15 · arxiv updated 2016/02/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We consider the ability of deep neural networks to represent data that lies near a low-dimensional manifold in a high-dimensional space. We show that deep networks can efficiently extract the intrinsic, low-dimensional coordinates of such data. We first show that the first two layers of a deep network can exactly embed points lying on a monotonic chain, a special type of piecewise linear manifold, mapping them to a low-dimensional Euclidean space. Remarkably, the network can do this using an almost optimal number of parameters. We also show that this network projects nearby points onto the manifold and then embeds them with little error. We then extend these results to more general manifolds.

Citations

• Laplacian Eigenmaps for Dimensionality Reduction and Data Representation
• Do Deep Nets Really Need to be Deep?
• The Power of Depth for Feedforward Neural Networks

Cited by

• Geometry of Neural Reinforcement Learning in Continuous State and Action Spaces
• PointShuffleNet: Learning Non-Euclidean Features with Homotopy Equivalence and Mutual Information
• Construction of neural networks for realization of localized deep learning
• Can I trust you more? Model-Agnostic Hierarchical Explanations
• A Neural Scaling Law from the Dimension of the Data Manifold

Related

• Deep nets for local manifold learning
• Convolutional Neural Networks on Manifolds: From Graphs and Back
• DeepAtlas: a tool for effective manifold learning
• Exploring the Manifold of Neural Networks Using Diffusion Geometry
• Representation Learning via Manifold Flattening and Reconstruction