On Geometric Structure of Activation Spaces in Neural Networks

2019/04/02 by Yuting Jia, Jia, Yuting, Haiwen Wang +9 · 1 citation
Computer Science · Physics and Astronomy · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Medical Image Segmentation Techniques #Model Reduction and Neural Networks #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.1904.01399

openalex publication_date 2019/04/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we investigate the geometric structure of activation spaces of fully connected layers in neural networks and then show applications of this study. We propose an efficient approximation algorithm to characterize the convex hull of massive points in high dimensional space. Based on this new algorithm, four common geometric properties shared by the activation spaces are concluded, which gives a rather clear description of the activation spaces. We then propose an alternative classification method grounding on the geometric structure description, which works better than neural networks alone. Surprisingly, this data classification method can be an indicator of overfitting in neural networks. We believe our work reveals several critical intrinsic properties of modern neural networks and further gives a new metric for evaluating them.

Citations

• Deep Learning in Neural Networks: An Overview
• Gradient-based learning applied to document recognition
• Computing the Approximate Convex Hull in High Dimensions
• SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning\n Dynamics and Interpretability
• Towards Understanding Learning Representations: To What Extent Do\n Different Neural Networks Learn the Same Representation
• On Generalization Error Bounds of Noisy Gradient Methods for Non-Convex Learning
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Solving hybrid machine learning tasks by traversing weight space\n geodesics

Related

• Activation Functions Not To Active: A Plausible Theory on Interpreting Neural Networks
• Topological Understanding of Neural Networks, a survey
• Characterizing the Shape of Activation Space in Deep Neural Networks
• Geometric Properties and Graph-Based Optimization of Neural Networks: Addressing Non-Linearity, Dimensionality, and Scalability
• On Space Folds of ReLU Neural Networks