The Local Elasticity of Neural Networks

2019/10/15 by Hangfeng He, Weijie Su, Weijie J. Su +2 · 7 citations
Computer Science · Mathematics · Physics and Astronomy · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Model Reduction and Neural Networks #Neural Networks and Applications #Stochastic Gradient Optimization Techniques #cs.CV #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1910.06943

To appear in ICLR 2020

openalex publication_date 2019/10/15 · arxiv created 2020/02/15 · arxiv updated 2020/02/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper presents a phenomenon in neural networks that we refer to as local elasticity. Roughly speaking, a classifier is said to be locally elastic if its prediction at a feature vector \bx' is not significantly perturbed, after the classifier is updated via stochastic gradient descent at a (labeled) feature vector \bx that is dissimilar to \bx' in a certain sense. This phenomenon is shown to persist for neural networks with nonlinear activation functions through extensive simulations on real-life and synthetic datasets, whereas this is not observed in linear classifiers. In addition, we offer a geometric interpretation of local elasticity using the neural tangent kernel \citepjacot2018neural. Building on top of local elasticity, we obtain pairwise similarity measures between feature vectors, which can be used for clustering in conjunction with K-means. The effectiveness of the clustering algorithm on the MNIST and CIFAR-10 datasets in turn corroborates the hypothesis of local elasticity of neural networks on real-life data. Finally, we discuss some implications of local elasticity to shed light on several intriguing aspects of deep neural networks.

Citations

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• The Loss Surfaces of Multilayer Networks
• On the Number of Linear Regions of Deep Neural Networks
• ImageNet classification with deep convolutional neural networks
• Globally Optimal Gradient Descent for a ConvNet with Gaussian Inputs
• Understanding Black-box Predictions via Influence Functions
• Geometry of Optimization and Implicit Regularization in Deep Learning
• Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Complexity of Linear Regions in Deep Networks
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Towards moderate overparameterization: global convergence guarantees for training shallow neural networks
• Stiffness: A New Perspective on Generalization in Neural Networks
• On Exact Computation with an Infinitely Wide Neural Net
• What Can ResNet Learn Efficiently, Going Beyond Kernels?
• Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel
• Implicit Regularization in Deep Matrix Factorization
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Beyond Reward Margin: Rethinking and Resolving Likelihood Displacement in Diffusion Models via Video Generation
• The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic View
• Efficient Reinforcement Learning by Reducing Forgetting with Elephant Activation Functions
• Generalization vs. Memorization in Autoregressive Deep Learning: Or, Examining Temporal Decay of Gradient Coherence
• LEAD: Exploring Logit Space Evolution for Model Selection
• On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization
• GradPCA: Leveraging NTK Alignment for Reliable Out-of-Distribution Detection

Related

• Dynamics of Local Elasticity During Training of Neural Nets
• Imitating Deep Learning Dynamics via Locally Elastic Stochastic Differential Equations
• Stiffness: A New Perspective on Generalization in Neural Networks
• Elastic Neural Networks for Classification
• Label-Aware Neural Tangent Kernel: Toward Better Generalization and Local Elasticity