Constraining the outputs of ReLU neural networks

2025/08/05 by Alexandr, Yulia, Montúfar, Guido
#Algebraic Geometry (math.AG) #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.2508.03867

Abstract

We introduce a class of algebraic varieties naturally associated with ReLU neural networks, arising from the piecewise linear structure of their outputs across activation regions in input space, and the piecewise multilinear structure in parameter space. By analyzing the rank constraints on the network outputs within each activation region, we derive polynomial equations that characterize the functions representable by the network. We further investigate conditions under which these varieties attain their expected dimension, providing insight into the expressive and structural properties of ReLU networks.

Citations

• Approximating Latent Manifolds in Neural Networks via Vanishing Ideals
• On the Local Complexity of Linear Regions in Deep ReLU Networks
• Implicit Bias of Mirror Flow for Shallow Neural Networks in Univariate Regression
• Bounds for the smallest eigenvalue of the NTK for arbitrary spherical data of arbitrary dimension
• The Real Tropical Geometry of Neural Networks
• Deep Networks Always Grok and Here is Why
• Mildly Overparameterized ReLU Networks Have a Favorable Loss Landscape
• Function Space and Critical Points of Linear Convolutional Networks
• Lower Bounds on the Depth of Integral ReLU Neural Networks via Lattice Polytopes
• A Combinatorial Perspective on the Optimization of Shallow ReLU Networks
• Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterization
• Parameter identifiability of a deep feedforward ReLU neural network
• Geometry of Linear Convolutional Networks
• On the Expected Complexity of Maxout Networks
• Sharp bounds for the number of regions of maxout networks and vertices of Minkowski sums
• Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networks
• The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Training
• Affine symmetries and neural network identifiability
• Implicit Bias of Gradient Descent for Mean Squared Error Regression with Two-Layer Wide Neural Networks
• Reverse-Engineering Deep ReLU Networks
• Best k-layer neural network approximations
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Restricted Boltzmann Machines: Introduction and Review
• A Tropical Approach to Neural Networks with Piecewise Linear Activations
• The Multilinear Structure of ReLU Networks
• Bounding and Counting Linear Regions of Deep Neural Networks
• Benefits of depth in neural networks
• On the Number of Linear Regions of Deep Neural Networks
• On the number of response regions of deep feed forward networks with\n piece-wise linear activations

Related

• Algorithmic Determination of the Combinatorial Structure of the Linear Regions of ReLU Neural Networks
• Toric geometry of ReLU neural networks
• The Geometric Structure of Fully-Connected ReLU Layers
• Locally Linear Attributes of ReLU Neural Networks
• Theoretical Exploration of Solutions of Feedforward ReLU Networks