Algebraic Approach to Ridge-Regularized Mean Squared Error Minimization in Minimal ReLU Neural Network

2025/08/25 by Fukasaku, Ryoya, Kabata, Yutaro, Okuno, Akifumi
#Artificial Intelligence (cs.AI) #Computation (stat.CO) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.2508.17783

Abstract

This paper investigates a perceptron, a simple neural network model, with ReLU activation and a ridge-regularized mean squared error (RR-MSE). Our approach leverages the fact that the RR-MSE for ReLU perceptron is piecewise polynomial, enabling a systematic analysis using tools from computational algebra. In particular, we develop a Divide-Enumerate-Merge strategy that exhaustively enumerates all local minima of the RR-MSE. By virtue of the algebraic formulation, our approach can identify not only the typical zero-dimensional minima (i.e., isolated points) obtained by numerical optimization, but also higher-dimensional minima (i.e., connected sets such as curves, surfaces, or hypersurfaces). Although computational algebraic methods are computationally very intensive for perceptrons of practical size, as a proof of concept, we apply the proposed approach in practice to minimal perceptrons with a few hidden units.

Citations

• Algebraic Approach for Orthomax Rotations
• Accurate structure prediction of biomolecular interactions with AlphaFold 3
• The Real Tropical Geometry of Neural Networks
• Algebraic approach to maximum likelihood factor analysis
• Convex Relaxations of ReLU Neural Networks Approximate Global Optima in Polynomial Time
• Learning to Compute Gröbner Bases
• A Survey on Evaluation of Large Language Models
• A Survey on Evaluation of Large Language Models
• A Generalist Agent
• Fast Convex Optimization for Two-Layer ReLU Networks: Equivalent Model Classes and Cone Decompositions
• Highly accurate protein structure prediction with AlphaFold
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Language Models are Few-Shot Learners
• Neural Networks are Convex Regularizers: Exact Polynomial-time Convex Optimization Formulations for Two-layer Networks
• On the Decision Boundaries of Neural Networks: A Tropical Geometry Perspective
• An Investigation into Neural Net Optimization via Hessian Eigenvalue\n Density
• Deep Neural Network Approximation Theory
• Deep Neural Network Approximation Theory
• The loss surface of deep linear networks viewed through the algebraic geometry lens
• Complexity of Training ReLU Neural Network
• Decoupling Gating from Linearity
• A Tropical Approach to Neural Networks with Piecewise Linear Activations
• Neural Algebra of Classifiers
• Visualizing the Loss Landscape of Neural Nets
• A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
• Spectrally-normalized margin bounds for neural networks
• Attention Is All You Need
• Sharp Minima Can Generalize For Deep Nets
• Multilayer Perceptron Algebra
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp\n Minima
• Semi-Supervised Classification with Graph Convolutional Networks
• Deep Learning without Poor Local Minima
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• On the Number of Linear Regions of Deep Neural Networks
• A Bayesian information criterion for singular models
• ImageNet classification with deep convolutional neural networks
• Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory
• Backpropagation Applied to Handwritten Zip Code Recognition
• Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position
• The Differentiation of Pseudo-Inverses and Nonlinear Least Squares Problems Whose Variables Separate
• The perceptron: A probabilistic model for information storage and organization in the brain.

Related

• Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry II
• Annihilation of Spurious Minima in Two-Layer ReLU Networks
• Deep ReLU Programming
• On the existence of minimizers in shallow residual ReLU neural network optimization landscapes
• Spurious Local Minima are Common in Two-Layer ReLU Neural Networks