Categorical Invariants of Learning Dynamics

2025/10/05 by Tamim, Abdulrahman
Computer Science · #18B99 #55N35 #68T07 #Advanced Graph Neural Networks #F.4.1 #FOS: Computer and information sciences #G.2.2 #I.2.6 #Machine Learning (cs.LG) #Stochastic Gradient Optimization Techniques #Topological and Geometric Data Analysis

paper · pdf · doi:10.48550/arxiv.2510.04376

openalex publication_date 2025/10/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Neural network training is typically viewed as gradient descent on a loss surface. We propose a fundamentally different perspective: learning is a structure-preserving transformation (a functor L) between the space of network parameters (Param) and the space of learned representations (Rep). This categorical framework reveals that different training runs producing similar test performance often belong to the same homotopy class (continuous deformation family) of optimization paths. We show experimentally that networks converging via homotopic trajectories generalize within 0.5% accuracy of each other, while non-homotopic paths differ by over 3%. The theory provides practical tools: persistent homology identifies stable minima predictive of generalization (R² = 0.82 correlation), pullback constructions formalize transfer learning, and 2-categorical structures explain when different optimization algorithms yield functionally equivalent models. These categorical invariants offer both theoretical insight into why deep learning works and concrete algorithmic principles for training more robust networks.

Citations

• Emergent properties of the local geometry of neural loss landscapes
• Transfusion: Understanding Transfer Learning for Medical Imaging
• Uniform convergence may be unable to explain generalization in deep learning
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
• Exploring Generalization in Deep Learning
• Geometric deep learning: going beyond Euclidean data
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Flat Minima

Related

• Network Dynamics-Based Framework for Understanding Deep Neural Networks
• Intrinsic Dimension, Persistent Homology and Generalization in Neural Networks
• Unnatural Algorithms in Machine Learning
• Persistent Homology Captures the Generalization of Neural Networks Without A Validation Set
• An analytic theory of generalization dynamics and transfer learning in deep linear networks