Description of the Training Process of Neural Networks via Ergodic Theorem : Ghost nodes

2025/07/01 by Eunji Park, Park, Eun-Ji, Sang-Won Yun +1
Computer Science · Physics and Astronomy · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and ELM #Model Reduction and Neural Networks #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2507.01003

openalex publication_date 2025/07/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent studies have proposed interpreting the training process from an ergodic perspective. Building on this foundation, we present a unified framework for understanding and accelerating the training of deep neural networks via stochastic gradient descent (SGD). By analyzing the geometric landscape of the objective function we introduce a practical diagnostic, the running estimate of the largest Lyapunov exponent, which provably distinguishes genuine convergence toward stable minimizers from mere statistical stabilization near saddle points. We then propose a ghost category extension for standard classifiers that adds auxiliary ghost output nodes so the model gains extra descent directions that open a lateral corridor around narrow loss barriers and enable the optimizer to bypass poor basins during the early training phase. We show that this extension strictly reduces the approximation error and that after sufficient convergence the ghost dimensions collapse so that the extended model coincides with the original one and there exists a path in the enlarged parameter space along which the total loss does not increase. Taken together, these results provide a principled architecture level intervention that accelerates early stage trainability while preserving asymptotic behavior and simultaneously serves as an architecture-friendly regularizer.

Citations

• Deep neural networks from the perspective of ergodic theory
• Stochastic Methods in Variational Inequalities: Ergodicity, Bias and Refinements
• Deep Double Descent: Where Bigger Models and More Data Hurt
• Reconciling modern machine-learning practice and the classical bias–variance trade-off
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• The Loss Surfaces of Multilayer Networks
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization

Related

• Edge of chaos as a guiding principle for modern neural network training
• The Break-Even Point on Optimization Trajectories of Deep Neural Networks
• Towards Guided Descent: Optimization Algorithms for Training Neural Networks At Scale
• On the generalization of learning algorithms that do not converge
• Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworks