Towards Critical Branching Mechanism in Recurrent Neural Networks

2026/06/09 by Feixiang Ren, Ling Feng · 1 voice
Physics and Astronomy · Computer Science · #nlin.AO #cs.AI #physics.comp-ph

paper · pdf

Abstract

Criticality has been proposed as a key organizing principle in biological neural systems, yet its origin and relevance in artificial neural networks remain unclear. We analyze hidden-state dynamics in trained long short-term memory (LSTM) networks and show that small networks near their optimal training epochs (steps) exhibit scale-free avalanche statistics and branching parameters close to unity, indicative of near-critical dynamics, while larger models remain subcritical. To explain the coexistence of subcritical branching with robust 1/f^β noise, we introduce a mixture branching process framework that links heterogeneous branching dynamics to long-range temporal correlations. These results identify critical-like behavior in LSTMs as an emergent, capacity-dependent dynamical regime.

Citations

• Toward a Physics of Deep Learning and Brains
• Multiple Descents in Deep Learning as a Sequence of Order-Chaos Transitions
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• Linear Transformers Are Secretly Fast Weight Programmers
• Self-Organized Criticality in the Brain
• Optimal Machine Intelligence at the Edge of Chaos
• Attention Is All You Need
• Inferring collective dynamical states from widely unobserved systems
• Identity Mappings in Deep Residual Networks
• On the difficulty of training Recurrent Neural Networks
• Are biological systems poised at criticality?
• Power spectra of self-organized critical sandpiles
• Long Short-Term Memory
• Self-Organized Branching Processes: A Mean-Field Theory for Avalanches
• Mosaic organization of DNA nucleotides
• Learning representations by back-propagating errors

Discussions

• Feixiang Ren, Ling Feng: Towards Critical Branching Mechanism in Recurrent Neural Networks https://arxiv.org/abs/2606.10384 https://arxiv.org/pdf/2606.10384 https://arxiv.org/html/2606.10384 [bsky, 0 points, 0 comments]

Related

• Critical neural networks with short and long term plasticity
• Robust transformations of firing patterns for neural networks
• Self-organized criticality in neural network models
• Adaptive self-organization in a realistic neural network model
• A General Description of Criticality in Neural Network Models