Stable Architectures for Deep Neural Networks

2017/05/31 by Eldad Haber, Lars Ruthotto · 60 citations
Computer Science · Mathematics · Physics and Astronomy · #Model Reduction and Neural Networks #Neural Networks and Reservoir Computing #Stochastic Gradient Optimization Techniques #acm:49N90 #acm:65L09 #acm:68T05 #cs.LG #cs.NA #math.NA #math.OC #msc:49N90 #msc:65L09 #msc:68T05

paper · pdf · doi:10.1088/1361-6420/aa9a90

published as Inverse Problems, Volume 34, Number 1 Inverse Problems, Volume 34, Number 1, 2017 · 23 pages, 7 figures

openalex created_date 2017/05/19 · openalex publication_date 2017/11/14 · arxiv created 2019/02/16 · arxiv updated 2019/02/19 · openalex updated_date 2026/07/28

Abstract

Deep neural networks have become invaluable tools for supervised machine learning, e.g., classification of text or images. While often offering superior results over traditional techniques and successfully expressing complicated patterns in data, deep architectures are known to be challenging to design and train such that they generalize well to new data. Important issues with deep architectures are numerical instabilities in derivative-based learning algorithms commonly called exploding or vanishing gradients. In this paper we propose new forward propagation techniques inspired by systems of Ordinary Differential Equations (ODE) that overcome this challenge and lead to well-posed learning problems for arbitrarily deep networks. The backbone of our approach is our interpretation of deep learning as a parameter estimation problem of nonlinear dynamical systems. Given this formulation, we analyze stability and well-posedness of deep learning and use this new understanding to develop new network architectures. We relate the exploding and vanishing gradient phenomenon to the stability of the discrete ODE and present several strategies for stabilizing deep learning for very deep networks. While our new architectures restrict the solution space, several numerical experiments show their competitiveness with state-of-the-art networks.

Citations

• Sample size selection in optimization methods for machine learning
• Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups

Cited by

• A Learning Stability Profile for Finite-Dimensional Learning Dynamics
• Block-Recurrent Dynamics in Vision Transformers
• Dual Computational Horizons: Incompleteness and Unpredictability in Intelligent Systems
• Optimal and Diffusion Transports in Machine Learning
• Exact and Parametric Dynamical System Representation of Nonlinear Functions
• MFI-ResNet: Efficient ResNet Architecture Optimization via MeanFlow Compression and Selective Incubation
• Mixed Precision Training of Neural ODEs
• Mixed Monotonicity Reachability Analysis of Neural ODE: A Trade-Off Between Tightness and Efficiency
• Convergence, design and training of continuous-time dropout as a random batch method
• Mathematical Modeling and Convergence Analysis of Deep Neural Networks with Dense Layer Connectivities in Deep Learning
• Stability of Transformers under Layer Normalization
• Deep Neural Networks Inspired by Differential Equations
• Achieving Universal Approximation and Universal Interpolation via Nonlinearity of Control Families
• Machine Learning and Control: Foundations, Advances, and Perspectives
• VNODE: A Piecewise Continuous Volterra Neural Network
• A Neural ODE Approach to Aircraft Flight Dynamics Modelling
• Where to Add PDE Diffusion in Transformers
• Neural Feature Geometry Evolves as Discrete Ricci Flow
• Looped Transformers with Source-Centered State Evolution
• Generalization Bounds on Optimal Control for Transformer Training and Wasserstein Distributional Robustness
• Quantum neural ordinary and partial differential equations
• Performance Optimization of YOLO-FEDER FusionNet for Robust Drone Detection in Visually Complex Environments
• A High-order Backpropagation Algorithm for Neural Stochastic Differential Equation Model
• Insights from Gradient Dynamics: Gradient Autoscaled Normalization
• Non-Dissipative Graph Propagation for Non-Local Community Detection
• Robust Convolution Neural ODEs via Contractivity-promoting regularization
• Neural Networks with Orthogonal Jacobian
• Weight-Parameterization in Continuous Time Deep Neural Networks for Surrogate Modeling
• Learning Latent Graph Geometry via Fixed-Point Schrödinger-Type Activation: A Theoretical Study
• Deep learning in ECG diagnosis: A review
• Computing with Canonical Microcircuits
• YuriiFormer: A Suite of Nesterov-Accelerated Transformers
• Neural fractional differential networks for modeling complex dynamical systems
• Recursive Bound-Constrained AdaGrad with Applications to Multilevel and Domain Decomposition Minimization
• Stochastic Operator Network: A Stochastic Maximum Principle Based Approach to Operator Learning
• SILVA Networks as Structured Implicit Layers and Vector Attractors via Dynamic Interaction Fields
• Implicit Machine Learning Force Fields Accelerate Molecular Dynamics Simulations
• Adjoint-based A Posteriori Error Analysis for Semi-explicit Index-1 and Hessenberg Index-2 Differential-Algebraic Equations
• A New Perspective On AI Safety Through Control Theory Methodologies
• A Two-Phase Deep Learning Framework for Adaptive Time-Stepping in High-Speed Flow Modeling
• Return of ChebNet: Understanding and Improving an Overlooked GNN on Long Range Tasks
• DiffusionBlocks: Block-wise Neural Network Training via Diffusion Interpretation
• Understanding LLM-Centric Challenges for Deep Learning Frameworks: An Empirical Analysis
• Bridging Neural ODE and ResNet: A Formal Error Bound for Safety Verification
• Recurrent Self-Attention Dynamics: An Energy-Agnostic Perspective from Jacobians
• Automatic and Structure-Aware Sparsification of Hybrid Neural ODEs
• Approach to Finding a Robust Deep Learning Model
• Do ideas have shape? Idea registration as the continuous limit of artificial neural networks
• EulerLoRA: Rank-Driven Jump Dynamics for Calibrated Parameter-Efficient Fine-Tuning
• Optimal Control for Transformer Architectures: Enhancing Generalization, Robustness and Efficiency
• On dynamical system modeling of learned primal-dual with a linear operator K : stability and convergence properties
• Continuous Temporal Learning of Probability Distributions via Neural ODEs with Applications in Continuous Glucose Monitoring Data
• IM-BERT: Enhancing Robustness of BERT through the Implicit Euler Method
• Deep Sequence Modeling with Quantum Dynamics: Language as a Wave Function
• Training Crossroads for Recurrent Vision Transformers: Recurrence, Neural ODEs, and Deep Supervision
• Deep learning in photoacoustic tomography: current approaches and future directions
• Robust Learning with Implicit Residual Networks
• A mean-field optimal control formulation of deep learning
• A neural operator view on U-Nets for inverse imaging problems
• The Pontryagin Maximum Principle for Training Convolutional Neural Networks

Related

• Reversible Architectures for Arbitrarily Deep Residual Neural Networks
• A PDE-based Explanation of Extreme Numerical Sensitivities and Edge of Stability in Training Neural Networks
• A Learning Stability Profile for Finite-Dimensional Learning Dynamics
• Stabilizing RNN Gradients through Pre-training
• Gradient Flow Equations for Deep Linear Neural Networks: A Survey from a Network Perspective