Deep Learning via Dynamical Systems: An Approximation Perspective

2019/12/22 by Qianxiao Li, Li, Qianxiao, Ting Lin +3 · 7 citations
Computer Science · Mathematics · Physics and Astronomy · #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Markov Chains and Monte Carlo Methods #Model Reduction and Neural Networks #Optimization and Control (math.OC)

paper · pdf · doi:10.48550/arxiv.1912.10382

openalex publication_date 2019/12/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We build on the dynamical systems approach to deep learning, where deep residual networks are idealized as continuous-time dynamical systems, from the approximation perspective. In particular, we establish general sufficient conditions for universal approximation using continuous-time deep residual networks, which can also be understood as approximation theories in L^p using flow maps of dynamical systems. In specific cases, rates of approximation in terms of the time horizon are also established. Overall, these results reveal that composition function approximation through flow maps present a new paradigm in approximation theory and contributes to building a useful mathematical framework to investigate deep learning.

Cited by

• Auslander-Yorke Dichotomy and Its Generalizations for Non-Autonomous Dynamical Systems
• Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding
• ODE-ViT: Plug & Play Attention Layer from the Generalization of the ViT as an Ordinary Differential Equation
• Deep Network With Approximation Error Being Reciprocal of Width to Power of Square Root of Depth
• Allocation of Parameters in Transformers
• Achieving Universal Approximation and Universal Interpolation via Nonlinearity of Control Families
• Weight-Parameterization in Continuous Time Deep Neural Networks for Surrogate Modeling

Related

• Deep Neural Network Approximation Theory
• Deep Neural Network Approximation Theory
• Deep Neural Network Approximation of Invariant Functions through Dynamical Systems
• Interpolation and approximation via Momentum ResNets and Neural ODEs
• Interpolation, Approximation and Controllability of Deep Neural Networks