An axiomatized PDE model of deep neural networks

2023/07/23 by Tangjun Wang, Wenqi Tao, Wang, Tangjun +5
Computer Science · Physics and Astronomy · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Model Reduction and Neural Networks #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2307.12333

openalex publication_date 2023/07/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Inspired by the relation between deep neural network (DNN) and partial differential equations (PDEs), we study the general form of the PDE models of deep neural networks. To achieve this goal, we formulate DNN as an evolution operator from a simple base model. Based on several reasonable assumptions, we prove that the evolution operator is actually determined by convection-diffusion equation. This convection-diffusion equation model gives mathematical explanation for several effective networks. Moreover, we show that the convection-diffusion model improves the robustness and reduces the Rademacher complexity. Based on the convection-diffusion equation, we design a new training method for ResNets. Experiments validate the performance of the proposed method.

Related

• Convection-Diffusion Equation: A Theoretically Certified Framework for Neural Networks
• Partial Differential Equations Meet Deep Neural Networks: A Survey
• PDE Models for Deep Neural Networks: Learning Theory, Calculus of Variations and Optimal Control
• Connections between Numerical Algorithms for PDEs and Neural Networks
• Deep Neural Networks Inspired by Differential Equations