2022/01/04 by Qunxi Zhu, Yifei Shen, Zhu, Qunxi +5
Computer Science · Physics and Astronomy · #34Kxx #92B2 #93Cxx #Artificial Intelligence (cs.AI) #Chaotic Dynamics (nlin.CD) #Dynamical Systems (math.DS) #FOS: Computer and information sciences #FOS: Mathematics #FOS: Physical sciences #I.2.6 #Machine Learning (cs.LG) #Model Reduction and Neural Networks #Neural Networks and Applications #Neural Networks and Reservoir Computing
paper · pdf · doi:10.48550/arxiv.2201.00960
openalex publication_date 2022/01/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Continuous-depth neural networks, such as the Neural Ordinary Differential Equations (ODEs), have aroused a great deal of interest from the communities of machine learning and data science in recent years, which bridge the connection between deep neural networks and dynamical systems. In this article, we introduce a new sort of continuous-depth neural network, called the Neural Piecewise-Constant Delay Differential Equations (PCDDEs). Here, unlike the recently proposed framework of the Neural Delay Differential Equations (DDEs), we transform the single delay into the piecewise-constant delay(s). The Neural PCDDEs with such a transformation, on one hand, inherit the strength of universal approximating capability in Neural DDEs. On the other hand, the Neural PCDDEs, leveraging the contributions of the information from the multiple previous time steps, further promote the modeling capability without augmenting the network dimension. With such a promotion, we show that the Neural PCDDEs do outperform the several existing continuous-depth neural frameworks on the one-dimensional piecewise-constant delay population dynamics and real-world datasets, including MNIST, CIFAR10, and SVHN.