On Dissipativity of Cross-Entropy Loss in Training ResNets

2024/05/29 by Jens Püttschneider, Timm Faulwasser, Püttschneider, Jens +1
Computer Science · #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2405.19013

Abstract

The training of ResNets and neural ODEs can be formulated and analyzed from the perspective of optimal control. This paper proposes a dissipative formulation of the training of ResNets and neural ODEs for classification problems by including a variant of the cross-entropy as a regularization in the stage cost. Based on the dissipative formulation of the training, we prove that the trained ResNet exhibit the turnpike phenomenon. We then illustrate that the training exhibits the turnpike phenomenon by training on the two spirals and MNIST datasets. This can be used to find very shallow networks suitable for a given classification task.

Related

• A Dual Process Model for Optimizing Cross Entropy in Neural Networks
• Towards an Optimal Control Perspective of ResNet Training
• Adaptively Solving the Local-Minimum Problem for Deep Neural Networks
• Towards Understanding Learning in Neural Networks with Linear Teachers
• On the Separability of Classes with the Cross-Entropy Loss Function