Stable ResNet

2020/10/24 by Soufiane Hayou, Hayou, Soufiane, Eugenio Clerico +9 · 5 citations
Computer Science · #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Human Pose and Action Recognition #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · pdf · doi:10.48550/arxiv.2010.12859

openalex publication_date 2020/10/24 · openalex created_date 2022/10/08 · openalex updated_date 2026/07/28

Abstract

Deep ResNet architectures have achieved state of the art performance on many tasks. While they solve the problem of gradient vanishing, they might suffer from gradient exploding as the depth becomes large (Yang et al. 2017). Moreover, recent results have shown that ResNet might lose expressivity as the depth goes to infinity (Yang et al. 2017, Hayou et al. 2019). To resolve these issues, we introduce a new class of ResNet architectures, called Stable ResNet, that have the property of stabilizing the gradient while ensuring expressivity in the infinite depth limit.

Cited by

• A Proof of Learning Rate Transfer under μP
• Probability Distribution Collapse: A Critical Bottleneck to Compact Unsupervised Neural Grammar Induction
• The Hidden Width of Deep ResNets: Tight Error Bounds and Phase Diagram
• The Importance of Being Lazy: Scaling Limits of Continual Learning
• Optimal Embedding Learning Rate in LLMs: The Effect of Vocabulary Size

Related

• Resnet in Resnet: Generalizing Residual Architectures
• Scaling ResNets in the Large-depth Regime
• Wide Residual Networks
• Learning Deep ResNet Blocks Sequentially using Boosting Theory
• ResNet: Enabling Deep Convolutional Neural Networks through Residual Learning