Big Neural Networks Waste Capacity

2013/01/16 by Yann Dauphin, Yoshua Bengio, Dauphin, Yann N. +1 · 4 citations
Computer Science · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.1301.3583

openalex publication_date 2013/01/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This article exposes the failure of some big neural networks to leverage added capacity to reduce underfitting. Past research suggest diminishing returns when increasing the size of neural networks. Our experiments on ImageNet LSVRC-2010 show that this may be due to the fact there are highly diminishing returns for capacity in terms of training error, leading to underfitting. This suggests that the optimization method - first order gradient descent - fails at this regime. Directly attacking this problem, either through the optimization method or the choices of parametrization, may allow to improve the generalization error on large datasets, for which a large capacity is required.

Cited by

• XNOR-Net: ImageNet Classification Using Binary Convolutional Neural\n Networks
• Wormhole Dynamics in Deep Neural Networks
• HALO: Learning to Prune Neural Networks with Shrinkage
• Multilayer Perceptron Algebra

Related

• Self-Expanding Neural Networks
• Scaling up deep neural networks: a capacity allocation perspective
• Greedy Network Enlarging
• Just How Flexible are Neural Networks in Practice?
• Low-Memory Neural Network Training: A Technical Report