2015/11/21 by Diogo Almeida, Almeida, Diogo, Nate Sauder +1
Computer Science · Physics and Astronomy · #Advanced Neural Network Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Model Reduction and Neural Networks #Neural Networks and Applications #Neural and Evolutionary Computing (cs.NE) #cs.LG #cs.NE
paper · pdf · doi:10.48550/arxiv.1511.06827
arxiv created 2015/11/21 · openalex publication_date 2015/11/21 · arxiv updated 2015/11/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
In machine learning, there is a fundamental trade-off between ease of optimization and expressive power. Neural Networks, in particular, have enormous expressive power and yet are notoriously challenging to train. The nature of that optimization challenge changes over the course of learning. Traditionally in deep learning, one makes a static trade-off between the needs of early and late optimization. In this paper, we investigate a novel framework, GradNets, for dynamically adapting architectures during training to get the benefits of both. For example, we can gradually transition from linear to non-linear networks, deterministic to stochastic computation, shallow to deep architectures, or even simple downsampling to fully differentiable attention mechanisms. Benefits include increased accuracy, easier convergence with more complex architectures, solutions to test-time execution of batch normalization, and the ability to train networks of up to 200 layers.