2020/06/25 by Dániel Lévai, Lévai, Dániel, Zsolt Zombori +1
Computer Science · #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning #Neural Networks and Applications
paper · pdf · doi:10.48550/arxiv.2006.14350
The Lottery Ticket Hypothesis postulates that a freshly initialized neural network contains a small subnetwork that can be trained in isolation to achieve similar performance as the full network. Our paper examines several alternatives to search for such subnetworks. We conclude that incorporating a data dependent component into the pruning criterion in the form of the gradient of the training loss -- as done in the SNIP method -- consistently improves the performance of existing pruning algorithms.