Deep Algorithms: designs for networks

2018/06/06 by Abhejit Rajagopal, Rajagopal, Abhejit, Shivkumar Chandrasekaran +3
Computer Science · Mathematics · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural and Evolutionary Computing (cs.NE) #cs.CV #cs.LG #cs.NE #stat.ML

paper · pdf · doi:10.48550/arxiv.1806.02003

submitted to Thirty-second Annual Conference on Neural Information Processing Systems (NIPS), May 2018

arxiv created 2018/06/06 · arxiv updated 2018/06/07

Abstract

A new design methodology for neural networks that is guided by traditional algorithm design is presented. To prove our point, we present two heuristics and demonstrate an algorithmic technique for incorporating additional weights in their signal-flow graphs. We show that with training the performance of these networks can not only exceed the performance of the initial network, but can match the performance of more-traditional neural network architectures. A key feature of our approach is that these networks are initialized with parameters that provide a known performance threshold for the architecture on a given task.

Related

• Optimization Algorithm Inspired Deep Neural Network Structure Design
• Automated Architecture Design for Deep Neural Networks
• Deep Networks from the Principle of Rate Reduction
• Training Neural Networks at Any Scale
• Activation function design for deep networks: linearity and effective initialisation