2016/04/27 by Ragav Venkatesan, Venkatesan, Ragav, Baoxin Li +1 · 14 citations
Computer Science · #Advanced Neural Network Applications #Artificial intelligence #Artificial neural network #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Convolutional neural network #Copying #Data science #Deep neural networks #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Data Classification #Machine learning #Neural and Evolutionary Computing (cs.NE) #Possession (linguistics) #Regularization (linguistics) #Scratch #Tweaking #cs.CV #cs.LG #cs.NE
paper · pdf · doi:10.48550/arxiv.1604.08220
published in arXiv (Cornell University) (Cornell University)
arxiv created 2016/04/27 · openalex publication_date 2016/04/27 · arxiv updated 2016/04/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/08
Modern computer vision is all about the possession of powerful image representations. Deeper and deeper convolutional neural networks have been built using larger and larger datasets and are made publicly available. A large swath of computer vision scientists use these pre-trained networks with varying degrees of successes in various tasks. Even though there is tremendous success in copying these networks, the representational space is not learnt from the target dataset in a traditional manner. One of the reasons for opting to use a pre-trained network over a network learnt from scratch is that small datasets provide less supervision and require meticulous regularization, smaller and careful tweaking of learning rates to even achieve stable learning without weight explosion. It is often the case that large deep networks are not portable, which necessitates the ability to learn mid-sized networks from scratch. In this article, we dive deeper into training these mid-sized networks on small datasets from scratch by drawing additional supervision from a large pre-trained network. Such learning also provides better generalization accuracies than networks trained with common regularization techniques such as l2, l1 and dropouts. We show that features learnt thus, are more general than those learnt independently. We studied various characteristics of such networks and found some interesting behaviors.