Do Deep Nets Really Need to be Deep?

2013/12/21 by Lei Jimmy Ba, Jimmy Ba, Rich Caruana +2 · 4 voices · 81 citations
Computer Science · #Generative Adversarial Networks and Image Synthesis #Music and Audio Processing #Speech Recognition and Synthesis #cs.LG #cs.NE

paper · pdf · doi:10.48550/arxiv.1312.6184

openalex publication_date 2013/12/21 · openalex created_date 2016/06/24 · openalex updated_date 2026/07/28

Abstract

Currently, deep neural networks are the state of the art on problems such as speech recognition and computer vision. In this extended abstract, we show that shallow feed-forward networks can learn the complex functions previously learned by deep nets and achieve accuracies previously only achievable with deep models. Moreover, in some cases the shallow neural nets can learn these deep functions using a total number of parameters similar to the original deep model. We evaluate our method on the TIMIT phoneme recognition task and are able to train shallow fully-connected nets that perform similarly to complex, well-engineered, deep convolutional architectures. Our success in training shallow neural nets to mimic deeper models suggests that there probably exist better algorithms for training shallow feed-forward nets than those currently available.

Citations

• Learning Multiple Layers of Features from Tiny Images

Cited by

• ADS-C: Antidistillation Sampling for Classification
• Is Smaller Always Faster? Tradeoffs in Compressing Self-Supervised Speech Transformers
• TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models
• Rethinking Knowledge Distillation in Collaborative Machine Learning: Memory, Knowledge, and Their Interactions
• Two Heads are Better than One: Distilling Large Language Model Features Into Small Models with Feature Decomposition and Mixture
• A Metamorphic Testing Perspective on Knowledge Distillation for Language Models of Code: Does the Student Deeply Mimic the Teacher?
• Knowledge Distillation of Noisy Force Labels for Improved Coarse-Grained Force Fields
• Tensorizing Neural Networks
• Long Short-Term Sample Distillation
• Learning From Less Data: Diversified Subset Selection and Active\n Learning in Image Classification Tasks
• Integrative neurocybernetic modeling in the era of large-scale neuroscience
• DyNet: Dynamic Convolution for Accelerating Convolutional Neural Networks
• PRGFlow: Benchmarking SWAP-Aware Unified Deep Visual Inertial Odometry
• Manipulating Identical Filter Redundancy for Efficient Pruning on Deep and Complicated CNN
• Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillation
• HYDRA: HYbrid knowledge Distillation and spectral Reconstruction Algorithm for high channel hyperspectral camera applications
• Stealing Neural Networks via Timing Side Channels
• Techniques for Symbol Grounding with SATNet
• Feature Representation Analysis of Deep Convolutional Neural Network using Two-stage Feature Transfer -An Application for Diffuse Lung Disease Classification-
• Teach an all-rounder with experts in different domains
• Collaborative Group Learning
• Ensemble Distillation for Neural Machine Translation
• Exact and Consistent Interpretation of Piecewise Linear Models Hidden behind APIs: A Closed Form Solution
• Compressing Convolutional Neural Networks
• Modeling Lost Information in Lossy Image Compression
• How to Manipulate CNNs to Make Them Lie: the GradCAM Case
• Cross-View Policy Learning for Street Navigation
• A Simple but Effective BERT Model for Dialog State Tracking on Resource-Limited Systems
• Self-Training With Noisy Student Improves ImageNet Classification
• Structured Knowledge Distillation for Dense Prediction
• Anomaly Detection in Univariate Time-series: A Survey on the State-of-the-Art
• Distillation of Large Language Models via Concrete Score Matching
• Towards Understanding Subliminal Learning: When and How Hidden Biases Transfer
• Rethinking Large Language Model Distillation: A Constrained Markov Decision Process Perspective
• On-the-fly Network Pruning for Object Detection
• Student Network Learning via Evolutionary Knowledge Distillation
• Zero-shot Knowledge Transfer via Adversarial Belief Matching
• Lets keep it simple, Using simple architectures to outperform deeper and\n more complex architectures
• Exact and Consistent Interpretation for Piecewise Linear Neural Networks: A Closed Form Solution
• Knowledge Squeezed Adversarial Network Compression
• Pretrained Transformers for Text Ranking: BERT and Beyond
• Privileged Knowledge Distillation for Online Action Detection
• ADD: Augmented Disentanglement Distillation Framework for Improving Stock Trend Forecasting
• When Does Preconditioning Help or Hurt Generalization?
• Improving the Learning of Multi-column Convolutional Neural Network for Crowd Counting
• Learning to Steer by Mimicking Features from Heterogeneous Auxiliary Networks
• Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation
• Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender System
• ReachNN: Reachability Analysis of Neural-Network Controlled Systems
• RGP: Neural Network Pruning through Its Regular Graph Structure
• Deep Learning for Medical Image Segmentation
• Knowledge Distillation for Sensing-Assisted Long-Term Beam Tracking in mmWave Communications
• Knowledge Transfer via Dense Cross-Layer Mutual-Distillation
• Sparsifying Neural Network Connections for Face Recognition
• Dataset Distillation
• Sequence-Level Knowledge Distillation
• Distilling Knowledge Using Parallel Data for Far-field Speech Recognition
• Network Implosion: Effective Model Compression for ResNets via Static Layer Pruning and Retraining
• From Consensus to Disagreement: Multi-Teacher Distillation for Semi-Supervised Relation Extraction
• Noisy Self-Knowledge Distillation for Text Summarization
• Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling
• Expandable Residual Approximation for Knowledge Distillation
• Improving the Interpretability of Deep Neural Networks with Knowledge Distillation
• NoScope: Optimizing Neural Network Queries over Video at Scale
• Verifiable Reinforcement Learning via Policy Extraction
• Variational Information Distillation for Knowledge Transfer
• Synthetic Adaptive Guided Embeddings (SAGE): A Novel Knowledge Distillation Method
• Enhancing Explainability of Neural Networks through Architecture Constraints
• Paying more attention to snapshots of Iterative Pruning: Improving Model Compression via Ensemble Distillation
• EntropyGS: An Efficient Entropy Coding on 3D Gaussian Splatting
• Why distillation helps: a statistical perspective
• Task complexity shapes internal representations and robustness in neural networks
• FEED: Feature-level Ensemble for Knowledge Distillation
• Knowledge Distillation for End-to-End Person Search
• A copula-based visualization technique for a neural network
• Giraffe: Using Deep Reinforcement Learning to Play Chess
• ShrinkTeaNet: Million-scale Lightweight Face Recognition via Shrinking Teacher-Student Networks
• ASNN: Learning to Suggest Neural Architectures from Performance Distributions
• A Case for Backward Compatibility for Human-AI Teams
• Detecting Forged Facial Videos using convolutional neural network
• FATE: Fast and Accurate Timing Error Prediction Framework for Low Power DNN Accelerator Design

Discussions

• Do Deep Nets Really Need to be Deep? [pdf] [hn, 126 points, 29 comments]
• Do Deep Nets Really Need to be Deep? [lobsters, 2 points, 0 comments]
• Shallow neural networks as good as Deep Neural Networks [hn, 2 points, 1 comments]
• Do Deep Nets Really Need to be Deep? [hn, 1 points, 0 comments]

Related

• Realizing data features by deep nets
• Do Deep Convolutional Nets Really Need to be Deep and Convolutional?
• How deep should be the depth of convolutional neural networks: a backyard dog case study
• Learning Functions: When Is Deep Better Than Shallow
• A Selective Overview of Deep Learning