Do Deep Nets Really Need to be Deep?

2013/12/21 by Lei Jimmy Ba, Jimmy Ba, Rich Caruana +2 · 4 voices · 1,484 citations
Computer Science · Engineering · #Artificial intelligence #Artificial neural network #Computer science #Convolutional neural network #Deep learning #Deep neural networks #Deep water #Engineering #Generative Adversarial Networks and Image Synthesis #Hidden Markov model #Music and Audio Processing #Speech Recognition and Synthesis #TIMIT #Task (project management) #cs.LG #cs.NE

paper · pdf · doi:10.48550/arxiv.1312.6184

published in arXiv (Cornell University) (Cornell University) · final revision coming soon

openalex publication_date 2013/12/21 · arxiv created 2014/10/11 · arxiv updated 2014/10/14 · openalex created_date 2016/06/24 · openalex updated_date 2026/07/28

Abstract

Currently, deep neural networks are the state of the art on problems such as speech recognition and computer vision. In this extended abstract, we show that shallow feed-forward networks can learn the complex functions previously learned by deep nets and achieve accuracies previously only achievable with deep models. Moreover, in some cases the shallow neural nets can learn these deep functions using a total number of parameters similar to the original deep model. We evaluate our method on the TIMIT phoneme recognition task and are able to train shallow fully-connected nets that perform similarly to complex, well-engineered, deep convolutional architectures. Our success in training shallow neural nets to mimic deeper models suggests that there probably exist better algorithms for training shallow feed-forward nets than those currently available.

Citations

• Learning Multiple Layers of Features from Tiny Images

Cited by

• ADS-C: Antidistillation Sampling for Classification
• Is Smaller Always Faster? Tradeoffs in Compressing Self-Supervised Speech Transformers
• TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models
• Rethinking Knowledge Distillation in Collaborative Machine Learning: Memory, Knowledge, and Their Interactions
• Two Heads are Better than One: Distilling Large Language Model Features Into Small Models with Feature Decomposition and Mixture
• A Metamorphic Testing Perspective on Knowledge Distillation for Language Models of Code: Does the Student Deeply Mimic the Teacher?
• Knowledge Distillation of Noisy Force Labels for Improved Coarse-Grained Force Fields
• Tensorizing Neural Networks
• Long Short-Term Sample Distillation
• Learning From Less Data: Diversified Subset Selection and Active Learning in Image Classification Tasks
• Integrative neurocybernetic modeling in the era of large-scale neuroscience
• DyNet: Dynamic Convolution for Accelerating Convolutional Neural Networks
• PRGFlow: Benchmarking SWAP-Aware Unified Deep Visual Inertial Odometry
• Manipulating Identical Filter Redundancy for Efficient Pruning on Deep and Complicated CNN
• Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillation
• HYDRA: HYbrid knowledge Distillation and spectral Reconstruction Algorithm for high channel hyperspectral camera applications
• Stealing Neural Networks via Timing Side Channels
• Techniques for Symbol Grounding with SATNet
• Feature Representation Analysis of Deep Convolutional Neural Network using Two-stage Feature Transfer -An Application for Diffuse Lung Disease Classification-
• Teach an all-rounder with experts in different domains
• Collaborative Group Learning
• Ensemble Distillation for Neural Machine Translation
• Exact and Consistent Interpretation of Piecewise Linear Models Hidden behind APIs: A Closed Form Solution
• Compressing Convolutional Neural Networks
• Modeling Lost Information in Lossy Image Compression
• How to Manipulate CNNs to Make Them Lie: the GradCAM Case
• Cross-View Policy Learning for Street Navigation
• A Simple but Effective BERT Model for Dialog State Tracking on Resource-Limited Systems
• Self-Training With Noisy Student Improves ImageNet Classification
• Structured Knowledge Distillation for Dense Prediction
• Anomaly Detection in Univariate Time-series: A Survey on the State-of-the-Art
• Distillation of Large Language Models via Concrete Score Matching
• Towards Understanding Subliminal Learning: When and How Hidden Biases Transfer
• Rethinking Large Language Model Distillation: A Constrained Markov Decision Process Perspective
• On-the-fly Network Pruning for Object Detection
• Student Network Learning via Evolutionary Knowledge Distillation
• Zero-shot Knowledge Transfer via Adversarial Belief Matching
• Lets keep it simple, Using simple architectures to outperform deeper and\n more complex architectures
• Exact and Consistent Interpretation for Piecewise Linear Neural Networks: A Closed Form Solution
• Knowledge Squeezed Adversarial Network Compression
• Pretrained Transformers for Text Ranking: BERT and Beyond
• Privileged Knowledge Distillation for Online Action Detection
• ADD: Augmented Disentanglement Distillation Framework for Improving Stock Trend Forecasting
• When Does Preconditioning Help or Hurt Generalization?
• Improving the Learning of Multi-column Convolutional Neural Network for Crowd Counting
• Learning to Steer by Mimicking Features from Heterogeneous Auxiliary Networks
• Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation
• Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender System
• ReachNN: Reachability Analysis of Neural-Network Controlled Systems
• RGP: Neural Network Pruning through Its Regular Graph Structure
• Deep Learning for Medical Image Segmentation
• Knowledge Distillation for Sensing-Assisted Long-Term Beam Tracking in mmWave Communications
• Knowledge Transfer via Dense Cross-Layer Mutual-Distillation
• Sparsifying Neural Network Connections for Face Recognition
• Dataset Distillation
• Sequence-Level Knowledge Distillation
• Distilling Knowledge Using Parallel Data for Far-field Speech Recognition
• Network Implosion: Effective Model Compression for ResNets via Static Layer Pruning and Retraining
• From Consensus to Disagreement: Multi-Teacher Distillation for Semi-Supervised Relation Extraction
• Noisy Self-Knowledge Distillation for Text Summarization
• Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling
• Expandable Residual Approximation for Knowledge Distillation
• Improving the Interpretability of Deep Neural Networks with Knowledge Distillation
• NoScope: Optimizing Neural Network Queries over Video at Scale
• Verifiable Reinforcement Learning via Policy Extraction
• Variational Information Distillation for Knowledge Transfer
• Synthetic Adaptive Guided Embeddings (SAGE): A Novel Knowledge Distillation Method
• Enhancing Explainability of Neural Networks through Architecture Constraints
• Designing Interpretable Approximations to Deep Reinforcement Learning
• Paying more attention to snapshots of Iterative Pruning: Improving Model Compression via Ensemble Distillation
• EntropyGS: An Efficient Entropy Coding on 3D Gaussian Splatting
• Why distillation helps: a statistical perspective
• Task complexity shapes internal representations and robustness in neural networks
• FEED: Feature-level Ensemble for Knowledge Distillation
• Knowledge Distillation for End-to-End Person Search
• A Note on Knowledge Distillation Loss Function for Object Classification
• A copula-based visualization technique for a neural network
• Giraffe: Using Deep Reinforcement Learning to Play Chess
• ShrinkTeaNet: Million-scale Lightweight Face Recognition via Shrinking Teacher-Student Networks
• ASNN: Learning to Suggest Neural Architectures from Performance Distributions
• A Case for Backward Compatibility for Human-AI Teams
• Detecting Forged Facial Videos using convolutional neural network
• FATE: Fast and Accurate Timing Error Prediction Framework for Low Power DNN Accelerator Design
• Sequence Training and Adaptation of Highway Deep Neural Networks
• ProGMLP: A Progressive Framework for GNN-to-MLP Knowledge Distillation with Efficient Trade-offs
• BasisConv: A method for compressed representation and learning in CNNs
• Hardware-Software Codesign of Accurate, Multiplier-free Deep Neural Networks
• Towards Modality Transferable Visual Information Representation with Optimal Model Compression
• Fast Human Pose Estimation
• Secure Your Ride: Real-time Matching Success Rate Prediction for Passenger-Driver Pairs
• Emotion Recognition in Speech using Cross-Modal Transfer in the Wild
• GASL: Guided Attention for Sparsity Learning in Deep Neural Networks
• Efficient Representation of Low-Dimensional Manifolds using Deep Networks
• Distilling Object Detectors with Feature Richness
• Efficient and Private Federated Learning with Partially Trainable Networks
• SECS: Efficient Deep Stream Processing via Class Skew Dichotomy
• Developing efficient transfer learning strategies for robust scene recognition in mobile robotics using pre-trained convolutional neural networks
• Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
• The Pupil Has Become the Master: Teacher-Student Model-Based Word Embedding Distillation with Ensemble Learning
• In Defense of the Direct Perception of Affordances
• Adversarial Learning with Margin-based Triplet Embedding Regularization
• Interactive Knowledge Distillation
• Distilling Knowledge from Graph Convolutional Networks
• Preparing Lessons: Improve Knowledge Distillation with Better Supervision
• Feature Statistics Guided Efficient Filter Pruning
• Exploring Linear Relationship in Feature Map Subspace for ConvNets Compression
• Interpreting Deep Classifier by Visual Distillation of Dark Knowledge
• Transferring Inter-Class Correlation
• Cross-Layer Distillation with Semantic Calibration
• Crypto-Nets: Neural Networks over Encrypted Data
• Nonparametric Neural Networks
• Deep Transfer Learning with Ridge Regression
• Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge
• On the Depth of Deep Neural Networks: A Theoretical View
• Structure-Level Knowledge Distillation For Multilingual Sequence Labeling
• Deep Model Compression: Distilling Knowledge from Noisy Teachers
• ThUnderVolt: Enabling Aggressive Voltage Underscaling and Timing Error Resilience for Energy Efficient Deep Neural Network Accelerators
• Person Identification Based on Hand Tremor Characteristics
• Modularized Morphing of Neural Networks
• FitNets: Hints for Thin Deep Nets
• DS-Net++: Dynamic Weight Slicing for Efficient Inference in CNNs and Transformers
• "So, Tell Me About Your Policy...": Distillation of interpretable policies from Deep Reinforcement Learning agents
• Label Augmentation via Time-based Knowledge Distillation for Financial Anomaly Detection
• Compact Global Descriptor for Neural Networks
• Self-supervised Moving Vehicle Tracking with Stereo Sound
• Log-DenseNet: How to Sparsify a DenseNet
• Error Forward-Propagation: Reusing Feedforward Connections to Propagate Errors in Deep Learning
• A Selective Survey on Versatile Knowledge Distillation Paradigm for Neural Network Models
• VERITAS: Verification and Explanation of Realness in Images for Transparency in AI Systems
• The Power of Sparsity in Convolutional Neural Networks
• Channel Distillation: Channel-Wise Attention for Knowledge Distillation
• Balanced Knowledge Distillation for Long-tailed Learning
• Shakeout: A New Approach to Regularized Deep Neural Network Training
• Modality Distillation with Multiple Stream Networks for Action Recognition
• BlockSwap: Fisher-guided Block Substitution for Network Compression on a Budget
• Competitive Distillation: A Simple Learning Strategy for Improving Visual Classification
• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
• RGB-based 3D Hand Pose Estimation via Privileged Learning with Depth Images
• A Large RGB-D Dataset for Semi-supervised Monocular Depth Estimation
• Learning Anytime Predictions in Neural Networks via Adaptive Loss Balancing
• Building Lightweight Semantic Segmentation Models for Aerial Images Using Dual Relation Distillation
• DiceHuBERT: Distilling HuBERT with a Self-Supervised Learning Objective
• Principal Component Networks: Parameter Reduction Early in Training
• Adversarial Examples: Attacks and Defenses for Deep Learning
• Self-Teaching Machines to Read and Comprehend with Large-Scale Multi-Subject Question-Answering Data
• Dynamic-TinyBERT: Boost TinyBERT's Inference Efficiency by Dynamic Sequence Length
• Kickstarting Deep Reinforcement Learning
• Generative Knowledge Transfer for Neural Language Models
• Toward Fast and Accurate Neural Chinese Word Segmentation with Multi-Criteria Learning
• Inner Ensemble Networks: Average Ensemble as an Effective Regularizer
• Scalable Syntax-Aware Language Models Using Knowledge Distillation
• Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective
• Iterative Machine Teaching
• Efficient Object Embedding for Spliced Image Retrieval
• Are wider nets better given the same number of parameters?
• Leveraging Just a Few Keywords for Fine-Grained Aspect Detection Through Weakly Supervised Co-Training
• A One-step Pruning-recovery Framework for Acceleration of Convolutional Neural Networks
• Stealix: Model Stealing via Prompt Evolution
• Interpretable Few-Shot Learning via Linear Distillation
• Deep Epitome for Unravelling Generalized Hamming Network: A Fuzzy Logic Interpretation of Deep Learning
• Moonshine: Distilling with Cheap Convolutions
• A Framework for Behavioral Biometric Authentication using Deep Metric Learning on Mobile Devices
• Bayesian Compression for Deep Learning
• Born Again Neural Networks
• Distilling Knowledge for Search-based Structured Prediction
• CHEER: Rich Model Helps Poor Model via Knowledge Infusion
• Pruning via Iterative Ranking of Sensitivity Statistics
• Fingerprints: Fixed Length Representation via Deep Networks and Domain Knowledge
• Effective training of convolutional neural networks for age estimation based on knowledge distillation
• Advancing Multi-Accented LSTM-CTC Speech Recognition using a Domain Specific Student-Teacher Learning Paradigm
• Compositionality Through Language Transmission, using Artificial Neural Networks
• Residual Knowledge Distillation
• Hydra: Preserving Ensemble Diversity for Model Distillation
• FastSal: a Computationally Efficient Network for Visual Saliency Prediction
• The Promise of Spiking Neural Networks for Ubiquitous Computing: A Survey and New Perspectives
• Knowledge Distillation: A Survey
• Mixture of Expert/Imitator Networks: Scalable Semi-supervised Learning Framework
• Multilingual AMR Parsing with Noisy Knowledge Distillation
• Sketch Down the FLOPs: Towards Efficient Networks for Human Sketch
• Inference of a Multi-Domain Machine Learning Model to Predict Mortality in Hospital Stays for Patients with Cancer upon Febrile Neutropenia Onset
• Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniques
• Disentangling Locality and Entropy in Ranking Distillation
• Learning From Noisy Large-Scale Datasets With Minimal Supervision
• DOGe: Defensive Output Generation for LLM Protection Against Knowledge Distillation
• Be Your Own Best Competitor! Multi-Branched Adversarial Knowledge Transfer
• Incremental Meta-Learning via Indirect Discriminant Alignment
• Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structure
• Kronecker Recurrent Units
• Domain Robustness in Neural Machine Translation
• Channel Planting for Deep Neural Networks using Knowledge Distillation
• Distilling Pixel-Wise Feature Similarities for Semantic Segmentation
• Everything old is new again: A multi-view learning approach to learning using privileged information and distillation
• Predicting Deeper into the Future of Semantic Segmentation
• Structural Knowledge Distillation: Tractably Distilling Information for Structured Predictor
• Extreme Low Resolution Activity Recognition with Confident Spatial-Temporal Attention Transfer
• CoCoPIE: Making Mobile AI Sweet As PIE --Compression-Compilation Co-Design Goes a Long Way
• Blending LSTMs into CNNs
• RMMSS: Towards Advanced Robust Multi-Modal Semantic Segmentation with Hybrid Prototype Distillation and Feature Selection
• Why Knowledge Distillation Works in Generative Models: A Minimal Working Explanation
• Distilling Knowledge from Deep Networks with Applications to Healthcare Domain
• The Shallow End: Empowering Shallower Deep-Convolutional Networks through Auxiliary Outputs
• Prime-Aware Adaptive Distillation
• Data-Free Knowledge Distillation with Soft Targeted Transfer Set Synthesis
• Guiding CTC Posterior Spike Timings for Improved Posterior Fusion and Knowledge Distillation
• On the Turnpike to Design of Deep Neural Nets: Explicit Depth Bounds
• Towards Practical Lottery Ticket Hypothesis for Adversarial Training
• ClusterFit: Improving Generalization of Visual Representations
• Shapley Value as Principled Metric for Structured Network Pruning
• Model Extraction Attacks against Recurrent Neural Networks
• Black-Box Ripper: Copying black-box models using generative evolutionary algorithms
• Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Model
• Real-time Memory Efficient Large-pose Face Alignment via Deep Evolutionary Network
• Low-Latency Incremental Text-to-Speech Synthesis with Distilled Context Prediction Network
• A Studious Approach to Semi-Supervised Learning
• Few Sample Knowledge Distillation for Efficient Network Compression
• Adaptive Distillation: Aggregating Knowledge from Multiple Paths for Efficient Distillation
• Knowledge Distillation By Sparse Representation Matching
• RDPD: Rich Data Helps Poor Data via Imitation
• Collaborative Distillation for Ultra-Resolution Universal Style Transfer
• Demystifying Parallel and Distributed Deep Learning
• Data Efficient Stagewise Knowledge Distillation
• On the Orthogonality of Knowledge Distillation with Other Techniques: From an Ensemble Perspective
• DBP: Discrimination Based Block-Level Pruning for Deep Model Acceleration
• Transfer Learning for Speech and Language Processing
• Fixing the Teacher-Student Knowledge Discrepancy in Distillation
• Deep Mutual Learning
• Teacher-Student Compression with Generative Adversarial Networks
• Towards the Automatic Classification of Avian Flight Calls for Bioacoustic Monitoring
• Accelerating Large Scale Knowledge Distillation via Dynamic Importance Sampling
• Stochastic Model Pruning via Weight Dropping Away and Back
• Human-in-the-loop Extraction of Interpretable Concepts in Deep Learning Models
• Contrastive Semi-supervised Learning for ASR
• Deep Architectures for Modulation Recognition
• Copycat CNN: Are random non-Labeled data enough to steal knowledge from black-box models?
• Ranking to Learn and Learning to Rank: On the Role of Ranking in Pattern Recognition Applications
• Cross-lingual Distillation for Text Classification
• Matrix Factorization on GPUs with Memory Optimization and Approximate Computing
• Deep Learning Meets Sparse Regularization: A signal processing perspective
• Learning from Noisy Labels with Noise Modeling Network
• Knowledge Distillation by On-the-Fly Native Ensemble
• Direct Speech-to-Image Translation
• High-Efficiency Split Computing for Cooperative Edge Systems: A Novel Compressed Sensing Bottleneck
• ConvShareViT: Enhancing Vision Transformers with Convolutional Attention Mechanisms for Free-Space Optical Accelerators
• Distill-and-Compare
• Model-Agnostic Policy Explanations with Large Language Models
• Robust testing of low-dimensional functions
• SmartDeal: Re-Modeling Deep Network Weights for Efficient Inference and Training

Discussions

• Do Deep Nets Really Need to be Deep? [pdf] [hn, 126 points, 29 comments]
• Do Deep Nets Really Need to be Deep? [lobsters, 2 points, 0 comments]
• Shallow neural networks as good as Deep Neural Networks [hn, 2 points, 1 comments]
• Do Deep Nets Really Need to be Deep? [hn, 1 points, 0 comments]

Related

• Realizing data features by deep nets
• Do Deep Convolutional Nets Really Need to be Deep and Convolutional?
• How deep should be the depth of convolutional neural networks: a backyard dog case study
• Learning Functions: When Is Deep Better Than Shallow
• A Selective Overview of Deep Learning