Deeply-Supervised Nets

2014/09/18 by Chen‐Yu Lee, Saining Xie, Lee, Chen-Yu +7 · 47 citations
Computer Science · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural and Evolutionary Computing (cs.NE)

paper · pdf · doi:10.48550/arxiv.1409.5185

openalex publication_date 2014/09/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Our proposed deeply-supervised nets (DSN) method simultaneously minimizes classification error while making the learning process of hidden layers direct and transparent. We make an attempt to boost the classification performance by studying a new formulation in deep networks. Three aspects in convolutional neural networks (CNN) style architectures are being looked at: (1) transparency of the intermediate layers to the overall classification; (2) discriminativeness and robustness of learned features, especially in the early layers; (3) effectiveness in training due to the presence of the exploding and vanishing gradients. We introduce "companion objective" to the individual hidden layers, in addition to the overall objective at the output layer (a different strategy to layer-wise pre-training). We extend techniques from stochastic gradient methods to analyze our algorithm. The advantage of our method is evident and our experimental result on benchmark datasets shows significant performance gain over existing methods (e.g. all state-of-the-art results on MNIST, CIFAR-10, CIFAR-100, and SVHN).

Citations

• Stochastic Gradient Descent for Non-smooth Optimization: Convergence\n Results and Optimal Averaging Schemes
• Gradient-based learning applied to document recognition
• Regularized M-estimators with nonconvexity: Statistical and algorithmic theory for local optima
• Visualizing and Understanding Convolutional Networks

Cited by

• SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervision
• AMD-HookNet++: Evolution of AMD-HookNet with Hybrid CNN-Transformer Feature Enhancement for Glacier Calving Front Segmentation
• Optimized Architectures for Kolmogorov-Arnold Networks
• FedDSR: Federated Deep Supervision and Regularization Towards Autonomous Driving
• MasHeNe: A Benchmark for Head and Neck CT Mass Segmentation using Window-Enhanced Mamba with Frequency-Domain Integration
• Panda: Self-distillation of Reusable Sensor-level Representations for High Energy Physics
• BackSplit: The Importance of Sub-dividing the Background in Biomedical Lesion Segmentation
• LAYA: Layer-wise Attention Aggregation for Interpretable Depth-Aware Neural Networks
• An ultrafast plenoptic-camera system for high-resolution 3D particle tracking in unsegmented scintillators
• Accelerating Training Speed of Tiny Recursive Models with Curriculum Guided Adaptive Recursion
• Multi-period Learning for Financial Time Series Forecasting
• Deep Roto-Translation Scattering for Object Classification
• Decoupled Multi-Predictor Optimization for Inference-Efficient Model Tuning
• Automatic Moth Detection from Trap Images for Pest Management
• FractalNet: Ultra-Deep Neural Networks without Residuals
• Synthetic-to-Real Transfer Learning for Chromatin-Sensitive PWS Microscopy
• YOLOv7: Trainable Bag-of-Freebies Sets New State-of-the-Art for Real-Time Object Detectors
• Deep Networks with Stochastic Depth
• Discriminative Unsupervised Feature Learning with Exemplar Convolutional\n Neural Networks
• Random Window Augmentations for Deep Learning Robustness in CT and Liver Tumor Segmentation
• End-to-End Test-Time Training for Long Context
• Wide Residual Networks
• CNN-based Segmentation of Medical Imaging Data
• Colorful Image Colorization
• PASCAL Boundaries: A Class-Agnostic Semantic Boundary Dataset
• Spatial Aggregation of Holistically-Nested Networks for Automated\n Pancreas Segmentation
• Learning to Diagnose with LSTM Recurrent Neural Networks
• MOCHA: Multi-modal Objects-aware Cross-arcHitecture Alignment
• DiracNets: Training Very Deep Neural Networks Without Skip-Connections
• WhisTLE: Deeply Supervised, Text-Only Domain Adaptation for Pretrained Speech Recognition Transformers
• A Critical Review of Recurrent Neural Networks for Sequence Learning
• Encoder-Only Image Registration
• Failure Prediction Is a Better Performance Proxy for Early-Exit Networks Than Calibration
• Similarity-based Text Recognition by Deeply Supervised Siamese Network
• Deep Haar Scattering Networks
• 3D Deeply Supervised Network for Automatic Liver Segmentation from CT Volumes
• Training Deeper Convolutional Networks with Deep Supervision
• KonfAI: A Modular and Fully Configurable Framework for Deep Learning in Medical Imaging
• GAPNet: A Lightweight Framework for Image and Video Salient Object Detection via Granularity-Aware Paradigm
• Guided Variational Autoencoder for Disentanglement Learning
• Eigen Neural Network: Unlocking Generalizable Vision with Eigenbasis
• Multi-output Deep-Supervised Classifier Chains for Plant Pathology
• Extreme Cardiac MRI Analysis under Respiratory Motion: Results of the CMRxMotion Challenge
• Collaborative Layer-wise Discriminative Learning in Deep Neural Networks
• A Closed-Form Learned Pooling for Deep Classification Networks
• Spatial Aggregation of Holistically-Nested Convolutional Neural Networks\n for Automated Pancreas Localization and Segmentation
• Higher Order Recurrent Neural Networks
• OPRD: On-Policy Representation Distillation

Related

• Training Deeper Convolutional Networks with Deep Supervision
• Greedy Layerwise Learning Can Scale to ImageNet
• Deep Networks from the Principle of Rate Reduction
• Deep Networks with Stochastic Depth
• How deep should be the depth of convolutional neural networks: a backyard dog case study