iMacHSR: Intermediate Multi-Access Heterogeneous Supervision and Regularization Scheme Toward Architecture-Agnostic Training

2025/05/01 by Wei-Bin Kou, Guangxu Zhu, Kou, Wei-Bin +10 · 1 citation
Computer Science · #Advanced Graph Neural Networks #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Robotics (cs.RO)

paper · pdf · doi:10.48550/arxiv.2505.00404

openalex publication_date 2025/05/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

While deep supervision is a powerful training strategy by supervising intermediate layers with auxiliary losses, it faces three underexplored problems: (I) Existing deep supervision techniques are generally bond with specific model architectures strictly, lacking generality. (II) The identical loss function for intermediate and output layers causes intermediate layers to prioritize output-specific features prematurely, limiting generalizable representations. (III) Lacking regularization on hidden activations risks overconfident predictions, reducing generalization to unseen scenarios. To tackle these challenges, we propose an architecture-agnostic, intermediate Multi-access Heterogeneous Supervision and Regularization (iMacHSR) scheme. Specifically, the proposed iMacHSR introduces below integral strategies: (I) we select multiple intermediate layers based on predefined architecture-agnostic standards; (II) loss functions (different from output-layer loss) are applied to those selected intermediate layers, which can guide intermediate layers to learn diverse and hierarchical representations; and (III) negative entropy regularization on selected layers' hidden features discourages overconfident predictions and mitigates overfitting. These intermediate terms are combined into the output-layer training loss to form a unified optimization objective, enabling comprehensive optimization across the network hierarchy. We then take the semantic understanding task as an example to assess iMacHSR and apply iMacHSR to several model architectures. Extensive experiments on multiple datasets demonstrate that iMacHSR outperforms conventional output-layer single-point supervision method up to 9.19% in mIoU.

Citations

• Nonlocal Attention Operator: Materializing Hidden Knowledge Towards Interpretable Physics Discovery
• Residual Alignment: Uncovering the Mechanisms of Residual Networks
• SCTNet: Single-Branch CNN with Transformer Semantic Information for Real-Time Segmentation
• DP-AdamBC: Your DP-Adam Is Actually DP-SGD (Unless You Apply Bias Correction)
• Agent Attention: On the Integration of Softmax and Linear Attention
• Boosting Residual Networks with Group Knowledge
• PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning
• Self-Supervised Primal-Dual Learning for Constrained Optimization
• Residual Local Feature Network for Efficient Super-Resolution
• TopFormer: Token Pyramid Transformer for Mobile Semantic Segmentation
• AdaAttN: Revisit Attention Mechanism in Arbitrary Neural Style Transfer
• Manipulating SGD with Data Ordering Attacks
• A Hybrid Attention Mechanism for Weakly-Supervised Temporal Action Localization
• LabelEnc: A New Intermediate Supervision Method for Object Detection
• Gradient Boosts the Approximate Vanishing Ideal
• Deep High-Resolution Representation Learning for Visual Recognition
• Deep High-Resolution Representation Learning for Visual Recognition
• Gated-SCNN: Gated Shape CNNs for Semantic Segmentation
• BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation
• Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
• Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?
• ICNet for Real-Time Semantic Segmentation on High-Resolution Images
• Deep Supervision with Shape Concepts for Occlusion-Aware 3D Object Parsing
• Pyramid Scene Parsing Network
• DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
• DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
• Cityscapes dataset for semantic urban scene understanding
• Deep Residual Learning for Image Recognition
• Training Deeper Convolutional Networks with Deep Supervision
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Adam: A Method for Stochastic Optimization
• Deeply-Supervised Nets
• Going Deeper with Convolutions
• Improving neural networks by preventing co-adaptation of feature detectors
• Ridge Regression: Biased Estimation for Nonorthogonal Problems
• SeaFormer++: Squeeze-enhanced Axial Transformer for Mobile Visual Recognition

Cited by

• PRISM: Privileged Probabilistic Latent Supervision for End-to-End Autonomous Driving Motion Planning

Related

• SHA256 at SemEval-2025 Task 4: Selective Amnesia -- Constrained Unlearning for Large Language Models via Knowledge Isolation
• Guillotine Regularization: Why removing layers is needed to improve generalization in Self-Supervised Learning
• Multi-Head Attention Residuals
• No Reason for No Supervision: Improved Generalization in Supervised Models
• A New Benchmark and Progress Toward Improved Weakly Supervised Learning