Universally Slimmable Networks and Improved Training Techniques

2019/03/12 by Jiahui Yu, Thomas S. Huang, Yu, Jiahui +1 · 10 citations
Computer Science · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences

paper · pdf · doi:10.48550/arxiv.1903.05134

openalex publication_date 2019/03/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Slimmable networks are a family of neural networks that can instantly adjust the runtime width. The width can be chosen from a predefined widths set to adaptively optimize accuracy-efficiency trade-offs at runtime. In this work, we propose a systematic approach to train universally slimmable networks (US-Nets), extending slimmable networks to execute at arbitrary width, and generalizing to networks both with and without batch normalization layers. We further propose two improved training techniques for US-Nets, named the sandwich rule and inplace distillation, to enhance training process and boost testing accuracy. We show improved performance of universally slimmable MobileNet v1 and MobileNet v2 on ImageNet classification task, compared with individually trained ones and 4-switch slimmable network baselines. We also evaluate the proposed US-Nets and improved training techniques on tasks of image super-resolution and deep reinforcement learning. Extensive ablation experiments on these representative tasks demonstrate the effectiveness of our proposed methods. Our discovery opens up the possibility to directly evaluate FLOPs-Accuracy spectrum of network architectures. Code and models are available at: https://github.com/JiahuiYu/slimmable_networks

Citations

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal\n Networks

Cited by

• Exploring Deep-to-Shallow Transformable Neural Networks for Intelligent Embedded Systems
• Breaking the Capacity Bottleneck in Model-Heterogeneous Federated Learning via Gradual Model Restoration
• Memory-Efficient Training with In-Place FFT Implementation
• Elastic ViTs from Pretrained Models without Retraining
• Optimally Deep Networks -- Adapting Model Depth to Datasets for Superior Efficiency
• MoE-PHDS: One MoE checkpoint for flexible runtime sparsity
• RAM-NAS: Resource-aware Multiobjective Neural Architecture Search Method for Robot Vision Tasks
• Detecting Positive Selection by Modeling Structure Within Images of Genetic Variation
• Deep Hierarchical Learning with Nested Subspace Networks for Large Language Models
• SOLAR: Switchable Output Layer for Accuracy and Robustness in Once-for-All Training

Related

• Slimmable Neural Networks
• Joslim: Joint Widths and Weights Optimization for Slimmable Neural Networks
• Slimmable Pruned Neural Networks
• Dynamic Slimmable Network
• DSPNet: Towards Slimmable Pretrained Networks based on Discriminative Self-supervised Learning