SHARCS: Efficient Transformers through Routing with Dynamic Width Sub-networks

2023/10/18 by Mohammadreza Salehi, Sachin Mehta, Salehi, Mohammadreza +7 · 1 citation
Computer Science · #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2310.12126

openalex publication_date 2023/10/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

We introduce SHARCS for adaptive inference that takes into account the hardness of input samples. SHARCS can train a router on any transformer network, enabling the model to direct different samples to sub-networks with varying widths. Our experiments demonstrate that: (1) SHARCS outperforms or complements existing per-sample adaptive inference methods across various classification tasks in terms of accuracy vs. FLOPs; (2) SHARCS generalizes across different architectures and can be even applied to compressed and efficient transformer encoders to further improve their efficiency; (3) SHARCS can provide a 2 times inference speed up at an insignificant drop in accuracy.

Cited by

• Efficiency Robustness of Dynamic Deep Learning Systems

Related

• Router-Tuning: A Simple and Effective Approach for Enabling Dynamic-Depth in Transformers
• Subjective Depth and Timescale Transformers: Learning Where and When to Compute
• ElastiFormer: Learned Redundancy Reduction in Transformer via Self-Distillation
• AdapterDrop: On the Efficiency of Adapters in Transformers
• Grounding latent algorithm routing in transformer reasoning