2019/12/31 by Jeffrey O. Zhang, Zhang, Jeffrey O, Alexander F. Sax +7 · 1 citation
Computer Science · Medicine · #COVID-19 diagnosis using AI #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Neural and Evolutionary Computing (cs.NE) #Robotics (cs.RO)
paper · pdf · doi:10.48550/arxiv.1912.13503
openalex publication_date 2019/12/31 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
When training a neural network for a desired task, one may prefer to adapt a\npre-trained network rather than starting from randomly initialized weights.\nAdaptation can be useful in cases when training data is scarce, when a single\nlearner needs to perform multiple tasks, or when one wishes to encode priors in\nthe network. The most commonly employed approaches for network adaptation are\nfine-tuning and using the pre-trained network as a fixed feature extractor,\namong others.\n In this paper, we propose a straightforward alternative: side-tuning.\nSide-tuning adapts a pre-trained network by training a lightweight "side"\nnetwork that is fused with the (unchanged) pre-trained network via summation.\nThis simple method works as well as or better than existing solutions and it\nresolves some of the basic issues with fine-tuning, fixed features, and other\ncommon approaches. In particular, side-tuning is less prone to overfitting, is\nasymptotically consistent, and does not suffer from catastrophic forgetting in\nincremental learning. We demonstrate the performance of side-tuning under a\ndiverse set of scenarios, including incremental learning (iCIFAR, iTaskonomy),\nreinforcement learning, imitation learning (visual navigation in Habitat), NLP\nquestion-answering (SQuAD v2), and single-task transfer learning (Taskonomy),\nwith consistently promising results.\n