2022/06/12 by Sanghoon Myung, In Huh, Myung, Sanghoon +13 · 1 citation
Computer Science · #Applications (stat.AP) #Artificial Intelligence (cs.AI) #Computational Physics (physics.comp-ph) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #FOS: Physical sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification #Machine Learning and ELM
paper · pdf · doi:10.48550/arxiv.2206.05703
openalex publication_date 2022/06/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Inductive transfer learning aims to learn from a small amount of training data for the target task by utilizing a pre-trained model from the source task. Most strategies that involve large-scale deep learning models adopt initialization with the pre-trained model and fine-tuning for the target task. However, when using over-parameterized models, we can often prune the model without sacrificing the accuracy of the source task. This motivates us to adopt model pruning for transfer learning with deep learning models. In this paper, we propose PAC-Net, a simple yet effective approach for transfer learning based on pruning. PAC-Net consists of three steps: Prune, Allocate, and Calibrate (PAC). The main idea behind these steps is to identify essential weights for the source task, fine-tune on the source task by updating the essential weights, and then calibrate on the target task by updating the remaining redundant weights. Under the various and extensive set of inductive transfer learning experiments, we show that our method achieves state-of-the-art performance by a large margin.