Improved Fine-Tuning by Better Leveraging Pre-Training Data

2021/11/24 by Ziquan Liu, Liu, Ziquan, Yi Xu +13 · 5 citations
Computer Science · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.2111.12292

openalex publication_date 2021/11/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

As a dominant paradigm, fine-tuning a pre-trained model on the target data is widely used in many deep learning applications, especially for small data sets. However, recent studies have empirically shown that training from scratch has the final performance that is no worse than this pre-training strategy once the number of training samples is increased in some vision tasks. In this work, we revisit this phenomenon from the perspective of generalization analysis by using excess risk bound which is popular in learning theory. The result reveals that the excess risk bound may have a weak dependency on the pre-trained model. The observation inspires us to leverage pre-training data for fine-tuning, since this data is also available for fine-tuning. The generalization result of using pre-training data shows that the excess risk bound on a target task can be improved when the appropriate pre-training data is included in fine-tuning. With the theoretical motivation, we propose a novel selection strategy to select a subset from pre-training data to help improve the generalization on the target task. Extensive experimental results for image classification tasks on 8 benchmark data sets verify the effectiveness of the proposed data selection based fine-tuning pipeline.

Cited by

• Fine-Tuning Diffusion-Based Recommender Systems via Reinforcement Learning with Reward Function Optimization
• An Empirical Study of Sample Selection Strategies for Large Language Model Repair
• Midtraining Bridges Pretraining and Posttraining Distributions
• Classifier Language Models: Unifying Sparse Finetuning and Adaptive Tokenization for Specialized Classification Tasks
• From Binary to Semantic: Utilizing Large-Scale Binary Occupancy Data for 3D Semantic Occupancy Prediction

Related

• On the Connection between Pre-training Data Diversity and Fine-tuning Robustness
• A Guide to Robust Generalization: The Impact of Architecture, Pre-training, and Optimization Strategy
• Rethinking the Hyperparameters for Fine-tuning
• Data Dropout: Optimizing Training Data for Convolutional Neural Networks
• An Empirical Study of Pre-trained Model Selection for Out-of-Distribution Generalization and Calibration