A Second-Order Perspective on Pruning at Initialization and Knowledge Transfer

2025/09/28 by Iurada, Leonardo, Occhiena, Beatrice, Tommasi, Tatiana
#Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2509.24066

Abstract

The widespread availability of pre-trained vision models has enabled numerous deep learning applications through their transferable representations. However, their computational and storage costs often limit practical deployment. Pruning-at-Initialization has emerged as a promising approach to compress models before training, enabling efficient task-specific adaptation. While conventional wisdom suggests that effective pruning requires task-specific data, this creates a challenge when downstream tasks are unknown in advance. In this paper, we investigate how data influences the pruning of pre-trained vision models. Surprisingly, pruning on one task retains the model's zero-shot performance also on unseen tasks. Furthermore, fine-tuning these pruned models not only improves performance on original seen tasks but can recover held-out tasks' performance. We attribute this phenomenon to the favorable loss landscapes induced by extensive pre-training on large-scale datasets.

Citations

• Efficient Model Editing with Task-Localized Sparse Fine-tuning
• Finding Lottery Tickets in Vision Models via Data-driven Spectral Foresight Pruning
• Accelerating Heterogeneous Federated Learning with Closed-form Classifiers
• Transfer Learning for Structured Pruning under Limited Task Data
• On the Convergence of Encoder-only Shallow Transformers
• Kronecker-Factored Approximate Curvature for Modern Neural Network Architectures
• What do neural networks learn in image classification? A frequency shortcut perspective
• A Simple and Effective Pruning Approach for Large Language Models
• NTK-SAP: Improving neural network pruning by aligning training dynamics
• SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot
• Emerging Properties in Self-Supervised Vision Transformers
• Emerging Properties in Self-Supervised Vision Transformers
• TransTailor: Pruning the Pre-trained Model for Improved Transfer Learning
• Learning Transferable Visual Models From Natural Language Supervision
• Unlearnable Examples: Making Personal Data Unexploitable
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• PHEW: Constructing Sparse Networks that Learn Fast and Generalize Well\n without Training Data
• Pruning neural networks without any data by iteratively conserving synaptic flow
• Improved Baselines with Momentum Contrastive Learning
• Compressing BERT: Studying the Effects of Weight Pruning on Transfer\n Learning
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Deep Residual Learning for Image Recognition
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• One-Shot Pruning for Fast-adapting Pre-trained Models on Devices
• When Does Pruning Benefit Vision Representations?
• Prospect Pruning: Finding Trainable Weights at Initialization using Meta-Gradients
• Reconstruction Task Finds Universal Winning Tickets
• Exploring The Neural Burden In Pruned Models: An Insight Inspired By Neuroscience