Preserve and Sculpt: Manifold-Aligned Fine-tuning of Vision-Language Models for Few-Shot Learning

2025/08/18 by Dexia Chen, Chen, Dexia, Zhu, Qianjie +8
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2508.12877

openalex publication_date 2025/08/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Pretrained vision-language models (VLMs), such as CLIP, have shown remarkable potential in few-shot image classification and led to numerous effective transfer learning strategies. These methods leverage the pretrained knowledge of VLMs to enable effective domain adaptation while mitigating overfitting through parameter-efficient tuning or instance-based consistency constraints. However, such regularizations often neglect the geometric structure of data distribution, which may lead to distortion of the overall semantic representation. To overcome this limitation, we propose a novel fine-tuning method, Manifold-Preserving and Sculpting Tuning (MPS-Tuning). Regarding the data distribution in feature space as a semantic manifold, MPS-Tuning explicitly constrains the intrinsic geometry of this manifold while further sculpting it to enhance class separability. Specifically, MPS-Tuning preserves both macroscopic and microscopic topological structures of the original manifold by aligning Gram matrices of features before and after fine-tuning. Theoretically, this constraint is shown to approximate an upper bound of the Gromov-Wasserstein distance. Furthermore, features from the image and text modalities are paired, and pairwise similarities are optimized to enhance the manifold's class discriminability. Extensive experiments demonstrate that MPS-Tuning significantly improves model performance while effectively preserving the structure of the semantic manifold. The code will be released.

Citations

• SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
• Skip Tuning: Pre-trained Vision-Language Models are Effective and Efficient Adapters Themselves
• TextRefiner: Internal Visual Feature as Efficient Refiner for Vision-Language Models Prompt Tuning
• Compositional Entailment Learning for Hyperbolic Vision-Language Models
• Interpreting CLIP's Image Representation via Text-Based Decomposition
• Self-regulating Prompts: Foundational Model Adaptation without Forgetting
• Structure-CLIP: Towards Scene Graph Knowledge to Enhance Multi-modal Structured Representations
• SoftCLIP: Softer Cross-modal Alignment Makes CLIP Stronger
• Sigmoid Loss for Language Image Pre-Training
• EVA-CLIP: Improved Training Techniques for CLIP at Scale
• CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attention
• Prompt-aligned Gradient for Prompt Tuning
• CLIP-Adapter: Better Vision-Language Models with Feature Adapters
• CLIP-Adapter: Better Vision-Language Models with Feature Adapters
• Learning Transferable Visual Models From Natural Language Supervision
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Do ImageNet Classifiers Generalize to ImageNet?
• EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification
• Describing Textures in the Wild
• Visualizing and Understanding Convolutional Networks
• Fine-Grained Visual Classification of Aircraft
• UCF-101: A dataset of 101 human actions classes from videos in the wild

Related

• Topology-Aware CLIP Few-Shot Learning
• Difference Vector Equalization for Robust Fine-tuning of Vision-Language Models
• Infusing fine-grained visual knowledge to Vision-Language Models
• Continual Learning with Vision-Language Models via Semantic-Geometry Preservation
• Cross-Domain Few-Shot Learning via Multi-View Collaborative Optimization with Vision-Language Models