Leveraging Vision-Language Models for Improving Domain Generalization in Image Classification

2023/10/12 by Addepalli, Sravanti, Asokan, Ashish Ramayee, Sharma, Lakshay +1 · 11 citations
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2310.08255

Abstract

Vision-Language Models (VLMs) such as CLIP are trained on large amounts of image-text pairs, resulting in remarkable generalization across several data distributions. However, in several cases, their expensive training and data collection/curation costs do not justify the end application. This motivates a vendor-client paradigm, where a vendor trains a large-scale VLM and grants only input-output access to clients on a pay-per-query basis in a black-box setting. The client aims to minimize inference cost by distilling the VLM to a student model using the limited available task-specific data, and further deploying this student model in the downstream application. While naive distillation largely improves the In-Domain (ID) accuracy of the student, it fails to transfer the superior out-of-distribution (OOD) generalization of the VLM teacher using the limited available labeled images. To mitigate this, we propose Vision-Language to Vision - Align, Distill, Predict (VL2V-ADiP), which first aligns the vision and language modalities of the teacher model with the vision modality of a pre-trained student model, and further distills the aligned VLM representations to the student. This maximally retains the pre-trained features of the student, while also incorporating the rich representations of the VLM image encoder and the superior generalization of the text embeddings. The proposed approach achieves state-of-the-art results on the standard Domain Generalization benchmarks in a black-box teacher setting as well as a white-box setting where the weights of the VLM are accessible.

Cited by

• All Centers Are at most a Few Tokens Apart: Knowledge Distillation with Domain Invariant Prompt Tuning
• Enhanced Motion Forecasting with Plug-and-Play Multimodal Large Language Models
• Revisiting Mixout: An Overlooked Path to Robust Finetuning
• Preserving Domain Generalization in Fine-Tuning via Joint Parameter Selection
• An Explainable Deep Neural Network with Frequency-Aware Channel and Spatial Refinement for Flood Prediction in Sustainable Cities
• Target-Oriented Single Domain Generalization
• Domain Generalization in-the-Wild: Disentangling Classification from Domain-Aware Representations
• Object Detection with Multimodal Large Vision-Language Models: An In-depth Review
• Multimodal Causal-Driven Representation Learning for Generalizable Medical Image Segmentation
• Negation-Aware Test-Time Adaptation for Vision-Language Models
• Advancing Vision-based Human Action Recognition: Exploring Vision-Language CLIP Model for Generalisation in Domain-Independent Tasks

Related

• A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language Guidance
• Generalizing Vision-Language Models with Dedicated Prompt Guidance
• Efficient and Long-Tailed Generalization for Pre-trained Vision-Language Model
• Simple yet Effective Semi-supervised Knowledge Distillation from Vision-Language Models via Dual-Head Optimization
• Latent Domain Prompt Learning for Vision-Language Models