Leveraging Vision-Language Models for Improving Domain Generalization in Image Classification

2023/10/12 by Sravanti Addepalli, Ashish Ramayee Asokan, Addepalli, Sravanti +5 · 17 citations
Computer Science · Engineering · Mathematics · Psychology · #Artificial intelligence #Computer Vision and Pattern Recognition (cs.CV) #Computer science #FOS: Computer and information sciences #Gaze Tracking and Assistive Technology #Generalization #Mathematics #Natural language processing #Safety Warnings and Signage #Spatial Cognition and Navigation #Vision science

paper · pdf · doi:10.48550/arxiv.2310.08255

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2023/10/12 · openalex created_date 2023/10/14 · openalex updated_date 2026/07/28

Abstract

Vision-Language Models (VLMs) such as CLIP are trained on large amounts of image-text pairs, resulting in remarkable generalization across several data distributions. However, in several cases, their expensive training and data collection/curation costs do not justify the end application. This motivates a vendor-client paradigm, where a vendor trains a large-scale VLM and grants only input-output access to clients on a pay-per-query basis in a black-box setting. The client aims to minimize inference cost by distilling the VLM to a student model using the limited available task-specific data, and further deploying this student model in the downstream application. While naive distillation largely improves the In-Domain (ID) accuracy of the student, it fails to transfer the superior out-of-distribution (OOD) generalization of the VLM teacher using the limited available labeled images. To mitigate this, we propose Vision-Language to Vision - Align, Distill, Predict (VL2V-ADiP), which first aligns the vision and language modalities of the teacher model with the vision modality of a pre-trained student model, and further distills the aligned VLM representations to the student. This maximally retains the pre-trained features of the student, while also incorporating the rich representations of the VLM image encoder and the superior generalization of the text embeddings. The proposed approach achieves state-of-the-art results on the standard Domain Generalization benchmarks in a black-box teacher setting as well as a white-box setting where the weights of the VLM are accessible.

Cited by

• All Centers Are at most a Few Tokens Apart: Knowledge Distillation with Domain Invariant Prompt Tuning
• Enhanced Motion Forecasting with Plug-and-Play Multimodal Large Language Models
• Revisiting Mixout: An Overlooked Path to Robust Finetuning
• Preserving Domain Generalization in Fine-Tuning via Joint Parameter Selection
• An Explainable Deep Neural Network with Frequency-Aware Channel and Spatial Refinement for Flood Prediction in Sustainable Cities
• Target-Oriented Single Domain Generalization
• Domain Generalization in-the-Wild: Disentangling Classification from Domain-Aware Representations
• Object Detection with Multimodal Large Vision-Language Models: An In-depth Review
• Multimodal Causal-Driven Representation Learning for Generalizable Medical Image Segmentation
• Negation-Aware Test-Time Adaptation for Vision-Language Models
• Advancing Vision-based Human Action Recognition: Exploring Vision-Language CLIP Model for Generalisation in Domain-Independent Tasks
• Harmonizing and Merging Source Models for CLIP-based Domain Generalization
• Prompt Disentanglement via Language Guidance and Representation Alignment for Domain Generalization
• Generalizing vision-language models to novel domains: A comprehensive survey
• Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directions
• Handling Imbalanced Pseudolabels for Vision-Language Models with Concept Alignment and Confusion-Aware Calibrated Margin
• A Review of 3D Object Detection with Vision-Language Models
• CLIP-Powered Domain Generalization and Domain Adaptation: A Comprehensive Survey

Related

• A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language Guidance
• Generalizing Vision-Language Models with Dedicated Prompt Guidance
• Efficient and Long-Tailed Generalization for Pre-trained Vision-Language Model
• Simple yet Effective Semi-supervised Knowledge Distillation from Vision-Language Models via Dual-Head Optimization
• Latent Domain Prompt Learning for Vision-Language Models