2024/09/22 by Cheng‐En Wu, Wu, Cheng-En, Jin‐Hong Lin +5 · 1 citation
Computer Science · Mathematics · #Artificial intelligence #Combinatorics #Computer science #Image Retrieval and Classification Techniques #Learning to rank #Mathematics #Rank (graph theory) #Ranking (information retrieval) #Rough Sets and Fuzzy Logic #Text and Document Classification Technologies
paper · pdf · doi:10.48550/arxiv.2409.14607
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2024/09/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Contrastive image-text pre-trained models such as CLIP have shown remarkable adaptability to downstream tasks. However, they face challenges due to the high computational requirements of the Vision Transformer (ViT) backbone. Current strategies to boost ViT efficiency focus on pruning patch tokens but fall short in addressing the multimodal nature of CLIP and identifying the optimal subset of tokens for maximum performance. To address this, we propose greedy search methods to establish a "Golden Ranking" and introduce a lightweight predictor specifically trained to approximate this Ranking. To compensate for any performance degradation resulting from token pruning, we incorporate learnable visual tokens that aid in restoring and potentially enhancing the model's performance. Our work presents a comprehensive and systematic investigation of pruning tokens within the ViT backbone of CLIP models. Through our framework, we successfully reduced 40% of patch tokens in CLIP's ViT while only suffering a minimal average accuracy loss of 0.3 across seven datasets. Our study lays the groundwork for building more computationally efficient multimodal models without sacrificing their performance, addressing a key challenge in the application of advanced vision-language models.