MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training

2023/11/28 by Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Vasu, Pavan Kumar Anasosalu +7 · 1 voice · 35 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Artificial intelligence #Closed captioning #Computer engineering #Computer science #Domain Adaptation and Few-Shot Learning #Encoder #Image (mathematics) #Latency (audio) #Machine learning #Modal #Multimodal Machine Learning Applications #Reinforcement learning #Robustness (evolution) #Transformer #Voltage #cs.CL #cs.CV #cs.LG

paper · pdf · doi:10.48550/arxiv.2311.17049

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2023/11/28 · arxiv published 2023/11/28 · arxiv updated 2024/04/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Contrastive pretraining of image-text foundation models, such as CLIP, demonstrated excellent zero-shot performance and improved robustness on a wide range of downstream tasks. However, these models utilize large transformer-based encoders with significant memory and latency overhead which pose challenges for deployment on mobile devices. In this work, we introduce MobileCLIP -- a new family of efficient image-text models optimized for runtime performance along with a novel and efficient training approach, namely multi-modal reinforced training. The proposed training approach leverages knowledge transfer from an image captioning model and an ensemble of strong CLIP encoders to improve the accuracy of efficient models. Our approach avoids train-time compute overhead by storing the additional knowledge in a reinforced dataset. MobileCLIP sets a new state-of-the-art latency-accuracy tradeoff for zero-shot classification and retrieval tasks on several datasets. Our MobileCLIP-S2 variant is 2.3× faster while more accurate compared to previous best CLIP model based on ViT-B/16. We further demonstrate the effectiveness of our multi-modal reinforced training by training a CLIP model based on ViT-B/16 image backbone and achieving +2.9% average performance improvement on 38 evaluation benchmarks compared to the previous best. Moreover, we show that the proposed approach achieves 10×-1000× improved learning efficiency when compared with non-reinforced CLIP training. Code and models are available at https://github.com/apple/ml-mobileclip .

Cited by

• Evaluation of Winning Solutions of 2025 Low Power Computer Vision Challenge
• InternVideo-Next: Towards General Video Foundation Models without Video-Text Supervision
• When Better Teachers Don't Make Better Students: Revisiting Knowledge Distillation for CLIP Models in VQA
• NeuCLIP: Efficient Large-Scale CLIP Training with Neural Normalizer Optimization
• Architecture, Simulation and Software Stack to Support Post-CMOS Accelerators: The ARCHYTAS Project
• A Matter of Time: Revealing the Structure of Time in Vision-Language Models
• Representing Beauty: Towards a Participatory but Objective Latent Aesthetics
• Generalized Contrastive Learning for Universal Multimodal Retrieval
• WAVECLIP: Wavelet Tokenization for Adaptive-Resolution CLIP
• Scaling Vision-Language Models Is Not Enough to Mitigate Bias
• EchoVLM: Dynamic Mixture-of-Experts Vision-Language Model for Universal Ultrasound Intelligence
• An Explainable Deep Neural Network with Frequency-Aware Channel and Spatial Refinement for Flood Prediction in Sustainable Cities
• MobileCLIP2: Improving Multi-Modal Reinforced Training
• Hierarchical Vision-Language Retrieval of Educational Metaverse Content in Agriculture
• MobileViCLIP: An Efficient Video-Text Model for Mobile Devices
• Dual Prompt Learning for Adapting Vision-Language Models to Downstream Image-Text Retrieval
• Evading Data Provenance in Deep Neural Networks
• Chat with AI: The Surprising Turn of Real-time Video Communication from Human to AI
• ZIP: Scalable Crowd Counting via Zero-Inflated Poisson Modeling
• Da Yu: Towards USV-Based Image Captioning for Waterway Surveillance and Scene Understanding
• X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIP
• DualMap: Online Open-Vocabulary Semantic Mapping for Natural Language Navigation in Dynamic Changing Scenes
• PCoreSet: Effective Active Learning through Knowledge Distillation from Vision-Language Models
• Using Knowledge Graphs to harvest datasets for efficient CLIP model training
• Puzzled by Puzzles: When Vision-Language Models Can't Take a Hint
• From What to How: Attributing CLIP's Latent Components Reveals Unexpected Semantic Reliance
• FDBPL: Faster Distillation-Based Prompt Learning for Region-Aware Vision-Language Models Adaptation
• Simple yet Effective Semi-supervised Knowledge Distillation from Vision-Language Models via Dual-Head Optimization
• Model Steering: Learning with a Reference Model Improves Generalization Bounds and Scaling Laws
• AdaDINO: Context-Adaptive DINO-Distilled Vision Foundation Models for Efficient Open-Vocabulary Edge Inference
• CATransformers: Carbon Aware Transformers Through Joint Model-Hardware Optimization
• Foreseeing the Invisible: Amodal Reconstruction of Leaf Fossil Images
• COSMO: Consensus-Driven Shift Modulation for Source-Free Domain Adaptation
• HAECcity: Open-Vocabulary Scene Understanding of City-Scale Point Clouds with Superpoint Graph Clustering
• Prior-SG: Task and Prior Driven Region Segmentation for Scene Graphs in Arbitrarily-Structured Environments
• Benchmarking Image Embeddings for E-Commerce: Evaluating Off-the Shelf Foundation Models, Fine-Tuning Strategies and Practical Trade-offs

Discussions

• MobileCLIP: Fast Image-Text Models Through Multi-Modal Reinforced Training [hn, 1 points, 0 comments]

Related

• MobileCLIP2: Improving Multi-Modal Reinforced Training
• CLIP with Quality Captions: A Strong Pretraining for Vision Tasks
• Getting More Juice Out of Your Data: Hard Pair Refinement Enhances Visual-Language Models Without Extra Data
• Simplifying CLIP: Unleashing the Power of Large-Scale Models on Consumer-level Computers
• FastCLIP: A Suite of Optimization Techniques to Accelerate CLIP Training with Limited Resources