LEAML: Label-Efficient Adaptation to Out-of-Distribution Visual Tasks for Multimodal Large Language Models

2025/10/03 by Ci-Siang Lin, Lin, Ci-Siang, Min-Hung Chen +4
Computer Science · #Multimodal Machine Learning Applications #Domain Adaptation and Few-Shot Learning #Generative Adversarial Networks and Image Synthesis

paper · pdf · doi:10.48550/arxiv.2510.03232

Abstract

Multimodal Large Language Models (MLLMs) have achieved strong performance on general visual benchmarks but struggle with out-of-distribution (OOD) tasks in specialized domains such as medical imaging, where labeled data is limited and expensive. We introduce LEAML, a label-efficient adaptation framework that leverages both scarce labeled VQA samples and abundant unlabeled images. Our approach generates domain-relevant pseudo question-answer pairs for unlabeled data using a QA generator regularized by caption distillation. Importantly, we selectively update only those neurons most relevant to question-answering, enabling the QA Generator to efficiently acquire domain-specific knowledge during distillation. Experiments on gastrointestinal endoscopy and sports VQA demonstrate that LEAML consistently outperforms standard fine-tuning under minimal supervision, highlighting the effectiveness of our proposed LEAML framework.

Citations

• Locate-then-Merge: Neuron-Level Parameter Fusion for Mitigating Catastrophic Forgetting in Multimodal LLMs
• Qwen2.5-VL Technical Report
• NVILA: Efficient Frontier Visual Language Models
• VideoICL: Confidence-based Iterative In-context Learning for Out-of-Distribution Video Understanding
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoning
• Qwen Technical Report
• CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say No
• BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs
• LLaMA: Open and Efficient Foundation Language Models
• Delving into Out-of-Distribution Detection with Vision-Language Representations
• Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space
• Knowledge Neurons in Pretrained Transformers
• Transformer Feed-Forward Layers Are Key-Value Memories
• Scaling Laws for Neural Language Models
• Self-training with Noisy Student improves ImageNet classification
• Self-Training With Noisy Student Improves ImageNet Classification
• Adapting Large Language Models to Domains via Reading Comprehension

Related

• Efficient Vocabulary-Free Fine-Grained Visual Recognition in the Age of Multimodal LLMs
• Data-free Multi-label Image Recognition via LLM-powered Prompt Tuning
• Less is More: A Simple yet Effective Token Reduction Method for Efficient Multi-modal LLMs
• A Training-Free, Task-Agnostic Framework for Enhancing MLLM Performance on High-Resolution Images
• WangLab at MEDIQA-M3G 2024: Multimodal Medical Answer Generation using Large Language Models