Optimizing Active Learning in Vision-Language Models via Parameter-Efficient Uncertainty Calibration

2025/07/29 by Narayanan, Athmanarayanan Lakshmi, Machireddy, Amrutha, Krishnan, Ranganath
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2507.21521

Abstract

Active Learning (AL) has emerged as a powerful approach for minimizing labeling costs by selectively sampling the most informative data for neural network model development. Effective AL for large-scale vision-language models necessitates addressing challenges in uncertainty estimation and efficient sampling given the vast number of parameters involved. In this work, we introduce a novel parameter-efficient learning methodology that incorporates uncertainty calibration loss within the AL framework. We propose a differentiable loss function that promotes uncertainty calibration for effectively selecting fewer and most informative data samples for fine-tuning. Through extensive experiments across several datasets and vision backbones, we demonstrate that our solution can match and exceed the performance of complex feature-based sampling techniques while being computationally very efficient. Additionally, we investigate the efficacy of Prompt learning versus Low-rank adaptation (LoRA) in sample selection, providing a detailed comparative analysis of these methods in the context of efficient AL.

Citations

• Online Zero-Shot Classification with CLIP
• Zero-Shot Visual Reasoning by Vision-Language Models: Benchmarking and Analysis
• Parameter-Efficient Active Learning for Foundational models
• Low-Rank Few-Shot Adaptation of Vision-Language Models
• Overconfidence is Key: Verbalized Uncertainty Evaluation in Large Language and Vision-Language Models
• DeepSeek-VL: Towards Real-World Vision-Language Understanding
• A Survey on Hallucination in Large Vision-Language Models
• Revisiting Active Learning in the Era of Vision Foundation Models
• Querying Easily Flip-flopped Samples for Deep Active Learning
• A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models
• Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning
• Parameter Efficient Fine-tuning via Cross Block Orchestration for Segment Anything Model
• Active Prompt Learning in Vision Language Models
• ActiveDC: Distribution Calibration for Active Finetuning
• LM-Polygraph: Uncertainty Estimation for Language Models
• Meta-Adapter: An Online Few-shot Learner for Vision-Language Model
• PELA: Learning Parameter-Efficient Models with Low-Rank Approximation
• A Survey of Hallucination in Large Foundation Models
• Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs
• Active Finetuning: Exploiting Annotation Budget in the Pretraining-Finetuning Paradigm
• MaPLe: Multi-modal Prompt Learning
• Entropy-based Active Learning for Object Detection with Progressive Diversity Constraint
• Conditional Prompt Learning for Vision-Language Models
• Active Learning Over Multiple Domains in Natural Language Tasks
• FLAVA: A Foundational Language And Vision Alignment Model
• Robust Contrastive Active Learning with Feature-guided Query Strategies
• Learning to Prompt for Vision-Language Models
• Soft Calibration Objectives for Neural Networks
• LoRA: Low-Rank Adaptation of Large Language Models
• The Power of Scale for Parameter-Efficient Prompt Tuning
• Active Learning for Deep Object Detection via Probabilistic Modeling
• Learning Transferable Visual Models From Natural Language Supervision
• Zero-Shot Text-to-Image Generation
• Deep Deterministic Uncertainty: A Simple Baseline
• Improving model calibration with accuracy versus uncertainty optimization
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• A Survey of Deep Active Learning
• Learning Loss for Active Learning
• EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification
• On Calibration of Modern Neural Networks
• Deep Residual Learning for Image Recognition
• Describing Textures in the Wild
• Bayesian Active Learning for Classification and Preference Learning
• Enhancing Trust in Large Language Models via Uncertainty-Calibrated Fine-Tuning

Related

• Active Learning for Vision-Language Models
• Deep Active Learning with Noise Stability
• Post-hoc Probabilistic Vision-Language Models
• Parameter-Efficient Active Learning for Foundational models
• Calibrated Uncertainty Sampling for Active Learning