DynaPrompt: Dynamic Test-Time Prompt Tuning

2025/01/27 by Zehao Xiao, Shilin Yan, Xiao, Zehao +13 · 14 citations
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer science #Engineering and Test Systems #FOS: Computer and information sciences #Geology #Machine Learning (cs.LG) #Software Testing and Debugging Techniques #Test (biology) #VLSI and Analog Circuit Testing

paper · pdf · doi:10.48550/arxiv.2501.16404

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/01/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Test-time prompt tuning enhances zero-shot generalization of vision-language models but tends to ignore the relatedness among test samples during inference. Online test-time prompt tuning provides a simple way to leverage the information in previous test samples, albeit with the risk of prompt collapse due to error accumulation. To enhance test-time prompt tuning, we propose DynaPrompt, short for dynamic test-time prompt tuning, exploiting relevant data distribution information while reducing error accumulation. Built on an online prompt buffer, DynaPrompt adaptively selects and optimizes the relevant prompts for each test sample during tuning. Specifically, we introduce a dynamic prompt selection strategy based on two metrics: prediction entropy and probability difference. For unseen test data information, we develop dynamic prompt appending, which allows the buffer to append new prompts and delete the inactive ones. By doing so, the prompts are optimized to exploit beneficial information on specific test data, while alleviating error accumulation. Experiments on fourteen datasets demonstrate the effectiveness of dynamic test-time prompt tuning.

Cited by

• From Tokens to Photons: Test-Time Physical Prompting for Vison-Language Models
• Doubly Debiased Test-Time Prompt Tuning for Vision-Language Models
• Fly-CL: A Fly-Inspired Framework for Enhancing Efficient Decorrelation and Reduced Training Time in Pre-trained Model-based Continual Representation Learning
• Prompt-based Adaptation in Large-scale Vision Models: A Survey
• D-TPT: Dimensional Entropy Maximization for Calibrating Test-Time Prompt Tuning in Vision-Language Models
• FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Experts
• Backpropagation-Free Test-Time Adaptation via Probabilistic Gaussian Alignment
• Adapting Vision-Language Models Without Labels: A Comprehensive Survey
• The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models
• Generalizing vision-language models to novel domains: A comprehensive survey
• Progressive Scaling Visual Object Tracking
• Adaptive Classifier-Free Guidance via Dynamic Low-Confidence Masking
• CrossLMM: Decoupling Long Video Sequences from LMMs via Dual Cross-Attention Mechanisms
• UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokens
• On-the-Fly VLA Adaptation via Test-Time Reinforcement Learning

Related

• Doubly Debiased Test-Time Prompt Tuning for Vision-Language Models
• Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuning
• Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models
• Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalization
• C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersion