2024/04/08 by Nan Zhou, Zhou, Nan, Jiaxin Chen +3 · 2 citations
Computer Science · Decision Sciences · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Personal Information Management and User Behavior #Virtual Reality Applications and Impacts
paper · pdf · doi:10.48550/arxiv.2404.05207
openalex publication_date 2024/04/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Recent progress has shown great potential of visual prompt tuning (VPT) when adapting pre-trained vision transformers to various downstream tasks. However, most existing solutions independently optimize prompts at each layer, thereby neglecting the usage of task-relevant information encoded in prompt tokens across layers. Additionally, existing prompt structures are prone to interference from task-irrelevant noise in input images, which can do harm to the sharing of task-relevant information. In this paper, we propose a novel VPT approach, iVPT. It innovatively incorporates a cross-layer dynamic connection (CDC) for input prompt tokens from adjacent layers, enabling effective sharing of task-relevant information. Furthermore, we design a dynamic aggregation (DA) module that facilitates selective sharing of information between layers. The combination of CDC and DA enhances the flexibility of the attention process within the VPT framework. Building upon these foundations, iVPT introduces an attentive reinforcement (AR) mechanism, by automatically identifying salient image tokens, which are further enhanced by prompt tokens in an additive manner. Extensive experiments on 24 image classification and semantic segmentation benchmarks clearly demonstrate the advantage of the proposed iVPT, compared to the state-of-the-art counterparts.