Progressive Conditioned Scale-Shift Recalibration of Self-Attention for Online Test-time Adaptation

2025/12/14 by Yushun Tang, Tang, Yushun, Ziqiong Liu +7
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Face recognition and analysis #Generative Adversarial Networks and Image Synthesis

paper · pdf · doi:10.48550/arxiv.2512.12673

openalex publication_date 2025/12/14 · openalex created_date 2025/12/17 · openalex updated_date 2026/07/28

Abstract

Online test-time adaptation aims to dynamically adjust a network model in real-time based on sequential input samples during the inference stage. In this work, we find that, when applying a transformer network model to a new target domain, the Query, Key, and Value features of its self-attention module often change significantly from those in the source domain, leading to substantial performance degradation of the transformer model. To address this important issue, we propose to develop a new approach to progressively recalibrate the self-attention at each layer using a local linear transform parameterized by conditioned scale and shift factors. We consider the online model adaptation from the source domain to the target domain as a progressive domain shift separation process. At each transformer network layer, we learn a Domain Separation Network to extract the domain shift feature, which is used to predict the scale and shift parameters for self-attention recalibration using a Factor Generator Network. These two lightweight networks are adapted online during inference. Experimental results on benchmark datasets demonstrate that the proposed progressive conditioned scale-shift recalibration (PCSR) method is able to significantly improve the online test-time domain adaptation performance by a large margin of up to 3.9% in classification accuracy on the ImageNet-C dataset.

Citations

• Domain-Conditioned Transformer for Fully Test-time Adaptation
• Dual-Path Adversarial Lifting for Domain Shift Correction in Online Test-time Adaptation
• Learning Visual Conditioning Tokens to Correct Domain Shift for Fully Test-time Adaptation
• Efficient Test-Time Adaptation of Vision-Language Models
• Dynamic Adapter Meets Prompt Tuning: Parameter-Efficient Transfer Learning for Point Cloud Analysis
• VPA: Fully Test-Time Visual Prompt Adaptation
• Cross-Inferential Networks for Source-free Unsupervised Domain Adaptation
• Guiding Pseudo-labels with Uncertainty Estimation for Source-free Unsupervised Domain Adaptation
• Neuro-Modulated Hebbian Learning for Fully Test-Time Adaptation
• Towards Stable Test-Time Adaptation in Dynamic Wild World
• Scalable Diffusion Models with Transformers
• Variational Model Perturbation for Source-Free Domain Adaptation
• Visual Prompt Tuning for Test-time Domain Adaptation
• Visual Prompt Tuning for Generative Transfer Learning
• Test-Time Training with Masked Autoencoders
• Prior Knowledge Guided Unsupervised Domain Adaptation
• Back to the Source: Diffusion-Driven Test-Time Adaptation
• AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition
• MM-TTA: Multi-Modal Test-Time Adaptation for 3D Semantic Segmentation
• Contrastive Test-Time Adaptation
• Continual Test-Time Domain Adaptation
• Visual Prompt Tuning
• Conditional Prompt Learning for Vision-Language Models
• MEMO: Test Time Robustness via Adaptation and Augmentation
• Model Adaptation: Historical Contrastive Learning for Unsupervised Domain Adaptation without Source Data
• VisDA-2021 Competition Universal Domain Adaptation to Improve Performance on Out-of-Distribution Data
• BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
• LoRA: Low-Rank Adaptation of Large Language Models
• Robustness of Object Detectors in Degrading Weather Conditions
• Cross-domain Contrastive Learning for Unsupervised Domain Adaptation
• Parameter-Efficient Transfer Learning with Diff Pruning
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Adaptive Risk Minimization: Learning to Adapt to Domain Shift
• The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
• Self-Guided Adaptation: Progressive Representation Alignment for Domain Adaptive Object Detection
• Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation
• Natural Adversarial Examples
• Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
• Parameter-Efficient Transfer Learning for NLP
• A Style-Based Generator Architecture for Generative Adversarial Networks
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Attention Is All You Need
• Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey and Benchmark

Related

• Domain-Conditioned Transformer for Fully Test-time Adaptation
• Dual-Path Adversarial Lifting for Domain Shift Correction in Online Test-time Adaptation
• Learning Visual Conditioning Tokens to Correct Domain Shift for Fully Test-time Adaptation
• Rapid Network Adaptation: Learning to Adapt Neural Networks Using Test-Time Feedback
• Open-World Test-Time Adaptation with Hierarchical Feature Aggregation and Attention Affine