LookSharp: Attention Entropy Minimization for Test-Time Adaptation

2025/11/24 by Shelhamer, Evan, Mali, Yash
Computer Science · #Domain Adaptation and Few-Shot Learning #Generative Adversarial Networks and Image Synthesis #Advanced Neural Network Applications

paper · doi:10.48550/arxiv.2511.18925

Abstract

Test-time adaptation (TTA) updates models during inference to reduce error on distribution shifts. While entropy minimization over the output distribution has proven effective as a TTA loss, we study using the intermediate distributions computed by transformers in the attention mechanism. We propose LookSharp, which minimizes the entropy of CLS-to-patch attention in the final layer as a novel TTA objective, encouraging the model to maintain focused attention on shifted data. We demonstrate that attention entropy minimization improves robustness on ImageNet-C. We also show that it is complementary to output entropy minimization and maintains performance on clean data.

Citations

• LookWhere? Efficient Visual Recognition by Learning Where to Look and What to See from Self-Supervision
• Vision Transformers Need Registers
• On Pitfalls of Test-Time Adaptation
• Towards Stable Test-Time Adaptation in Dynamic Wild World
• Efficient Test-Time Model Adaptation without Forgetting
• MEMO: Test Time Robustness via Adaptation and Augmentation
• Do Vision Transformers See Like Convolutional Neural Networks?
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
• ImageNet classification with deep convolutional neural networks

Related

• Attention Bootstrapping for Multi-Modal Test-Time Adaptation
• Towards Open-Set Test-Time Adaptation Utilizing the Wisdom of Crowds in Entropy Minimization
• REALM: Robust Entropy Adaptive Loss Minimization for Improved Single-Sample Test-Time Adaptation
• Ranked Entropy Minimization for Continual Test-Time Adaptation
• Tent: Fully Test-time Adaptation by Entropy Minimization