Test-Time Canonicalization by Foundation Models for Robust Perception

2025/07/14 by Utkarsh Singhal, Singhal, Utkarsh, Ryan Feng +5 · 2 citations
Computer Science · Engineering · #Advanced Vision and Imaging #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Image Processing Techniques and Applications #Industrial Vision Systems and Defect Detection #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2507.10375

openalex publication_date 2025/07/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Perception in the real world requires robustness to diverse viewing conditions. Existing approaches often rely on specialized architectures or training with predefined data augmentations, limiting adaptability. Taking inspiration from mental rotation in human vision, we propose FOCAL, a test-time robustness framework that transforms the input into the most typical view. At inference time, FOCAL explores a set of transformed images and chooses the one with the highest likelihood under foundation model priors. This test-time optimization boosts robustness while requiring no retraining or architectural changes. Applied to models like CLIP and SAM, it significantly boosts robustness across a wide range of transformations, including 2D and 3D rotations, contrast and lighting shifts, and day-night changes. We also explore potential applications in active vision. By reframing invariance as a test-time optimization problem, FOCAL offers a general and scalable approach to robustness. Our code is available at: https://github.com/sutkarsh/focal.

Citations

• Trading Inference-Time Compute for Adversarial Robustness
• Structured 3D Latents for Scalable and Versatile 3D Generation
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
• Beyond Viewpoint: Robust 3D Object Recognition under Arbitrary Views through Joint Multi-Part Representation
• ImageNet3D: Towards General-Purpose Object-Level 3D Understanding
• Omniview-Tuning: Boosting Viewpoint Invariance of Vision-Language Pre-training Models
• A Generative Model of Symmetry Transformations
• Understanding the Detrimental Class-level Effects of Data Augmentation
• Equivariant Adaptation of Large Pretrained Models
• Learning to Transform for Generalizable Instance-wise Invariance
• 3D Gaussian Splatting for Real-Time Radiance Field Rendering
• 3D Gaussian Splatting for Real-Time Radiance Field Rendering
• Towards Viewpoint-Invariant Visual Recognition via Adversarial Training
• UrbanIR: Large-Scale Urban Scene Inverse Rendering from a Single Video
• Micrograph segmentations for DDEVD
• Segment Anything
• Objaverse: A Universe of Annotated 3D Objects
• Denoising diffusion models for out-of-distribution detection
• Equivariance with Learned Canonicalization Functions
• LAION-5B: An open large-scale dataset for training next generation image-text models
• Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP
• ViewFool: Evaluating the Robustness of Visual Recognition to Adversarial Viewpoints
• Diffusion models as plug-and-play priors
• Do Deep Networks Transfer Invariances Across Classes?
• Local Latent Space Bayesian Optimization over Structured Inputs
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• Grounding inductive biases in natural images:invariance stems from variations in data
• Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges
• Vector Neurons: A General Framework for SO(3)-Equivariant Networks
• Learning Transferable Visual Models From Natural Language Supervision
• MVTN: Multi-View Transformation Network for 3D Shape Recognition
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Energy-based Out-of-distribution Detection
• Category Level Object Pose Estimation via Neural Analysis-by-Synthesis
• What Should Not Be Contrastive in Contrastive Learning
• Sample-Efficient Optimization in the Latent Space of Deep Generative Models via Weighted Retraining
• A Multi-Hypothesis Approach to Color Constancy
• Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One
• LVIS: A Dataset for Large Vocabulary Instance Segmentation
• A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
• A Tutorial on Bayesian Optimization
• On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups
• Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
• Fast Fourier Color Constancy
• Compensating for Large In-Plane Rotations in Natural Images
• Los carnavales de La Bañeza: Aspectos antropológicos
• Deep Residual Learning for Image Recognition
• Spatial Transformer Networks
• Multi-view Convolutional Neural Networks for 3D Shape Recognition
• Microsoft COCO: Common Objects in Context

Cited by

• Adaptive Canonicalization with Application to Invariant Anisotropic Geometric Networks
• Agentic Aerial Cinematography: From Dialogue Cues to Cinematic Trajectories

Related

• On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?
• Noise is an Efficient Learner for Zero-Shot Vision-Language Models
• TRUST: Test-Time Refinement using Uncertainty-Guided SSM Traverses
• The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models
• NC-TTT: A Noise Contrastive Approach for Test-Time Training