Invisible Shortcuts: Why Vision Encoders Know Your Camera

2026/08/05 by Vladan Stojnić, Ryan Ramos, Giorgos Kordopatis-Zilos +2
Computer Science · #cs.CV #cs.LG

paper · pdf

ECCV 2026

arxiv created 2026/08/05 · arxiv updated 2026/08/07

Abstract

Deep vision models exploit shortcuts, relying on cues that correlate with supervision signals. Prior work has focused on visible biases, such as object-background or texture correlations. We identify a different source of shortcut learning: invisible metadata traces embedded at the pixel level, for metadata such as image processing and photo acquisition. We hypothesize that large-scale semantic supervision, whether through categorical labels (ImageNet) or billion-scale captions (LAION), naturally induces metadata-semantics correlations during pretraining, leading models to convert low-level signals into predictive features. By introducing controlled metadata-semantics correlations, we show that stronger ones produce systematically higher sensitivity to metadata traces and larger performance degradation under metadata distribution shifts. We further explore mitigation strategies applied during and after pretraining that reduce sensitivity not only to targeted metadata but also to unseen ones, without sacrificing performance on downstream tasks. Metadata sensitivity also has a positive side: it partly explains the strong generated-image detection ability of some encoders, while its mitigation can improve out-of-distribution generalization. Code: https://github.com/ryan-caesar-ramos/visual-encoder-traces

Citations

• Processing and acquisition traces in visual encoders: What does CLIP know about your camera?
• Do ImageNet-trained models learn shortcuts? The impact of frequency shortcuts on generalization
• Shortcut Learning Susceptibility in Vision Classifiers
• MAVias: Mitigate any Visual Bias
• RaVL: Discovering and Mitigating Spurious Correlations in Fine-Tuned Vision-Language Models
• DeCLIP: Decoding CLIP representations for deepfake localization
• The Group Robustness is in the Details: Revisiting Finetuning under Spurious Correlations
• Fake or JPEG? Revealing Common Biases in Generated Image Detection Datasets
• A Sober Look at the Robustness of CLIPs to Spurious Features
• A Decade's Battle on Dataset Bias: Are We There Yet?
• The Neglected Tails in Vision-Language Models
• Scaling Laws of Synthetic Images for Model Training ... for Now
• FACTS: First Amplify Correlations and Then Slice to Discover Bias
• What do neural networks learn in image classification? A frequency shortcut perspective
• What Makes ImageNet Look Unlike LAION
• Discover and Cure: Concept-aware Mitigation of Spurious Correlation
• FLAC: Fairness-Aware Representation Learning by Suppressing Attribute-Class Associations
• DINOv2: Learning Robust Visual Features without Supervision
• Mitigating Spurious Correlations in Multi-modal Models during Fine-tuning
• Sigmoid Loss for Language Image Pre-Training
• Towards Universal Fake Image Detectors that Generalize Across Generative Models
• Fake it till you make it: Learning transferable representations from synthetic ImageNet clones
• FlexiViT: One Model for All Patch Sizes
• A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Others
• Spurious Features Everywhere -- Large-Scale Detection of Harmful Spurious Features in ImageNet
• Avoiding spurious correlations via logit correction
• On Feature Learning in the Presence of Spurious Correlations
• DE-FAKE: Detection and Attribution of Fake Images Generated by Text-to-Image Generation Models
• Nuisances via Negativa: Adjusting for Spurious Correlations via Data Augmentation
• MaskTune: Mitigating Spurious Correlations by Forcing to Explore
• Learning Debiased Classifier with Biased Committee
• Gender Artifacts in Visual Datasets
• Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations
• Spread Spurious Attribute: Improving Worst-group Accuracy with Spurious Attribute Estimation
• BERTopic: Neural topic modeling with a class-based TF-IDF procedure
• Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlations
• A ConvNet for the 2020s
• A ConvNet for the 2020s
• Improving Out-of-Distribution Robustness via Selective Augmentation
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• Salient ImageNet: How to discover spurious features in Deep Learning?
• ResNet strikes back: An improved training procedure in timm
• Just Train Twice: Improving Group Robustness without Training Group Information
• Fairness via Representation Neutralization
• Emerging Properties in Self-Supervised Vision Transformers
• Learning Transferable Visual Models From Natural Language Supervision
• Understanding the Failure Modes of Out-of-Distribution Generalization
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Learning from Failure: Training Debiased Classifier from Biased Classifier
• The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
• The Pitfalls of Simplicity Bias in Neural Networks
• An Investigation of Why Overparameterization Exacerbates Spurious Correlations
• Shortcut learning in deep neural networks
• Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
• Learning De-biased Representations with Biased Representations
• RandAugment: Practical automated data augmentation with a reduced search space
• Natural Adversarial Examples
• On the "steerability" of generative adversarial networks
• Invariant Risk Minimization
• Learning Robust Global Representations by Penalizing Local Predictive Power
• REPAIR: Removing Representation Bias by Dataset Resampling
• Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
• Do ImageNet Classifiers Generalize to ImageNet?
• Learning Not to Learn: Training Deep Neural Networks with Biased Data
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• Learning Robust Representations by Projecting Superficial Statistics Out
• Turning a Blind Eye: Explicit Removal of Biases and Variation from Deep Neural Network Embeddings
• Recognition in Terra Incognita
• Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study
• UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
• mixup: Beyond Empirical Risk Minimization
• Deep Residual Learning for Image Recognition
• Unsupervised Visual Representation Learning by Context Prediction
• Unsupervised Visual Representation Learning by Context Prediction
• ImageNet Large Scale Visual Recognition Challenge
• ImageNet Large Scale Visual Recognition Challenge
• Visualizing and Understanding Convolutional Networks
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows