Supervised Learning Has a Geometric Blind Spot

2026/04/23 by Vishal Rajput · 1 voice
Computer Science · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Adversarial system #Bounded function #Empirical risk minimization #Explainable Artificial Intelligence (XAI) #Jacobian matrix and determinant #Metric (unit) #Monotonic function #Spurious relationship #Supervised learning #acm:68T05 #acm:68T45 #cs.AI #cs.CV #cs.LG #msc:68T05 #msc:68T45

paper · pdf · doi:10.48550/arxiv.2604.21395

35 pages. v2: JMLR-aligned revision of arXiv:2604.21395; Proposition 6 corrected to minimax (worst-case) anisotropy; title shortened to Supervised Learning Has a Geometric Blind Spot. Under submission at JMLR. Companion: arXiv:2605.22800

openalex publication_date 2026/04/23 · arxiv published 2026/04/23 · openalex created_date 2026/04/25 · openalex updated_date 2026/07/28 · arxiv created 2026/08/06 · arxiv updated 2026/08/07

Abstract

Ordinary supervised training minimises the task loss and then stops. It never pays for how far the representation moves when the input is nudged along directions that helped fit training labels---including directions that are nuisance at deployment. We call that leftover sensitivity the geometric blind spot of empirical risk minimisation. In a Gaussian linear model where the nuisance enters the label conditional and the decoder has finite Lipschitz constant, population MSE forces a floor on linearised representation drift. The same distinction predicts a failure mode of adversarial training: Jacobian magnitude can fall while clean class geometry worsens. We track that dissociation with a class-layout score and study isotropic encoder matching---penalising the squared distance between phi(x) and phi(x+delta) for Gaussian delta under a task-loss cap---when nuisance axes are unknown. On a Vision Transformer trained from scratch on CIFAR-10, projected gradient descent attains the smallest Jacobian Frobenius yet the worst clean layout score (1.353+/-0.020 over three seeds), above task-only training (1.093); isotropic matching attains the best (0.904). The drift floor is proved for the linear-Gaussian case; deep nets and cross-task orderings are protocol empirics. Design rule: report class-layout geometry beside the task score; prefer isotropic encoder matching when axes are unknown.

Citations

• The Intrinsic Dimension of Images and Its Impact on Learning
• Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth
• Robust Learning with Jacobian Regularization
• Intrinsic dimension of data representations in deep neural networks
• Adversarial Examples Are Not Bugs, They Are Features
• Similarity of Neural Network Representations Revisited
• Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• Robustness May Be at Odds with Accuracy
• Improving DNN Robustness to Adversarial Attacks using Jacobian Regularization
• Spectral Normalization for Generative Adversarial Networks
• Towards Deep Learning Models Resistant to Adversarial Attacks
• SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
• Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning
• Temporal Ensembling for Semi-Supervised Learning
• Exponential expressivity in deep neural networks through transient chaos
• The Matching Principle: A Geometric Theory of Loss Functions for Nuisance-Robust Representation Learning

Discussions

• We proved that every supervised model you've ever trained has a geometric blind spot; and adversarial training makes it worse, not better [lemmy, 0 points, 0 comments]

Related

• The Matching Principle: A Geometric Theory of Loss Functions for Nuisance-Robust Representation Learning
• Landscape and training regimes in deep learning
• Robustness via curvature regularization, and vice versa
• Training Instabilities Induce Flatness Bias in Gradient Descent
• Intriguing Properties of Robust Classification