GradPCA: Leveraging NTK Alignment for Reliable Out-of-Distribution Detection

2025/05/21 by Mariia Seleznova, Seleznova, Mariia, Hung-Hsu Chou +5
Computer Science · Engineering · #Anomaly Detection Techniques and Applications #Computer Vision and Pattern Recognition (cs.CV) #Distributed Sensor Networks and Detection Algorithms #FOS: Computer and information sciences #Fault Detection and Control Systems #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2505.16017

openalex publication_date 2025/05/21 · openalex created_date 2025/10/19 · openalex updated_date 2026/07/28

Abstract

We introduce GradPCA, an Out-of-Distribution (OOD) detection method that exploits the low-rank structure of neural network gradients induced by Neural Tangent Kernel (NTK) alignment. GradPCA applies Principal Component Analysis (PCA) to gradient class-means, achieving more consistent performance than existing methods across standard image classification benchmarks. We provide a theoretical perspective on spectral OOD detection in neural networks to support GradPCA, highlighting feature-space properties that enable effective detection and naturally emerge from NTK alignment. Our analysis further reveals that feature quality -- particularly the use of pretrained versus non-pretrained representations -- plays a crucial role in determining which detectors will succeed. Extensive experiments validate the strong performance of GradPCA, and our theoretical framework offers guidance for designing more principled spectral OOD detectors.

Citations

• SeTAR: Out-of-Distribution Detection with Selective Low-Rank Approximation
• Pursuing Feature Separation based on Neural Collapse for Out-of-Distribution Detection
• Gradient-Regularized Out-of-Distribution Detection
• Kernel PCA for Out-of-Distribution Detection
• GAIA: Delving into Gradient-based Attribution Abnormality for Out-of-distribution Detection
• Detecting Out-of-Distribution Through the Lens of Neural Collapse
• Low-Dimensional Gradient Helps Out-of-Distribution Detection
• Neural Collapse in the Intermediate Hidden Layers of Classification Neural Networks
• GradOrth: A Simple yet Efficient Out-of-Distribution Detection with Orthogonal Projection of Gradients
• OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detection
• Gradient-Based Adversarial and Out-of-Distribution Detection
• Out-of-Distribution Detection with Deep Nearest Neighbors
• VOS: Learning What You Don't Know by Virtual Outlier Synthesis
• Gradient-based Novelty Detection Boosted by Self-supervised Binary Classification
• ReAct: Out-of-distribution Detection With Rectified Activations
• DICE: Leveraging Sparsification for Out-of-Distribution Detection
• A Unified Survey on Anomaly, Novelty, Open-Set, and Out-of-Distribution Detection: Solutions and Future Challenges
• Generalized Out-of-Distribution Detection: A Survey
• Generalized Out-of-Distribution Detection: A Survey
• On the Importance of Gradients for Detecting Distributional Shifts in the Wild
• No True State-of-the-Art? OOD Detection Methods are Inconsistent across Datasets
• Knowledge distillation: A good teacher is patient and consistent
• A Theory of Neural Tangent Kernel Alignment and Its Influence on Training
• MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Space
• SSD: A Unified Framework for Self-Supervised Outlier Detection
• Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Label-Aware Neural Tangent Kernel: Toward Better Generalization and Local Elasticity
• Energy-based Out-of-distribution Detection
• Gradients as a Measure of Uncertainty in Neural Networks
• Backpropagated Gradient Representations for Anomaly Detection
• Big Transfer (BiT): General Visual Representation Learning
• The Local Elasticity of Neural Networks
• Likelihood Ratios for Out-of-Distribution Detection
• Do ImageNet Classifiers Generalize to ImageNet?
• A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
• Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods
• Adversarial examples in the physical world
• Concrete Problems in AI Safety
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• TurkerGaze: Crowdsourcing Saliency with Webcam based Eye Tracking
• Explaining and Harnessing Adversarial Examples
• Deep Neural Networks are Easily Fooled: High Confidence Predictions for\n Unrecognizable Images
• A useful variant of the Davis--Kahan theorem for statisticians
• Describing Textures in the Wild
• The Rotation of Eigenvectors by a Perturbation. III
• Analysis of a complex of statistical variables into principal components.

Related

• Kernel PCA for Out-of-Distribution Detection: Non-Linear Kernel Selections and Approximations
• Kernel PCA for Out-of-Distribution Detection
• Low-Dimensional Gradient Helps Out-of-Distribution Detection
• GROOD: GRadient-Aware Out-of-Distribution Detection
• WeiPer: OOD Detection using Weight Perturbations of Class Projections