Zero-Direction Probing: A Linear-Algebraic Framework for Deep Analysis of Large-Language-Model Drift

2025/08/09 by Amit Pandey, Pandey, Amit
Computer Science · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Data Stream Mining Techniques #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · pdf · doi:10.48550/arxiv.2508.06776

openalex publication_date 2025/08/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present Zero-Direction Probing (ZDP), a theory-only framework for detecting model drift from null directions of transformer activations without task labels or output evaluations. Under assumptions A1--A6, we prove: (i) the Variance--Leak Theorem, (ii) Fisher Null-Conservation, (iii) a Rank--Leak bound for low-rank updates, and (iv) a logarithmic-regret guarantee for online null-space trackers. We derive a Spectral Null-Leakage (SNL) metric with non-asymptotic tail bounds and a concentration inequality, yielding a-priori thresholds for drift under a Gaussian null model. These results show that monitoring right/left null spaces of layer activations and their Fisher geometry provides concrete, testable guarantees on representational change.

Citations

• A Free Probabilistic Framework for Analyzing the Transformer-based Language Models
• Put the Space of LoRA Initialization to the Extreme to Preserve Pre-trained Knowledge
• On the Variance of the Fisher Information for Deep Learning
• Similarity of Neural Network Representations Revisited
• Insights on representational similarity in neural networks with canonical correlation
• Understanding intermediate layers using linear classifier probes

Related

• First-Order Predictable but Pairwise Fragile: Local Task Adaptation in Trained Transformers
• Momentum Point-Perplexity Mechanics in Large Language Models
• Eliciting Latent Predictions from Transformers with the Tuned Lens
• An Adjoint-Sensitivity Framework for Lost-in-the-Middle Phenomena in Causal Residual Transformers
• The Asymptotic Behavior of Attention in Transformers