Grounding Representation Similarity with Statistical Testing

2021/08/03 by Frances Ding, Jean-Stanislas Denain, Ding, Frances +3 · 1 citation
Computer Science · Mathematics · Psychology · #Artificial intelligence #Canonical correlation #Computer science #Data mining #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Initialization #Kernel (algebra) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine learning #Mathematics #Neural Networks and Applications #Psychology #Representation (politics) #Robustness (evolution) #Set (abstract data type) #Similarity (geometry) #Strengths and weaknesses #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2108.01661

Accepted at NeurIPS 2021. 10 pages, 3 figures

openalex publication_date 2021/08/03 · arxiv created 2021/11/03 · arxiv updated 2021/11/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

To understand neural network behavior, recent works quantitatively compare different networks' learned representations using canonical correlation analysis (CCA), centered kernel alignment (CKA), and other dissimilarity measures. Unfortunately, these widely used measures often disagree on fundamental observations, such as whether deep networks differing only in random initialization learn similar representations. These disagreements raise the question: which, if any, of these dissimilarity measures should we believe? We provide a framework to ground this question through a concrete test: measures should have sensitivity to changes that affect functional behavior, and specificity against changes that do not. We quantify this through a variety of functional behaviors including probing accuracy and robustness to distribution shift, and examine changes such as varying random initialization and deleting principal components. We find that current metrics exhibit different weaknesses, note that a classical baseline performs surprisingly well, and highlight settings where all metrics appear to fail, thus providing a challenge set for further improvement.

Citations

• A Generalized Solution of the Orthogonal Procrustes Problem
• Convergent Learning: Do different neural networks learn the same representations?
• What do Neural Machine Translation Models Learn about Morphology?
• SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
• Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning
• Sanity Checks for Saliency Maps
• Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
• Similarity of Neural Network Representations Revisited
• Dissecting Contextual Word Embeddings: Architecture and Representation
• Weight Poisoning Attacks on Pre-trained Models
• Transferred Discrepancy: Quantifying the Difference Between Representations
• Towards falsifiable interpretability research
• Underspecification Presents Challenges for Credibility in Modern Machine Learning
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Decoding Alignment without Encoding Alignment: A critique of similarity analysis in neuroscience

Related

• Similarity of Neural Network Representations Revisited
• Reliability of CKA as a Similarity Measure in Deep Learning
• Deconfounded Representation Similarity for Comparison of Neural Networks
• Evaluating Representational Similarity Measures from the Lens of Functional Correspondence
• Insights on representational similarity in neural networks with canonical correlation