Generalized Shape Metrics on Neural Representations

2021/10/27 by Alex H. Williams, Erin Kunz, Williams, Alex H. +6 · 1 voice · 23 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · Mathematics · #Cell Image Analysis Techniques #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Morphological variations and asymmetry #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2110.14739

26 pages, 7 figures, NeurIPS 2021

openalex publication_date 2021/10/27 · arxiv created 2022/01/13 · arxiv updated 2022/01/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Understanding the operation of biological and artificial networks remains a difficult and important challenge. To identify general principles, researchers are increasingly interested in surveying large collections of networks that are trained on, or biologically adapted to, similar tasks. A standardized set of analysis tools is now needed to identify how network-level covariates -- such as architecture, anatomical brain region, and model organism -- impact neural representations (hidden layer activations). Here, we provide a rigorous foundation for these analyses by defining a broad family of metric spaces that quantify representational dissimilarity. Using this framework we modify existing representational similarity measures based on canonical correlation analysis to satisfy the triangle inequality, formulate a novel metric that respects the inductive biases in convolutional layers, and identify approximate Euclidean embeddings that enable network representations to be incorporated into essentially any off-the-shelf machine learning method. We demonstrate these methods on large-scale datasets from biology (Allen Institute Brain Observatory) and deep learning (NAS-Bench-101). In doing so, we identify relationships between neural representations that are interpretable in terms of anatomical features and model performance.

Cited by

• Revisiting the Platonic Representation Hypothesis: An Aristotelian View
• You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations
• Superposition disentanglement of neural representations reveals hidden alignment
• Manifold Approximation leads to Robust Kernel Alignment
• Revisiting the Relation Between Robustness and Universality
• Model-brain comparison using inter-animal transforms
• Representational Alignment Across Model Layers and Brain Regions with Hierarchical Optimal Transport
• Uncovering the Computational Ingredients of Human-Like Representations in LLMs
• Knowledge distillation through geometry-aware representational alignment
• Discovering alternative solutions beyond the simplicity bias in recurrent neural networks
• Seeing Through Words, Speaking Through Pixels: Deep Representational Alignment Between Vision and Language Models
• Train Small, Deploy Large: Zero-Shot GNN Transfer Through Geometric Renormalization
• Transporting Task Vectors across Different Architectures without Training
• Measuring the Measures: Discriminative Capacity of Representational Similarity Metrics Across Model Families
• Algorithm Development in Neural Networks: Insights from the Streaming Parity Task
• The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions
• Can Biologically Plausible Temporal Credit Assignment Rules Match BPTT for Neural Similarity? E-prop as an Example
• Pruning Everything, Everywhere, All at Once
• Quantifying task-relevant representational similarity using decision variable correlation
• Koopman Autoencoders Learn Neural Representation Dynamics
• Functional divergence between the two cerebral hemispheres contributes to human fluid intelligence
• Scalable and Interpretable Representation Alignment with Ordinal Similarity
• Summary statistics of learning link changing neural representations to behavior

Discussions

• I don't think there's a "best" metric but Procrustes is my personal favorite because it has very explicit geometric intuition and generalizes very naturally to other metrics. It also is less sensitive [bsky, 1 points, 0 comments]

Related

• Integrated representational signatures strengthen specificity in brains and models
• The topology and geometry of neural representations
• Evaluating Representational Similarity Measures from the Lens of Functional Correspondence
• Measuring the Measures: Discriminative Capacity of Representational Similarity Metrics Across Model Families
• Estimating Shape Distances on Neural Representations with Limited Samples