Spectral Analysis of Latent Representations

2019/07/19 by Justin Shenk, Mats L. Richter, Shenk, Justin +5 · 1 citation
Computer Science · #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.1907.08589

openalex publication_date 2019/07/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose a metric, Layer Saturation, defined as the proportion of the number of eigenvalues needed to explain 99% of the variance of the latent representations, for analyzing the learned representations of neural network layers. Saturation is based on spectral analysis and can be computed efficiently, making live analysis of the representations practical during training. We provide an outlook for future applications of this metric by outlining the behaviour of layer saturation in different neural architectures and problems. We further show that saturation is related to the generalization and predictive performance of neural networks.

Citations

• ImageNet classification with deep convolutional neural networks
• SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning\n Dynamics and Interpretability
• Visualizing and Understanding Convolutional Networks

Cited by

• Neural Coherence : Find higher performance to out-of-distribution tasks from few samples

Related

• Exploring the Properties and Evolution of Neural Network Eigenspaces during Training
• A Fine-Grained Spectral Perspective on Neural Networks
• A Quantitative Approach to Predicting Representational Learning and Performance in Neural Networks
• Neural Network Layer Matrix Decomposition reveals Latent Manifold Encoding and Memory Capacity
• Spectral Pruning: Compressing Deep Neural Networks via Spectral Analysis and its Generalization Error