Trade-Offs of Diagonal Fisher Information Matrix Estimators

2024/02/08 by Alexander Soen, Ke Sun, Soen, Alexander +1 · 1 citation
Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2402.05379

openalex publication_date 2024/02/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The Fisher information matrix can be used to characterize the local geometry of the parameter space of neural networks. It elucidates insightful theories and useful tools to understand and optimize neural networks. Given its high computational cost, practitioners often use random estimators and evaluate only the diagonal entries. We examine two popular estimators whose accuracy and sample complexity depend on their associated variances. We derive bounds of the variances and instantiate them in neural networks for regression and classification. We navigate trade-offs for both estimators based on analytical and numerical studies. We find that the variance quantities depend on the non-linearity wrt different parameter groups and should not be neglected when estimating the Fisher information.

Cited by

• Deterministic Bounds and Random Estimates of Metric Tensors on Neuromanifolds

Related

• On the Variance of the Fisher Information for Deep Learning
• Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach
• Fisher Information and Natural Gradient Learning of Random Deep Networks
• Pathological spectra of the Fisher information metric and its variants in deep neural networks
• Fishers for Free? Approximating the Fisher Information Matrix by Recycling the Squared Gradient Accumulator