Predicting Neural Network Accuracy from Weights

2020/02/26 by Thomas Unterthiner, Unterthiner, Thomas, Daniel Keysers +7 · 1 voice · 22 citations
Computer Science · Mathematics · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2002.11448

Updated the Small CNN Zoo dataset: reduced the maximal learning rate and got rid of multiple bad runs. Replaced all the experiments with the new numbers. Added MLP. Fixed typo in the abstract (R2 score instead of Kendall's tau). Added several earlier related works to the literature overview

openalex publication_date 2020/02/26 · arxiv published 2020/02/26 · arxiv created 2021/04/09 · arxiv updated 2021/04/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We show experimentally that the accuracy of a trained neural network can be predicted surprisingly well by looking only at its weights, without evaluating it on input data. We motivate this task and introduce a formal setting for it. Even when using simple statistics of the weights, the predictors are able to rank neural networks by their performance with very high accuracy (R2 score more than 0.98). Furthermore, the predictors are able to rank networks trained on different, unobserved datasets and with different architectures. We release a collection of 120k convolutional neural networks trained on four different datasets to encourage further research in this area, with the goal of understanding network training and performance better.

Citations

• Learning Multiple Layers of Features from Tiny Images

Cited by

• Metanetworks as Regulatory Operators: Learning to Edit for Requirement Compliance
• Leveraging weights signals -- Predicting and improving generalizability in reinforcement learning
• Symmetry-Aware Graph Metanetwork Autoencoders: Model Merging through Parameter Canonicalization
• Learning Model Representations Using Publicly Available Model Hubs
• Symmetry-Aware Fully-Amortized Optimization with Scale Equivariant Graph Metanetworks
• SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass
• Deep Learning Through the Lens of Example Difficulty
• Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics
• Increasing Trustworthiness of Deep Neural Networks via Accuracy Monitoring
• NeuroVoxel-LM: Language-Aligned 3D Perception via Dynamic Voxelization and Meta-Embedding
• An open dataset of neural networks for hypernetwork research
• GradMetaNet: An Equivariant Architecture for Learning on Gradients
• Symmetry in Neural Network Parameter Spaces
• ICLR 2021 Challenge for Computational Geometry & Topology: Design and Results
• Neural Complexity Measures
• Weight-Space Linear Recurrent Neural Networks
• Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings
• Meta Pruning via Graph Metanetworks : A Universal Meta Learning Framework for Network Pruning
• A Model Zoo on Phase Transitions in Neural Networks
• Scaling LLaNA: Advancing NeRF-Language Understanding Through Large-Scale Training
• A Model Zoo of Vision Transformers
• The Impact of Model Zoo Size and Composition on Weight Space Learning

Discussions

• Predicting Neural Network Accuracy from Weights (2021) [hn, 1 points, 0 comments]

Related

• Test Sample Accuracy Scales with Training Sample Density in Neural Networks
• Enhancing Accuracy and Parameter-Efficiency of Neural Representations for Network Parameterization
• A comparative analysis of a neural network with calculated weights and a neural network with random generation of weights based on the training dataset size
• Weight Agnostic Neural Networks
• Bit-wise Training of Neural Network Weights