An Empirical Analysis of the Advantages of Finite- v.s. Infinite-Width Bayesian Neural Networks

2022/11/16 by Jiayu Yao, Yaniv Yacoby, Yao, Jiayu +7 · 1 citation
Computer Science · #Anomaly Detection Techniques and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2211.09184

openalex publication_date 2022/11/16 · openalex created_date 2022/11/26 · openalex updated_date 2026/07/28

Abstract

Comparing Bayesian neural networks (BNNs) with different widths is challenging because, as the width increases, multiple model properties change simultaneously, and, inference in the finite-width case is intractable. In this work, we empirically compare finite- and infinite-width BNNs, and provide quantitative and qualitative explanations for their performance difference. We find that when the model is mis-specified, increasing width can hurt BNN performance. In these cases, we provide evidence that finite-width BNNs generalize better partially due to the properties of their frequency spectrum that allows them to adapt under model mismatch.

Cited by

• A ZeNN architecture to avoid the Gaussian trap

Related

• Are wider nets better given the same number of parameters?
• Finite Versus Infinite Neural Networks: an Empirical Study
• Feature-Learning Networks Are Consistent Across Widths At Realistic Scales
• When narrower is better: the narrow width limit of Bayesian parallel branching neural networks
• Infinite Width Models That Work: Why Feature Learning Doesn't Matter as Much as You Think