Hierarchical Indian Buffet Neural Networks for Bayesian Continual Learning

2019/12/04 by Samuel Kessler, Kessler, Samuel, Vu Nguyen +5 · 2 citations
Computer Science · Mathematics · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1912.02290

22 pages, 19 figures including references and appendix. Accepted at UAI 2021

openalex publication_date 2019/12/04 · openalex created_date 2020/03/06 · arxiv created 2021/08/02 · arxiv updated 2021/08/03 · openalex updated_date 2026/07/28

Abstract

We place an Indian Buffet process (IBP) prior over the structure of a Bayesian Neural Network (BNN), thus allowing the complexity of the BNN to increase and decrease automatically. We further extend this model such that the prior on the structure of each hidden layer is shared globally across all layers, using a Hierarchical-IBP (H-IBP). We apply this model to the problem of resource allocation in Continual Learning (CL) where new tasks occur and the network requires extra resources. Our model uses online variational inference with reparameterisation of the Bernoulli and Beta distributions, which constitute the IBP and H-IBP priors. As we automatically learn the number of weights in each layer of the BNN, overfitting and underfitting problems are largely overcome. We show empirically that our approach offers a competitive edge over existing methods in CL.

Citations

• Structured Stochastic Variational Inference
• Variational Dropout and the Local Reparameterization Trick
• Auto-Encoding Variational Bayes
• Matplotlib: A 2D Graphics Environment
• Variational Dropout Sparsifies Deep Neural Networks
• Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting
• Task Agnostic Continual Learning Using Online Variational Bayes
• A Unifying Bayesian View of Continual Learning
• Three scenarios for continual learning
• Continual Unsupervised Representation Learning
• A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning

Cited by

• The Bayesian Approach to Continual Learning: An Overview
• DrNAS: Dirichlet Neural Architecture Search

Related

• Continual Learning using a Bayesian Nonparametric Dictionary of Weight Factors
• Bayesian Structure Adaptation for Continual Learning
• Hierarchically Structured Task-Agnostic Continual Learning
• Robust Continual Learning through a Comprehensively Progressive Bayesian Neural Network
• Continual Learning Using Bayesian Neural Networks