Benchmarking the Neural Linear Model for Regression

2019/12/18 by Sebastian W. Ober, Ober, Sebastian W., Carl Edward Rasmussen +1 · 5 citations
Computer Science · Mathematics · #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Machine Learning and Data Classification #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1912.08416

Advances in Approximate Bayesian Inference (AABI 2019)

arxiv created 2019/12/18 · openalex publication_date 2019/12/18 · arxiv updated 2019/12/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The neural linear model is a simple adaptive Bayesian linear regression method that has recently been used in a number of problems ranging from Bayesian optimization to reinforcement learning. Despite its apparent successes in these settings, to the best of our knowledge there has been no systematic exploration of its capabilities on simple regression tasks. In this work we characterize these on the UCI datasets, a popular benchmark for Bayesian regression models, as well as on the recently introduced UCI "gap" datasets, which are better tests of out-of-distribution uncertainty. We demonstrate that the neural linear model is a simple method that shows generally good performance on these tasks, but at the cost of requiring good hyperparameter tuning.

Citations

• Adam: A Method for Stochastic Optimization
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Scalable Bayesian Optimization Using Deep Neural Networks
• Variational Dropout and the Local Reparameterization Trick
• Practical Bayesian Optimization of Machine Learning Algorithms
• Weight Uncertainty in Neural Networks
• Deep Gaussian Processes for Regression using Approximate Expectation Propagation
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Bayesian Batch Active Learning as Sparse Subset Approximation

Cited by

• Approximating Full Conformal Prediction for Neural Network Regression with Gauss-Newton Influence
• Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks
• Enhancing Uncertainty Estimation and Interpretability via Bayesian Non-negative Decision Layer
• Spectral methods: crucial for machine learning, natural for quantum computers?
• Joint Model and Data Sparsification via the Marginal Likelihood

Related

• Bayesian Linear Regression on Deep Representations
• Benchmarking Bayesian neural networks and evaluation metrics for\n regression tasks
• A Novel Explanation Against Linear Neural Networks
• Bayesian Inference for Consistent Predictions in Overparameterized Nonlinear Regression
• Adaptive Bayesian Linear Regression for Automated Machine Learning