Uncertainty in Gradient Boosting via Ensembles

2020/06/18 by Andrey Malinin, Malinin, Andrey, Liudmila Prokhorenkova +3 · 6 citations
Computer Science · Decision Sciences · Mathematics · #Advanced Bandit Algorithms Research #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Applications #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2006.10562

openalex publication_date 2020/06/18 · openalex created_date 2020/06/25 · arxiv created 2021/04/02 · arxiv updated 2021/04/05 · openalex updated_date 2026/07/28

Abstract

For many practical, high-risk applications, it is essential to quantify uncertainty in a model's predictions to avoid costly mistakes. While predictive uncertainty is widely studied for neural networks, the topic seems to be under-explored for models based on gradient boosting. However, gradient boosting often achieves state-of-the-art results on tabular data. This work examines a probabilistic ensemble-based framework for deriving uncertainty estimates in the predictions of gradient boosting classification and regression models. We conducted experiments on a range of synthetic and real datasets and investigated the applicability of ensemble approaches to gradient boosting models that are themselves ensembles of decision trees. Our analysis shows that ensembles of gradient boosting models successfully detect anomalous inputs while having limited ability to improve the predicted total uncertainty. Importantly, we also propose a concept of a virtual ensemble to get the benefits of an ensemble via only one gradient boosting model, which significantly reduces complexity.

Citations

• Stochastic gradient boosting
• Boosted decision trees as an alternative to artificial neural networks for particle identification
• Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysis
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• Reverse KL-Divergence Training of Prior Networks: Improved Uncertainty and Adversarial Robustness
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

Cited by

• Murmur2Vec: A Hashing Based Solution For Embedding Generation Of COVID-19 Spike Sequences
• MorphBoost: Self-Organizing Universal Gradient Boosting with Adaptive Tree Morphing
• Statistical Inference for Gradient Boosting Regression
• TabResFlow: A Normalizing Spline Flow Model for Probabilistic Univariate Tabular Regression
• Unveiling the Role of Data Uncertainty in Tabular Deep Learning
• Learning Pole Structures of Hadronic States using Predictive Uncertainty Estimation

Related

• Ensembles for Uncertainty Estimation: Benefits of Prior Functions and Bootstrapping
• Gradients as a Measure of Uncertainty in Neural Networks
• Improving Data Quality with Training Dynamics of Gradient Boosting Decision Trees
• Confidence-Nets: A Step Towards better Prediction Intervals for regression Neural Networks on small datasets
• Explaining Predictions from Tree-based Boosting Ensembles