Gradient Boosted Mixed Models: Flexible Estimation of Mean and Variance Components for Clustered Data

2025/10/31 by Mitchell L. Prevett, Prevett, Mitchell L., Francis K. C. Hui +7
Mathematics · #Advanced Causal Inference Techniques #Statistical Methods and Bayesian Inference #Statistical Methods and Inference

paper · pdf · doi:10.48550/arxiv.2511.00217

Abstract

We introduce Gradient Boosted Mixed Models (GBMixed), a framework which extends boosting to clustered data by jointly modeling the mean and variance components in a linear mixed model via likelihood-based gradients. GBMixed estimates a nonparametric fixed effects function characterizing the overall mean of the response, while also allowing the random effects covariance matrix along with the residual variance to depend on covariates in a flexible manner. We demonstrate how GBMixed facilitates covariate-dependent random effect predictions, and subsequently point predictions and prediction intervals for individual treatment effects, that can adapt between population-level and cluster-level information. Simulations and applications to two real-world datasets demonstrate that GBMixed can accurately recover complex nonlinear fixed effect functions and covariate-dependent covariances in a linear mixed model, while also improving point and probabilistic predictive performance compared with several existing approaches such as parametric linear mixed models, Natural Gradient Boosting, and Gaussian Process Boosting.

Citations

• Robust Linear Mixed Models using Hierarchical Gamma-Divergence
• Conformal Inference of Counterfactuals and Individual Treatment Effects
• Gaussian Process Boosting
• NGBoost: Natural Gradient Boosting for Probabilistic Prediction
• Random forests for high-dimensional longitudinal data
• Bayesian additive regression trees and the General BART model
• Bayesian Deep Net GLM and GLMM
• Estimation and Inference of Heterogeneous Treatment Effects using Random Forests
• Estimation and Inference of Heterogeneous Treatment Effects using Random Forests
• Model Selection in Linear Mixed Models
• A working guide to boosted regression trees
• Strictly Proper Scoring Rules, Prediction, and Estimation
• Stochastic gradient boosting
• Greedy function approximation: A gradient boosting machine.
• Nonlinear Mixed Effects Models for Repeated Measures Data
• Estimating causal effects of treatments in randomized and nonrandomized studies.

Related

• Gradient Boosting for Linear Mixed Models
• Addressing cluster-constant covariates in mixed effects models via likelihood-based boosting techniques
• Gradient Boosting for Hierarchical Data in Small Area Estimation
• Exact Gradient Evaluation for Adaptive Quadrature Approximate Marginal Likelihood in Mixed Models for Grouped Data
• Robust Linear Mixed Models using Hierarchical Gamma-Divergence