Scalable Spike-and-Slab

2022/04/04 by Niloy Ranjan Biswas, Lester Mackey, Biswas, Niloy +2
Computer Science · Mathematics · #Bayesian Methods and Mixture Models #Computation (stat.CO) #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Markov Chains and Monte Carlo Methods #Methodology (stat.ME)

paper · pdf · doi:10.48550/arxiv.2204.01668

openalex publication_date 2022/04/04 · openalex created_date 2022/04/26 · openalex updated_date 2026/07/28

Abstract

Spike-and-slab priors are commonly used for Bayesian variable selection, due to their interpretability and favorable statistical properties. However, existing samplers for spike-and-slab posteriors incur prohibitive computational costs when the number of variables is large. In this article, we propose Scalable Spike-and-Slab (S³), a scalable Gibbs sampling implementation for high-dimensional Bayesian regression with the continuous spike-and-slab prior of George and McCulloch (1993). For a dataset with n observations and p covariates, S³ has order max\ n² p_t, np \ computational cost at iteration t where p_t never exceeds the number of covariates switching spike-and-slab states between iterations t and t-1 of the Markov chain. This improves upon the order n² p per-iteration cost of state-of-the-art implementations as, typically, p_t is substantially smaller than p. We apply S³ on synthetic and real-world datasets, demonstrating orders of magnitude speed-ups over existing exact samplers and significant gains in inferential quality over approximate samplers with comparable cost.

Related

• Fast and Scalable Spike and Slab Variable Selection in High-Dimensional\n Gaussian Processes
• Bayesian high-dimensional linear regression with generic spike-and-slab priors
• From Estimation to Sampling for Bayesian Linear Regression with Spike-and-Slab Prior
• Spike-and-Slab Posterior Sampling in High Dimensions
• Bayesian Bootstrap Spike-and-Slab LASSO