Empirical Bayes Multistage Testing for Large-Scale Experiments

2022/09/13 by Hui Xu, Weinan Wang, Xu, Hui +1 · 1 citation
Mathematics · #FOS: Computer and information sciences #Methodology (stat.ME) #Statistical Methods and Bayesian Inference #Statistical Methods and Inference #Statistical Methods in Clinical Trials

paper · pdf · doi:10.48550/arxiv.2209.05788

openalex publication_date 2022/09/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/04

Abstract

Modern application of A/B tests is challenging due to its large scale in various dimensions, which demands flexibility to deal with multiple testing sequentially. The state-of-the-art practice first reduces the observed data stream to always-valid p-values, and then chooses a cut-off as in conventional multiple testing schemes. Here we propose an alternative method called AMSET (adaptive multistage empirical Bayes test) by incorporating historical data in decision-making to achieve efficiency gains while retaining marginal false discovery rate (mFDR) control that is immune to peeking. We also show that a fully data-driven estimation in AMSET performs robustly to various simulation and real data settings at a large mobile app social network company.

Cited by

• Learning Across Experiments and Time: Tackling Heterogeneity in A/B Testing

Related

• BONuS: Multiple multivariate testing with a data-adaptivetest statistic
• Sparse Recovery With Multiple Data Streams: A Sequential Adaptive Testing Approach
• Design of Bayesian A/B Tests Controlling False Discovery Rates and Power
• A Conformalized Empirical Bayes Method for Multiple Testing with Side Information
• A Bandit Approach to Multiple Testing with False Discovery Control