Adaptive Experimental Design and Counterfactual Inference

2022/10/25 by Tanner Fiez, Sergio Gamez, Fiez, Tanner +7
Decision Sciences · Mathematics · #Advanced Statistical Process Monitoring #FOS: Computer and information sciences #Machine Learning (cs.LG) #Methodology (stat.ME) #Optimal Experimental Design Methods #Statistical Methods in Clinical Trials

paper · pdf · doi:10.48550/arxiv.2210.14369

openalex publication_date 2022/10/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Adaptive experimental design methods are increasingly being used in industry as a tool to boost testing throughput or reduce experimentation cost relative to traditional A/B/N testing methods. This paper shares lessons learned regarding the challenges and pitfalls of naively using adaptive experimentation systems in industrial settings where non-stationarity is prevalent, while also providing perspectives on the proper objectives and system specifications in these settings. We developed an adaptive experimental design framework for counterfactual inference based on these experiences, and tested it in a commercial environment.

Related

• AExGym: Benchmarks and Environments for Adaptive Experimentation
• Adaptive Experiments and a Rigorous Framework for Type I Error Verification and Computational Experiment Design
• Improving the Power of Economic Experiments Using Adaptive Designs
• Adaptive Experimentation at Scale: A Computational Framework for Flexible Batches
• Experimental Design For Causal Inference Through An Optimization Lens