2022/12/21 by Nicholas Larsen, Larsen, Nicholas, Jonathan W. Stallrich +9 · 4 citations
Decision Sciences · Mathematics · #Applications (stat.AP) #FOS: Computer and information sciences #Optimal Experimental Design Methods #Statistical Methods in Clinical Trials
paper · pdf · doi:10.48550/arxiv.2212.11366
openalex publication_date 2022/12/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
The rise of internet-based services and products in the late 1990's brought about an unprecedented opportunity for online businesses to engage in large scale data-driven decision making. Over the past two decades, organizations such as Airbnb, Alibaba, Amazon, Baidu, Booking, Alphabet's Google, LinkedIn, Lyft, Meta's Facebook, Microsoft, Netflix, Twitter, Uber, and Yandex have invested tremendous resources in online controlled experiments (OCEs) to assess the impact of innovation on their customers and businesses. Running OCEs at scale has presented a host of challenges requiring solutions from many domains. In this paper we review challenges that require new statistical methodologies to address them. In particular, we discuss the practice and culture of online experimentation, as well as its statistics literature, placing the current methodologies within their relevant statistical lineages and providing illustrative examples of OCE applications. Our goal is to raise academic statisticians' awareness of these new research opportunities to increase collaboration between academia and the online industry.