2021/11/03 by Jialiang Mao, Mao, Jialiang, Iavor Bojinov +1 · 2 citations
Computer Science · Engineering · Mathematics · #Advanced Causal Inference Techniques #Algorithm #Applications (stat.AP) #Business #Computer science #Engineering #FOS: Computer and information sciences #Industrial engineering #Iterative and incremental development #Iterative method #Machine learning #Marketing #Mathematics #Methodology (stat.ME) #New product development #Open Source Software Innovations #Process (computing) #Product (mathematics) #Sequence (biology) #Software engineering #Statistical Methods in Clinical Trials #Value (mathematics) #stat.AP #stat.ME
paper · pdf · doi:10.48550/arxiv.2111.02334
published in arXiv (Cornell University) (Cornell University)
arxiv created 2021/11/03 · openalex publication_date 2021/11/03 · arxiv updated 2021/11/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Over the past decade, most technology companies and a growing number of conventional firms have adopted online experimentation (or A/B testing) into their product development process. Initially, A/B testing was deployed as a static procedure in which an experiment was conducted by randomly splitting half of the users to see the control-the standard offering-and the other half the treatment-the new version. The results were then used to augment decision-making around which version to release widely. More recently, as experimentation has matured, firms have developed a more dynamic approach to experimentation in which a new version (the treatment) is gradually released to a growing number of units through a sequence of randomized experiments, known as iterations. In this paper, we develop a theoretical framework to quantify the value brought on by such dynamic or iterative experimentation. We apply our framework to seven months of LinkedIn experiments and show that iterative experimentation led to an additional 20% improvement in one of the firm's primary metrics.