2023/12/28 by Peter C. Rigby, Seth Rogers, Rigby, Peter C. +13 · 1 citation
Computer Science · #Code (set theory) #Code review #Computer science #Data science #FOS: Computer and information sciences #Latency (audio) #Operating system #Set (abstract data type) #Software #Software Engineering (cs.SE) #Software Engineering Research #Software Engineering Techniques and Practices #Software Testing and Debugging Techniques #Software development #Software quality #Workload
paper · pdf · doi:10.48550/arxiv.2312.17169
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2023/12/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
The code review team at Meta is continuously improving the code review process. To evaluate the new recommenders, we conduct three A/B tests which are a type of randomized controlled experimental trial. Expt 1. We developed a new recommender based on features that had been successfully used in the literature and that could be calculated with low latency. In an A/B test on 82k diffs in Spring of 2022, we found that the new recommender was more accurate and had lower latency. Expt 2. Reviewer workload is not evenly distributed, our goal was to reduce the workload of top reviewers. We then ran an A/B test on 28k diff authors in Winter 2023 on a workload balanced recommender. Our A/B test led to mixed results. Expt 3. We suspected the bystander effect might be slowing down reviews of diffs where only a team was assigned. We conducted an A/B test on 12.5k authors in Spring 2023 and found a large decrease in the amount of time it took for diffs to be reviewed when a recommended individual was explicitly assigned. Our findings also suggest there can be a discrepancy between historical back-testing and A/B test experimental findings.