2026/04/24 by DIANA JORDAN, TRENT OLLERENSHAW, ANDREW TREXLER · 1 voice
Mathematics · Social Sciences · Decision Sciences · #Advanced Causal Inference Techniques #Survey Methodology and Nonresponse #Psychometric Methodologies and Testing
paper · pdf · doi:10.1017/s0003055426101671
We re-examine recent influential claims that repeated measure experimental designs offer large precision gains without biasing treatment effect estimates in survey research. We test these claims by experimentally varying the design of six classic political science experiments across three distinct large samples of U.S. adults (total N=13,163 ). In contrast to prior evidence, we observe consistent attenuation of treatment effects in repeated measure designs. However, we show in simulations that this average design effect is small enough, and the precision gains large enough, that we recommend repeated measure designs for broad application—though (large-N) post-only designs may be preferable when research priorities include estimating the precise magnitude of a treatment effect. We additionally explore how several design considerations affect the bias-precision trade-off, such as within-subject versus between-groups designs, the relative separation of repeated measures within single surveys, and differences in respondent characteristics across sample types.