vix.ing · top · new · best · stats

How Much Should We Trust Differences-In-Differences Estimates?

2004/02/01 by M. Bertrand, Esther Duflo, E. Duflo +2 · 10,698 citations
Economics, Econometrics and Finance · Mathematics · Social Sciences · #Advanced Causal Inference Techniques #Autocorrelation #Covariance #Econometrics #Economics #Estimator #Income, Poverty, and Inequality #Mathematics #Monte Carlo method #Parametric statistics #Population #Sample size determination #Series (stratigraphy) #Spatial and Panel Data Analysis #Standard deviation #Standard error #Statistics #Variance (accounting)

paper · open access · doi:10.1162/003355304772839588

published in The Quarterly Journal of Economics 119(1), 249-275 (Oxford University Press)

openalex publication_date 2004/02/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06

Abstract

Most papers that employ Differences-in-Differences estimation (DD) use many years of data and focus on serially correlated outcomes but ignore that the resulting standard errors are inconsistent. To illustrate the severity of this issue, we randomly generate placebo laws in state-level data on female wages from the Current Population Survey. For each law, we use OLS to compute the DD estimate of its “effect” as well as the standard error of this estimate. These conventional DD standard errors severely understate the standard deviation of the estimators: we find an “effect” significant at the 5 percent level for up to 45 percent of the placebo interventions. We use Monte Carlo simulations to investigate how well existing methods help solve this problem. Econometric corrections that place a specific parametric form on the time-series process do not perform well. Bootstrap (taking into account the autocorrelation of the data) works well when the number of states is large enough. Two corrections based on asymptotic approximation of the variance-covariance matrix work well for moderate numbers of states and one correction that collapses the time series information into a “pre”- and “post”-period and explicitly takes into account the effective sample size works well even for small numbers of states.

Citations

Cited by

Related