Leniency Designs: An Operator's Manual

2025/11/05 by Paul Goldsmith-Pinkham, Peter Hull, Goldsmith-Pinkham, Paul +3 · 2 voices · 1 citation
Business, Management and Accounting · Economics, Econometrics and Finance · Mathematics · #Advanced Causal Inference Techniques #Estimator #Innovation Policy and R&D #Instrumental variable #Intellectual Property and Patents #Jackknife resampling #Key (lock) #Omitted-variable bias #Value (mathematics)

paper · pdf · open access · doi:10.48550/arxiv.2511.03572

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/11/05 · openalex created_date 2025/11/07 · openalex updated_date 2026/07/28

Abstract

We develop a step-by-step guide to leniency (a.k.a. judge or examiner instrument) designs, drawing on recent econometric literatures. The unbiased jackknife instrumental variables estimator (UJIVE) is purpose-built for leveraging exogenous leniency variation, avoiding subtle biases even in the presence of many decision-makers or controls. We show how UJIVE can also be used to assess key assumptions underlying leniency designs, including quasi-random assignment and average first-stage monotonicity, and to probe the external validity of treatment effect estimates. We further discuss statistical inference, arguing that non-clustered standard errors are often appropriate. A reanalysis of Farre-Mensa et al. (2020), using quasi-random examiner assignment to estimate the value of patents to startups, illustrates our checklist.

Citations

• Contamination Bias in Linear Regressions
• SOME THOUGHTS ON THE DISTRIBUTION OF EARNINGS 1
• Estimation of the Parameters of a Single Equation in a Complete System of Stochastic Equations

Cited by

• A Practical Guide to Instrumental Variables Methods with Heterogeneous Treatment Effects

Discussions

• New paper with @instrumenthull.bsky.social and Michal Kolesár on leniency/judge IV designs. This article is targeted for the Journal of Economic Perspectives, and so we wrote it with the goal of being [bsky, 52 points, 4 comments]
• See also for leniency designs: arxiv.org/abs/2511.03572 [bsky, 2 points, 1 comments]

Related

• A Sharp Test for the Judge Leniency Design
• Causal Judge Evaluation: Calibrated Surrogate Metrics for LLM Systems
• Reconciling Predictive and Statistical Parity: A Causal Approach
• Inspection-Guided Randomization: A Flexible and Transparent Restricted Randomization Framework for Better Experimental Design
• Keeping Designers in the Loop: Communicating Inherent Algorithmic Trade-offs Across Multiple Objectives