Leniency Designs: An Operator's Manual

2025/11/05 by Paul Goldsmith-Pinkham, Peter Hull, Goldsmith-Pinkham, Paul +3 · 2 voices · 1 citation
Mathematics · Economics, Econometrics and Finance · Business, Management and Accounting · #Advanced Causal Inference Techniques #Innovation Policy and R&D #Intellectual Property and Patents

paper · pdf · doi:10.48550/arxiv.2511.03572

Abstract

We develop a step-by-step guide to leniency (a.k.a. judge or examiner instrument) designs, drawing on recent econometric literatures. The unbiased jackknife instrumental variables estimator (UJIVE) is purpose-built for leveraging exogenous leniency variation, avoiding subtle biases even in the presence of many decision-makers or controls. We show how UJIVE can also be used to assess key assumptions underlying leniency designs, including quasi-random assignment and average first-stage monotonicity, and to probe the external validity of treatment effect estimates. We further discuss statistical inference, arguing that non-clustered standard errors are often appropriate. A reanalysis of Farre-Mensa et al. (2020), using quasi-random examiner assignment to estimate the value of patents to startups, illustrates our checklist.

Citations

• Contamination Bias in Linear Regressions
• SOME THOUGHTS ON THE DISTRIBUTION OF EARNINGS 1
• Estimation of the Parameters of a Single Equation in a Complete System of Stochastic Equations

Cited by

• A Practical Guide to Instrumental Variables Methods with Heterogeneous Treatment Effects

Discussions

• New paper with @instrumenthull.bsky.social and Michal Kolesár on leniency/judge IV designs. This article is targeted for the Journal of Economic Perspectives, and so we wrote it with the goal of being [bsky, 52 points, 4 comments]
• See also for leniency designs: arxiv.org/abs/2511.03572 [bsky, 2 points, 1 comments]

Related

• Causal Judge Evaluation: Calibrated Surrogate Metrics for LLM Systems
• Reconciling Predictive and Statistical Parity: A Causal Approach
• Keeping Designers in the Loop: Communicating Inherent Algorithmic Trade-offs Across Multiple Objectives
• Mind the Gap: A Causal Perspective on Bias Amplification in Prediction & Decision-Making
• JustEva: A Toolkit to Evaluate LLM Fairness in Legal Knowledge Inference