Information Equivalence in Survey Experiments

2018/08/03 by Allan Dafoe, Baobao Zhang, Devin Caughey · 361 citations
Economics, Econometrics and Finance · Mathematics · Social Sciences · #Advanced Causal Inference Techniques #Artificial intelligence #Computer science #Democracy #Econometrics #Economic and Environmental Valuation #Economics #Electoral Systems and Political Participation #Embedding #Equivalence (formal languages) #Ex-ante #Mathematics #Natural experiment #Political science #Statistics

paper · pdf · doi:10.1017/pan.2018.9

published in Political Analysis 26(4), 399-416 (Cambridge University Press)

openalex publication_date 2018/08/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/04

Abstract

Survey experiments often manipulate the description of attributes in a hypothetical scenario, with the goal of learning about those attributes’ real-world effects. Such inferences rely on an underappreciated assumption: experimental conditions must be information equivalent (IE) with respect to background features of the scenario. IE is often violated because subjects, when presented with information about one attribute, update their beliefs about others too. Labeling a country “a democracy,” for example, affects subjects’ beliefs about the country’s geographic location. When IE is violated, the effect of the manipulation need not correspond to the quantity of interest (the effect of beliefs about the focal attribute). We formally define the IE assumption, relating it to the exclusion restriction in instrumental-variable analysis. We show how to predict IE violations ex ante and diagnose them ex post with placebo tests. We evaluate three strategies for achieving IE. Abstract encouragement is ineffective. Specifying background details reduces imbalance on the specified details and highly correlated details, but not others. Embedding a natural experiment in the scenario can reduce imbalance on all background beliefs, but raises other issues. We illustrate with four survey experiments, focusing on an extension of a prominent study of the democratic peace.

Citations

• Are Survey Experiments Externally Valid?
• On the psychology of prediction.
• The Dangers of Extreme Counterfactuals
• Unpacking the Black Box of Causality: Learning about Causal Mechanisms from Experimental and Observational Studies
• Attitudes toward Highly Skilled and Low-skilled Immigration: Evidence from a Survey Experiment
• Public Opinion and the Democratic Peace
• Causal Inference in Conjoint Analysis: Understanding Multidimensional Choices via Stated Preference Experiments
• Authoritarian Audiences, Rhetoric, and Propaganda in International Crises: Evidence from China

Cited by

• Delegating Destruction: Coercive Threats and Automated Nuclear Systems
• Inclusion to exclude: how femonationalism impacts policy preferences
• Race, Democracy, and Public Support for War
• Do Markets Shape Management? Experimental Evidence for the Effects of Competition on Contract Management
• Ingroup preferences, segregation, and intergroup contact in neighborhoods and civic organizations
• Procedural Polarization: How Election Rules Shape Voters’ Confidence
• Gendered perceptions of police conduct: public reactions to officer gender in a traffic stop scenario
• Artificial Intelligence in Government: Why People Feel They Lose Control
• What is the Causal Effect of a Conversation? Estimands and Inference in AI Mediated Conversations
• Out of the Loop Again: How Dangerous is Weaponizing Automated Nuclear Systems?
• Electoral Gender Quotas and Democratic Legitimacy

Related

• Are Survey Experiments Externally Valid?
• Choosing The Best Incentives for Belief Elicitation with an Application to Political Protests
• Identifying Causal Effects in Information Provision Experiments
• Demand Effects in Survey Experiments: An Empirical Assessment
• More than meets the ITT: A guide for anticipating and investigating nonsignificant results in survey experiments