Measurement Validity: A Shared Standard for Qualitative and Quantitative Research

2001/09/01 by Robert Adcock, David Collier · 186 citations
Engineering · Psychology · Social Sciences · #Artificial intelligence #Computer science #Context (archaeology) #Data science #Electoral Systems and Political Participation #Engineering #Epistemology #External validity #Generality #Geography #Law #Management science #Operationalization #Political science #Politics #Psychology #Social and Cultural Dynamics #Social psychology #Sociology #Variety (cybernetics)

paper · open access · doi:10.1017/s0003055401003100

published in American Political Science Review 95(3), 529-546 (Cambridge University Press)

openalex publication_date 2001/09/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/30

Abstract

Scholars routinely make claims that presuppose the validity of the observations and measurements that operationalize their concepts. Yet, despite recent advances in political science methods, surprisingly little attention has been devoted to measurement validity. We address this gap by exploring four themes. First, we seek to establish a shared framework that allows quantitative and qualitative scholars to assess more effectively, and communicate about, issues of valid measurement. Second, we underscore the need to draw a clear distinction between measurement issues and disputes about concepts. Third, we discuss the contextual specificity of measurement claims, exploring a variety of measurement strategies that seek to combine generality and validity by devoting greater attention to context. Fourth, we address the proliferation of terms for alternative measurement validation procedures and offer an account of the three main types of validation most relevant to political scientists.

Cited by

• Validating LLMs in social science: Epistemic threats and emerging norms
• What is a protest anyway? Codebook conceptualization is still a first-order concern in LLM-era classification
• Measuring Scalar Constructs in Social Science with LLMs
• Understanding and Meeting Practitioner Needs When Measuring Representational Harms Caused by LLM-Based Systems
• Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge
• Capturing legitimacy in captured states: an interview-based exploration of the validity of existing legitimacy measures in Serbia’s hybrid regime
• A new measurement model and database of the democratic qualities of regulatory bodies
• Aligning Recommendations with User Popularity Preferences
• The State of the Art in Visualization Literacy
• Toward Valid Measurement Of (Un)fairness For Generative AI: A Proposal For Systematization Through The Lens Of Fair Equality of Chances
• Measurement as Bricolage: Examining How Data Scientists Construct Target Variables for Predictive Modeling Tasks
• Wordkrill: Extending Wordfish into the multidimensional political space
• Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigor
• Large Language Models Can Be a Viable Substitute for Expert Political Surveys When a Shock Disrupts Traditional Measurement Approaches
• Identities are not Interchangeable: The Problem of Overgeneralization in Fair Machine Learning
• Bye Bye Perspective API: Lessons for Measurement Infrastructure in NLP, CSS and LLM Evaluation
• When Better Codebooks Are Not Enough: Predictive Performance and Behavioral Reliability in LLM Political Event Coding
• Health literacy and depression in the context of home visitation. [europepmc]
• Political leaders and the media. Can we measure political leadership images in newspapers using computer-assisted content analysis? [europepmc]
• Why simulations are appropriate for evaluating Qualitative Comparative Analysis. [europepmc]
• Can We Trust Measures of Political Trust? Assessing Measurement Equivalence in Diverse Regime Types. [europepmc]
• Missing the Unhealthy? Examining Empirical Validity of Material Deprivation Indices (MDIs) Using a Partial Criterion Variable. [europepmc]
• Integrating Qualitative and Quantitative Data in the Development of Outcome Measures: The Case of the Recovering Quality of Life (ReQoL) Measures in Mental Health Populations. [europepmc]
• Exploring how to evaluate a qualitative patient-centered outcome measure: literature review and illustrative example - a Perthes child-friendly measure. [europepmc]
• How We Do Things With Words: Analyzing Text as Social and Cultural Data. [europepmc]
• There Is More than Multiple Choice: Crowd-Sourced Assessment Tips for Online, Hybrid, and Face-to-Face Environments. [europepmc]
• The Academic Freedom Index and Its indicators: Introduction to new global time-series V-Dem data. [europepmc]
• Mixed methods instrument validation: Evaluation procedures for practitioners developed from the validation of the Swiss Instrument for Evaluating Interprofessional Collaboration. [europepmc]
• Empirical validation study and psychometric evaluation of the properties of the populist attitudes scale for the portuguese population. [europepmc]
• Measurements for the institutional cohesion dimension of the standard model of military group cohesion. [europepmc]
• 'If you are feeling alone and you are not feeling safe, it impacts everything': a mixed-methods exploration of international students' accommodation, subjective wellbeing and mental health help-seeking. [europepmc]
• Operationalizing data-driven campaigning: designing a new tool for mapping and guiding regulatory intervention. [europepmc]
• Cognitive warfare: a conceptual analysis of the NATO ACT cognitive warfare exploratory concept. [europepmc]

Related

• How Have Interpretivism and Big Data Changed the Landscape for Measurement Validation?
• Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them
• The Concept of Validity.
• Measure for Measure: Politics of Quantifying Individuals to Govern Them
• A Comparative Discussion of the Notion of 'Validity' in Qualitative and Quantitative Research