Rethinking Scaling Up Content Analysis: A Reappraisal of Justifications and Practices for Large-Scale Content Analysis

2025/10/16 by Maximilian Linde, Chung‐hong Chan, Paul Balluff · 1 voice · 1 citation
Social Sciences · #Computational and Text Analysis Methods

paper · doi:10.1177/10776990251378122

openalex publication_date 2025/10/16 · openalex created_date 2025/10/17 · openalex updated_date 2026/07/22

Abstract

The use of automated coding procedures to scale up content analysis has risen over the last years. Using a mixed-method approach, we examine researchers’ justifications to scale up content analysis and assess the methodological adjustments—or lack thereof—when employing supervised machine learning for 38 large-scale content analyses. Almost all of the included studies displayed deficiencies in study design, primarily related to the uncritical use of frequentist statistics on datasets containing the entire statistical population, or employing supervised machine learning without methodological adjustments to account for misclassifications. Our findings question the need for large datasets and automated coding in the first place.

Citations

• Using thematic analysis in psychology
• A practical solution to the pervasive problems ofp values
• Content Analysis and the Algorithmic Coder
• Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author)
• Group sequential methods in the design and analysis of clinical trials
• Bayesian inference for psychology. Part I: Theoretical advantages and practical ramifications
• effectsize: Estimation of Effect Size Indices and Standardized Parameters
• Conceptual and design thinking for thematic analysis.
• Bayes Factors
• Advancing Automated Content Analysis for a New Era of Media Effects Research: The Key Role of Transfer Learning
• Topic Model Validation Methods and their Impact on Model Selection and Evaluation
• Automated Visual Analysis for the Study of Social Media Effects: Opportunities, Approaches, and Challenges
• Integrating Communication Science and Computational Methods to Study Content-Based Social Media Effects
• Misclassification in Automated Content Analysis Causes Bias in Regression. Can We Fix It? Yes We Can!
• Tools of the Trade – When Are Software Tools Mentioned in Computational Text Analysis Research?
• How to Make Causal Inferences Using Texts
• From digital positivism and administrative big data analytics towards critical digital and social media research!
• Optional stopping: No problem for Bayesians
• Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts
• CRITICAL QUESTIONS FOR BIG DATA

Cited by

• The Role of the Term “Fake News” in U.S. Alternative Media: Shaping Political Discourse and Hyperpartisanship

Discussions

• 🚨 A free ad for Maximilian Linde, Chung-Hong Chan & Paul Balluff's important Public Service Announcement for communication researchers: 🛑 Stop competing on text corpus size! Scale is rarely needed a [bsky, 1 points, 0 comments]

Related

• Content Analysis in an Era of Big Data: A Hybrid Approach to Computational and Manual Methods
• Content Analysis and the Algorithmic Coder
• The Future of Coding: A Comparison of Hand-Coding and Three Types of Computer-Assisted Text Analysis Methods
• SCALE: Towards Collaborative Content Analysis in Social Science with Large Language Model Agents and Human Intervention
• Misclassification in Automated Content Analysis Causes Bias in Regression. Can We Fix It? Yes We Can!