Algorithmic Tradeoffs, Applied NLP, and the State-of-the-Art Fallacy

2025/09/10 by AJ Alvero, Alvero, AJ, Rong Dong +5
Social Sciences · #Applications (stat.AP) #Computational and Text Analysis Methods #Computers and Society (cs.CY) #Ethics and Social Impacts of AI #FOS: Computer and information sciences #Qualitative Research Methods and Applications

paper · pdf · doi:10.48550/arxiv.2509.08199

openalex publication_date 2025/09/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Computational sociology is growing in popularity, yet the analytic tools employed differ widely in power, transparency, and interpretability. In computer science, methods gain popularity after surpassing benchmarks of predictive accuracy, becoming the "state of the art." Computer scientists favor novelty and innovation for different reasons, but prioritizing technical prestige over methodological fit could unintentionally limit the scope of sociological inquiry. To illustrate, we focus on computational text analysis and revisit a prior study of college admissions essays, comparing analyses with both older and newer methods. These methods vary in flexibility and opacity, allowing us to compare performance across distinct methodological regimes. We find that newer techniques did not outperform prior results in meaningful ways. We also find that using the current state of the art, generative AI and large language models, could introduce bias and confounding that is difficult to extricate. We therefore argue that sociological inquiry benefits from methodological pluralism that aligns analytic choices with theoretical and empirical questions. While we frame this sociologically, scholars in other disciplines may confront what we call the "state-of-the-art fallacy", the belief that the tool computer scientists deem to be the best will work across topics, domains, and questions.

Citations

• Generative AI Meets Open-Ended Survey Responses: Research Participant Use of AI and Homogenization
• From Codebooks to Promptbooks: Extracting Information from Text with Generative Large Language Models
• The Mixed Subjects Design: Treating Large Language Models as Potentially Informative Observations
• Poor Alignment and Steerability of Large Language Models: Evidence from College Admission Essays
• What Good Is a Regression? Inference to the Best Explanation and the Practice of Political Science Research
• Large language models, social demography, and hegemony: comparing authorship in human and synthetic text
• BERTs are Generative In-Context Learners
• In Silico Sociology: Forecasting COVID-19 Polarization with Large Language Models
• Cultural bias and cultural alignment of large language models
• Whose Opinions Do Language Models Reflect?
• Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning
• From Ends to Means: The Promise of Computational Text Analysis for Theoretically Driven Sociological Research
• Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty
• Out of One, Many: Using Language Models to Simulate Human Samples
• BERTopic: Neural topic modeling with a class-based TF-IDF procedure
• Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research
• A Primer on Deep Learning for Causal Inference
• Torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision
• Theoretical foundations and limits of word embeddings: what types of meaning can they capture?
• Computational Social Science and Sociology
• Computational Grounded Theory: A Methodological Framework
• Troubling Trends in Machine Learning Scholarship
• The Geometry of Culture: Analyzing the Meanings of Class through Word Embeddings
• How to Make Causal Inferences Using Texts
• 50 Years of Data Science
• Automated Hate Speech Detection and the Problem of Offensive Language
• Sociology in the Era of Big Data: The Ascent of Forensic Social Science
• Convolutional Neural Networks for Sentence Classification
• Structural Topic Models for Open‐Ended Survey Responses
• Distributed Representations of Words and Phrases and their Compositionality
• Model Uncertainty in Sociological Research: An Application to Religion and Economic Growth
• A Sociology of Quantification
• Transcending General Linear Reality
• The file drawer problem and tolerance for null results.
• AI-Augmented Surveys: Leveraging Large Language Models and Surveys for Opinion Prediction
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• From Ends to Means: The Promise of Computational Text Analysis for Theoretically Driven Sociological Research
• Generative AI in Sociological Research: State of the Discipline
• Computational Text Analysis for Building and Testing Social Theory
• Qualitative Research in an Era of AI: A Pragmatic Approach to Data Analysis, Workflow, and Computation
• Tools of the Trade – When Are Software Tools Mentioned in Computational Text Analysis Research?