Is External Information Useful for Stance Detection with LLMs?

2025/07/02 by Quang Minh Nguyen, Nguyen, Quang Minh, Taegyoon Kim +1 · 1 citation
Computer Science · Social Sciences · #Affect (linguistics) #Code (set theory) #Computation and Language (cs.CL) #Contrast (vision) #FOS: Computer and information sciences #Information Retrieval and Search Behavior #Macro #Misinformation and Its Impacts #Motivated reasoning #Topic Modeling #Work (physics)

paper · pdf · doi:10.48550/arxiv.2507.01543

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/07/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

In the stance detection task, a text is classified as either favorable, opposing, or neutral towards a target. Prior work suggests that the use of external information, e.g., excerpts from Wikipedia, improves stance detection performance. However, whether or not such information can benefit large language models (LLMs) remains an unanswered question, despite their wide adoption in many reasoning tasks. In this study, we conduct a systematic evaluation on how Wikipedia and web search external information can affect stance detection across eight LLMs and in three datasets with 12 targets. Surprisingly, we find that such information degrades performance in most cases, with macro F1 scores dropping by up to 27.9%. We explain this through experiments showing LLMs' tendency to align their predictions with the stance and sentiment of the provided information rather than the ground truth stance of the given text. We also find that performance degradation persists with chain-of-thought prompting, while fine-tuning mitigates but does not fully eliminate it. Our findings, in contrast to previous literature on BERT-based systems which suggests that external information enhances performance, highlight the risks of information biases in LLM-based stance classifiers. Code is available at https://github.com/ngqm/acl2025-stance-detection.

Citations

• Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting
• Stance Detection with Collaborative Role-Infused LLM-Based Agents
• The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Large Language Models are Zero-Shot Reasoners
• Infusing Knowledge from Wikipedia to Enhance Stance Detection
• Mitigating Biases of Large Language Models in Stance Detection with Counterfactual Augmented Calibration

Cited by

• A Systematic Comparison of Prompting and Multi-Agent Methods for LLM-based Stance Detection

Related

• Infusing Knowledge from Wikipedia to Enhance Stance Detection
• Chain of Stance: Stance Detection with Large Language Models
• Prompting and Fine-Tuning Open-Sourced Large Language Models for Stance Classification
• Stance Detection on Social Media with Fine-Tuned Large Language Models
• Large Language Models Meet Stance Detection: A Survey of Tasks, Methods, Applications, Challenges and Future Directions