Stay Tuned: Improving Sentiment Analysis and Stance Detection Using Large Language Models

2025/12/17 by Max Griswold, Michael W. Robbins, Michael S. Pollard +1
Computer Science · Social Sciences · #Computational and Text Analysis Methods #Misinformation and Its Impacts #Sentiment Analysis and Opinion Mining

paper · pdf · doi:10.1017/pan.2025.10023

openalex created_date 2025/12/17 · openalex publication_date 2025/12/17 · openalex updated_date 2026/07/10

Abstract

Abstract Sentiment analysis and stance detection are key tasks in text analysis, with applications ranging from understanding political opinions to tracking policy positions. Recent advances in large language models (LLMs) offer significant potential to enhance sentiment analysis techniques and to evolve them into the more nuanced task of detecting stances expressed toward specific subjects. In this study, we evaluate lexicon-based models, supervised models, and LLMs for stance detection using two corpuses of social media data—a large corpus of tweets posted by members of the U.S. Congress on Twitter and a smaller sample of tweets from general users—which both focus on opinions concerning presidential candidates during the 2020 election. We consider several fine-tuning strategies to improve performance—including cross-target tuning using an assumption of congressmembers’ stance based on party affiliation—and strategies for fine-tuning LLMs, including few shot and chain-of-thought prompting. Our findings demonstrate that: 1) LLMs can distinguish stance on a specific target even when multiple subjects are mentioned, 2) tuning leads to notable improvements over pretrained models, 3) cross-target tuning can provide a viable alternative to in-target tuning in some settings, and 4) complex prompting strategies lead to improvements over pretrained models but underperform tuning approaches.

Citations

• Cross-Target Stance Detection: A Survey of Techniques, Datasets, and Challenges
• Stance Detection on Social Media with Fine-Tuned Large Language Models
• The Political Preferences of LLMs
• Cross-target Stance Detection by Exploiting Target Analytical Perspectives
• Stance Detection with Collaborative Role-Infused LLM-Based Agents
• More human than human: measuring ChatGPT political bias
• Less Annotating, More Classifying: Addressing the Data Scarcity Issue of Supervised Machine Learning with Deep Transfer Learning and BERT-NLI
• Sentiment Analysis in the Era of Large Language Models: A Reality Check
• Navigating Prompt Complexity for Zero-Shot Classification: A Study of Large Language Models in Computational Social Science
• Automated stance detection in complex topics and small languages: the challenging case of immigration in polarizing news media
• Stance Detection: A Practical Guide to Classifying Political Beliefs in Text
• Investigating Chain-of-thought with ChatGPT for Stance Detection on Social Media
• AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators
• Legal and Political Stance Detection of SCOTUS Language
• More than a Feeling: Accuracy and Application of Sentiment Analysis
• RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning
• TimeLMs: Diachronic Language Models from Twitter
• True Few-Shot Learning with Language Models
• Making Pre-trained Language Models Better Few-shot Learners
• Political Polarization on the Digital Sphere: A Cross-platform, Over-time Analysis of Interactional, Positional, and Affective Polarization on Social Media
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• Language Models are Few-Shot Learners
• X-Stance: A Multilingual Multi-Target Dataset for Stance Detection
• Stance Detection Benchmark: How Robust Is Your Stance Detection?
• The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation
• DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• WASSA-2017 Shared Task on Emotion Intensity
• Attention Is All You Need
• Stance and Sentiment in Tweets
• Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts
• Lexicon-Based Methods for Sentiment Analysis
• Thumbs up? Sentiment Classification using Machine Learning Techniques
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Large Language Models are Zero-Shot Reasoners
• ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning
• Open-Source LLMs for Text Annotation: A Practical Guide for Model Setting and Fine-Tuning
• Positioning Political Texts with Large Language Models by Asking and Averaging
• Building Efficient Universal Classifiers with Natural Language Inference

Related

• Large Language Models Meet Stance Detection: A Survey of Tasks, Methods, Applications, Challenges and Future Directions
• Predicting User Stances from Target-Agnostic Information using Large Language Models
• Advancing Annotation of Stance in Social Media Posts: A Comparative Analysis of Large Language Models and Crowd Sourcing
• A Survey of Stance Detection on Social Media: New Directions and Perspectives
• Stance Detection on Social Media with Fine-Tuned Large Language Models