Does Tone Change the Answer? Evaluating Prompt Politeness Effects on Modern LLMs: GPT, Gemini, and LLaMA

2025/12/14 by Hanyu Cai, Cai, Hanyu, Binqi Shen +6 · 3 citations
Computer Science · Medicine · Social Sciences · #Artificial Intelligence (cs.AI) #Artificial Intelligence in Healthcare and Education #Computation and Language (cs.CL) #Computational and Text Analysis Methods #FOS: Computer and information sciences #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2512.12812

openalex publication_date 2025/12/14 · openalex created_date 2025/12/17 · openalex updated_date 2026/07/28

Abstract

Prompt engineering has emerged as a critical factor influencing large language model (LLM) performance, yet the impact of pragmatic elements such as linguistic tone and politeness remains underexplored, particularly across different model families. In this work, we propose a systematic evaluation framework to examine how interaction tone affects model accuracy and apply it to three recently released and widely available LLMs: GPT-4o mini (OpenAI), Gemini 2.0 Flash (Google DeepMind), and Llama 4 Scout (Meta). Using the MMMLU benchmark, we evaluate model performance under Very Polite, Neutral, and Very Rude prompt variants across six tasks spanning STEM and Humanities domains, and analyze pairwise accuracy differences with statistical significance testing. Our results show that tone sensitivity is both model-dependent and domain-specific. Neutral or Very Polite prompts generally yield higher accuracy than Very Rude prompts, but statistically significant effects appear only in a subset of Humanities tasks, where rude tone reduces accuracy for GPT and Llama, while Gemini remains comparatively tone-insensitive. When performance is aggregated across tasks within each domain, tone effects diminish and largely lose statistical significance. Compared with earlier research, these findings suggest that dataset scale and coverage materially influence the detection of tone effects. Overall, our study indicates that while interaction tone can matter in specific interpretive settings, modern LLMs are broadly robust to tonal variation in typical mixed-domain use, providing practical guidance for prompt design and model selection in real-world deployments.

Citations

• Let Language Constrain Geometry: Vision-Language Models as Semantic and Spatial Critics for 3D Generation
• Mind Your Tone: Investigating How Prompt Politeness Affects LLM Accuracy (short paper)
• Hy-Facial: Hybrid Feature Extraction by Dimensionality Reduction Methods for Enhanced Facial Expression Classification
• Vision-Language Models as Differentiable Semantic and Spatial Rewards for Text-to-3D Generation
• CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models
• Depth-Aware Super-Resolution via Distance-Adaptive Variational Formulation
• Instruction Tuning and CoT Prompting for Contextual Medical QA with LLMs
• FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Driving
• Dreamcrafter: Immersive Editing of 3D Radiance Fields Through Flexible, Generative Inputs and Outputs
• Offline Reinforcement Learning for LLM Multi-Step Reasoning
• Mitigating the Risk of Health Inequity Exacerbated by Large Language Models
• NEVLP: Noise-Robust Framework for Efficient Vision-Language Pre-training
• ComposerX: Multi-Agent Symbolic Music Composition with LLMs
• Should We Respect LLMs? A Cross-Lingual Study on the Influence of Prompt Politeness on LLM Performance
• Transformer-based Selective Super-Resolution for Efficient Image Refinement
• Efficient Large Language Models: A Survey
• Training language models to follow instructions with human feedback
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing
• Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing
• Measuring Massive Multitask Language Understanding
• Attention Is All You Need
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

Cited by

• Understanding Tone-Dependent Inference Cost in Large Language Models
• MARS: Multi-Agent Re-ranking for Repeat-Order Food Delivery Recommendation
• Enhancing Fake-News Detection with Node-Level Topological Features
• From Efficiency to Adaptivity: A Deeper Look at Adaptive Reasoning in Large Language Models
• Planning Waste-to-Energy-Coupled AI Data Centers Through Grade-Matched Cooling and Corridor Screening
• Stiff Circuit System Modeling via Transformer
• JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigation

Related

• Mind Your Tone: Investigating How Prompt Politeness Affects LLM Accuracy (short paper)
• Comparing human and LLM politeness strategies in free production
• Bias Beneath the Tone: Empirical Characterisation of Tone Bias in LLM-Driven UX Systems
• Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs
• Should We Respect LLMs? A Cross-Lingual Study on the Influence of Prompt Politeness on LLM Performance