The Few-shot Dilemma: Over-prompting Large Language Models

2025/09/16 by Yongjian Tang, Tang, Yongjian, Doruk Tuncel +5 · 2 citations
#cs.CL

paper · pdf · doi:10.48550/arxiv.2509.13196

Abstract

Over-prompting, a phenomenon where excessive examples in prompts lead to diminished performance in Large Language Models (LLMs), challenges the conventional wisdom about in-context few-shot learning. To investigate this few-shot dilemma, we outline a prompting framework that leverages three standard few-shot selection methods - random sampling, semantic embedding, and TF-IDF vectors - and evaluate these methods across multiple LLMs, including GPT-4o, GPT-3.5-turbo, DeepSeek-V3, Gemma-3, LLaMA-3.1, LLaMA-3.2, and Mistral. Our experimental results reveal that incorporating excessive domain-specific examples into prompts can paradoxically degrade performance in certain LLMs, which contradicts the prior empirical conclusion that more relevant few-shot examples universally benefit LLMs. Given the trend of LLM-assisted software engineering and requirement analysis, we experiment with two real-world software requirement classification datasets. By gradually increasing the number of TF-IDF-selected and stratified few-shot examples, we identify their optimal quantity for each LLM. This combined approach achieves superior performance with fewer examples, avoiding the over-prompting problem, thus surpassing the state-of-the-art by 1% in classifying functional and non-functional requirements.

Citations

• Gemma 3 Technical Report
• Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks
• GPT-4o System Card
• The Llama 3 Herd of Models
• FsPONER: Few-shot Prompt Optimization for Named Entity Recognition in Domain-specific Scenarios
• In-Context Learning with Long-Context Models: An In-Depth Exploration
• Long-context LLMs Struggle with Long In-context Learning
• Let's Learn Step by Step: Enhancing In-Context Learning Ability with Curriculum Learning
• Which AI Technique Is Better to Classify Requirements? An Experiment with SVM, LSTM, and ChatGPT
• Mistral 7B
• In-Context Learning for Text Classification with Many Labels
• YaRN: Efficient Context Window Extension of Large Language Models
• Graph of Thoughts: Solving Elaborate Problems with Large Language Models
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Lost in the Middle: How Language Models Use Long Contexts
• Tree of Thoughts: Deliberate Problem Solving with Large Language Models
• Symbol tuning improves in-context learning in language models
• GPT-NER: Named Entity Recognition via Large Language Models
• A Survey of Large Language Models
• GPT-4 Technical Report
• Larger language models do in-context learning differently
• Investigating the Effectiveness of Task-Agnostic Prefix Prompt for Instruction Following
• Augmented Language Models: a Survey
• SwitchPrompt: Learning Domain-Specific Gated Soft Prompts for Classification in Low-Resource Domains
• Large Language Models Can Be Easily Distracted by Irrelevant Context
• A Survey on In-context Learning
• Emergent Abilities of Large Language Models
• Large Language Models are Zero-Shot Reasoners
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing
• Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing
• Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• What Makes Good In-Context Examples for GPT-3?
• Making Pre-trained Language Models Better Few-shot Learners
• Language Models are Few-Shot Learners
• Longformer: The Long-Document Transformer
• DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• Generating Long Sequences with Sparse Transformers
• What Works Better? A Study of Classifying Requirements
• Attention Is All You Need
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• From Words to Wisdom: Discourse Annotation and Baseline Models for Student Dialogue Understanding
• Your thoughts tell who you are: Characterize the reasoning patterns of LRMs

Related

• Are Prompts All You Need? Evaluating Prompt-Based Large Language Models (LLM)s for Software Requirements Classification
• When Many-Shot Prompting Fails: An Empirical Study of LLM Code Translation
• Does Few-Shot Learning Help LLM Performance in Code Synthesis?
• Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models
• Making Pre-trained Language Models Better Few-shot Learners