The Impact of Role Design in In-Context Learning for Large Language Models

2025/09/27 by Rouzegar, Hamidreza, Makrehchi, Masoud
#68T50 #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2.7

paper · doi:10.48550/arxiv.2509.23501

Abstract

In-context learning (ICL) enables Large Language Models (LLMs) to generate predictions based on prompts without additional fine-tuning. While prompt engineering has been widely studied, the impact of role design within prompts remains underexplored. This study examines the influence of role configurations in zero-shot and few-shot learning scenarios using GPT-3.5 and GPT-4o from OpenAI and Llama2-7b and Llama2-13b from Meta. We evaluate the models' performance across datasets, focusing on tasks like sentiment analysis, text classification, question answering, and math reasoning. Our findings suggest the potential of role-based prompt structuring to enhance LLM performance.

Citations

• Task Facet Learning: A Structured Approach to Prompt Optimization
• MEND: Meta dEmonstratioN Distillation for Efficient and Effective In-Context Learning
• In-context Pretraining: Language Modeling Beyond Document Boundaries
• Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functions
• Generative AI Text Classification using Ensemble LLM Approaches
• Instruction Tuning for Large Language Models: A Survey
• Better Zero-Shot Reasoning with Role-Play Prompting
• Sentiment Analysis in the Era of Large Language Models: A Reality Check
• MoT: Memory-of-Thought Enables ChatGPT to Self-Improve
• Investigating the Effectiveness of Task-Agnostic Prefix Prompt for Instruction Following
• LLaMA: Open and Efficient Foundation Language Models
• A Survey on In-context Learning
• Self-Instruct: Aligning Language Models with Self-Generated Instructions
• Demystifying Prompts in Language Models via Perplexity Estimation
• Active Example Selection for In-Context Learning
• Large Language Models Are Human-Level Prompt Engineers
• Measuring and Narrowing the Compositionality Gap in Language Models
• Complexity-Based Prompting for Multi-Step Reasoning
• Instruction Induction: From Few Examples to Natural Language Task Descriptions
• Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
• Measuring Mathematical Problem Solving With the MATH Dataset
• What Makes Good In-Context Examples for GPT-3?
• Investigating Societal Biases in a Poetry Composition System
• Language Models are Few-Shot Learners
• CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge

Related

• Designing Role Vectors to Improve LLM Inference Behaviour
• Role-Play Paradox in Large Language Models: Reasoning Performance Gains and Ethical Dilemmas
• Deconstructing In-Context Learning: Understanding Prompts via Corruption
• Effects of Prompt Length on Domain-specific Tasks for Large Language Models
• Supervised Knowledge Makes Large Language Models Better In-context Learners