Measuring and Controlling Instruction (In)Stability in Language Model Dialogs

2024/02/13 by Kenneth Li, Tianle Liu, Li, Kenneth +12 · 1 voice · 25 citations
Computer Science · #Computer science #Human–computer interaction #Linguistics #Natural language processing #Persona #Persona Design and Applications #Philosophy

paper · pdf · doi:10.48550/arxiv.2402.10962

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2024/02/13 · openalex created_date 2024/02/21 · openalex updated_date 2026/07/28

Abstract

System-prompting is a standard tool for customizing language-model chatbots, enabling them to follow a specific instruction. An implicit assumption in the use of system prompts is that they will be stable, so the chatbot will continue to generate text according to the stipulated instructions for the duration of a conversation. We propose a quantitative benchmark to test this assumption, evaluating instruction stability via self-chats between two instructed chatbots. Testing popular models like LLaMA2-chat-70B and GPT-3.5, we reveal a significant instruction drift within eight rounds of conversations. An empirical and theoretical analysis of this phenomenon suggests the transformer attention mechanism plays a role, due to attention decay over long exchanges. To combat attention decay and instruction drift, we propose a lightweight method called split-softmax, which compares favorably against two strong baselines.

Cited by

• ContextEcho: A Benchmark for Persona Drift in Long Agentic-Coding Sessions
• Language Shapes Instruction Hierarchy Compliance in Multilingual LLMs
• Prompt Governance? On Governing Technologies Governed by Natural Language
• Persistent Personas? Role-Playing, Instruction Following, and Safety in Extended Interactions
• Reasoning Up the Instruction Ladder for Controllable Language Models
• Attention-Aligned Reasoning for Large Language Models
• Drift No More? Context Equilibria in Multi-Turn LLM Interactions
• When Instructions Multiply: Measuring and Estimating LLM Capabilities of Multiple Instructions Following
• Psychometric Personality Shaping Modulates Capabilities and Safety in Language Models
• Are LLM Agents Behaviorally Coherent? Latent Profiles for Social Simulation
• IROTE: Human-like Traits Elicitation of Large Language Model via In-Context Self-Reflective Optimization
• Persistent Instability in LLM's Personality Measurements: Effects of Scale, Reasoning, and Conversation History
• Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representations
• Goal Alignment in LLM-Based User Simulators for Conversational AI
• A Comprehensive Review on Harnessing Large Language Models to Overcome Recommender System Challenges
• When Harry Meets Superman: The Role of The Interlocutor in Persona-Based Dialogue Generation
• Position is Power: System Prompts as a Mechanism of Bias in Large Language Models (LLMs)
• Beyond Prompt Engineering: Robust Behavior Control in LLMs via Steering Target Atoms
• Spotlight Your Instructions: Instruction-following with Dynamic Attention Steering
• Role Steering of Language Models for Social Simulations
• Suffix-Constrained Greedy Search Algorithms for Causal Language Models
• Measuring What Persists: Conditioning Mechanisms and a Geometric Framework for AI Agent Identity
• Attractor States Emerge in Multi-Turn LLM Conversations
• Mitigating Misalignment Contagion by Steering with Implicit Traits
• Mechanisms of Prompt-Induced Hallucination in Vision-Language Models
• Position: It's Time to Optimize LLMs for Self-Consistency

Discussions

• You can prompt your AI, but will it stay on task? New paper from our lab quantifies and analyzes "persona drift"—the tendency of a chatbot to wander away from its initial instructions. We also propose [bsky, 20 points, 1 comments]

Related

• CantTalkAboutThis: Aligning Language Models to Stay on Topic in Dialogues
• Navigating the Synchrony-Stability Frontier in Adaptive Chatbots
• ConsistentChat: Building Skeleton-Guided Consistent Multi-Turn Dialogues for Large Language Models from Scratch
• Can You Follow Me? Testing Situational Understanding in ChatGPT
• Enhancing Chat Language Models by Scaling High-quality Instructional Conversations