System 2 Attention (is something you might need too)

2023/11/20 by Jason Weston, Sainbayar Sukhbaatar, Weston, Jason +1 · 5 voices · 10 citations
Computer Science · #Advanced Text Analysis Techniques #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Topic Modeling #cs.AI #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.2311.11829

openalex publication_date 2023/11/20 · arxiv published 2023/11/20 · arxiv updated 2023/11/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Soft attention in Transformer-based Large Language Models (LLMs) is susceptible to incorporating irrelevant information from the context into its latent representations, which adversely affects next token generations. To help rectify these issues, we introduce System 2 Attention (S2A), which leverages the ability of LLMs to reason in natural language and follow instructions in order to decide what to attend to. S2A regenerates the input context to only include the relevant portions, before attending to the regenerated context to elicit the final response. In experiments, S2A outperforms standard attention-based LLMs on three tasks containing opinion or irrelevant information, QA, math word problems and longform generation, where S2A increases factuality and objectivity, and decreases sycophancy.

Cited by

• Test-time Scaling of LLMs: A Survey from A Subproblem Structure Perspective
• Metacognition Should Be the Scientific Framework for Bounded and Effective Self-Governance in Generative AI
• LightMem: Lightweight and Efficient Memory-Augmented Generation
• Think Right: Learning to Mitigate Under-Over Thinking via Adaptive, Attentive Compression
• Expanding Computation Spaces of LLMs at Inference Time
• HEART: Emotionally-driven test-time scaling of Language Models
• WoW: Towards a World omniscient World model Through Embodied Interaction
• Enhancing Retrieval Augmentation via Adversarial Collaboration
• Survey of Specialized Large Language Model
• Leveraging Language Prior for Infrared Small Target Detection

Discussions

• Great name for a paper "System 2 Attention (is something you might need too)" Soft attention in Transformer-based Large Language Models (LLMs) is susceptible to incorporating irrelevant information [bsky, 3 points, 0 comments]
• System 2 Attention (is something you might need too) [hn, 3 points, 0 comments]
• System 2 Attention (is something you might need too) [hn, 2 points, 0 comments]
• System 2 Attention (is something you might need too) [hn, 1 points, 2 comments]
• One recent example: arxiv.org/pdf/2311.11829 [bsky, 0 points, 0 comments]

Related

• Pay Attention to What You Need
• Attention-likelihood relationship in transformers
• Don't Take Things Out of Context: Attention Intervention for Enhancing Chain-of-Thought Reasoning in Large Language Models
• Model Tells Itself Where to Attend: Faithfulness Meets Automatic Attention Steering
• Selective Attention Improves Transformer