Extracting Paragraphs from LLM Token Activations

2024/09/10 by Nicholas Pochinkov, A. Benoı̂t, Pochinkov, Nicholas +6 · 3 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Mathematics, Computing, and Information Processing #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2409.06328

openalex publication_date 2024/09/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Generative large language models (LLMs) excel in natural language processing tasks, yet their inner workings remain underexplored beyond token-level predictions. This study investigates the degree to which these models decide the content of a paragraph at its onset, shedding light on their contextual understanding. By examining the information encoded in single-token activations, specifically the "\textbackslash n\textbackslash n" double newline token, we demonstrate that patching these activations can transfer significant information about the context of the following paragraph, providing further insights into the model's capacity to plan ahead.

Cited by

• The Curious Case of Analogies: Investigating Analogical Reasoning in Large Language Models
• ParaScopes: What do Language Models Activations Encode About Future Text?
• Detecting and Characterizing Planning in Language Models

Related

• Unraveling Token Prediction Refinement and Identifying Essential Layers in Language Models
• Token Prepending: A Training-Free Approach for Eliciting Better Sentence Embeddings from LLMs
• ParaScopes: What do Language Models Activations Encode About Future Text?
• Taking a Deep Breath: Enhancing Language Modeling of Large Language Models with Sentinel Tokens
• PromptExp: Multi-granularity Prompt Explanation of Large Language Models