ParaScopes: What do Language Models Activations Encode About Future Text?

2025/10/31 by Nicky Pochinkov, Yulia Volkova, Pochinkov, Nicky +5
Computer Science · #Explainable Artificial Intelligence (XAI) #AI-based Problem Solving and Planning #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2511.00180

Abstract

Interpretability studies in language models often investigate forward-looking representations of activations. However, as language models become capable of doing ever longer time horizon tasks, methods for understanding activations often remain limited to testing specific concepts or tokens. We develop a framework of Residual Stream Decoders as a method of probing model activations for paragraph-scale and document-scale plans. We test several methods and find information can be decoded equivalent to 5+ tokens of future context in small models. These results lay the groundwork for better monitoring of language models and better understanding how they might encode longer-term planning information.

Citations

• Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models
• Gemma 3 Technical Report
• Measuring AI Ability to Complete Long Software Tasks
• Auditing language models for hidden objectives
• On the Emergence of Position Bias in Transformers
• Large Concept Models: Language Modeling in a Sentence Representation Space
• LatentQA: Teaching LLMs to Decode Activations Into Natural Language
• Extracting Paragraphs from LLM Token Activations
• The Llama 3 Herd of Models
• Gemma 2: Improving Open Language Models at a Practical Size
• Planning in a recurrent neural network that plays Sokoban
• Predicting vs. Acting: A Trade-off Between World Modeling & Agent Modeling
• The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
• Scaling and evaluating sparse autoencoders
• Evidence of Learned Look-Ahead in a Chess-Playing Neural Network
• Do language models plan ahead for future tokens?
• Speculative Streaming: Fast LLM Inference without Auxiliary Models
• Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models
• Prometheus: Inducing Fine-grained Evaluation Capability in Language Models
• Sparse Autoencoders Find Highly Interpretable Features in Language Models
• SONAR: Sentence-Level Multimodal and Language-Agnostic Representations
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
• Discovering Latent Knowledge in Language Models Without Supervision
• Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small
• In-context Learning and Induction Heads
• Locating and Editing Factual Associations in GPT
• Learning Compact Metrics for MT
• Probing Classifiers: Promises, Shortcomings, and Advances
• Probing Classifiers: Promises, Shortcomings, and Advances
• What do you learn from context? Probing for sentence structure in contextualized word representations
• Attention Is All You Need

Related

• Extracting Paragraphs from LLM Token Activations
• Unlocking the Future: Exploring Look-Ahead Planning Mechanistic Interpretability in Large Language Models
• Internal Planning in Language Models: Characterizing Horizon and Branch Awareness
• Mind the Gap: Assessing Temporal Generalization in Neural Language Models
• Superscopes: Amplifying Internal Feature Representations for Language Model Interpretation