Directed Information γ-covering: An Information-Theoretic Framework for Context Engineering

2025/09/30 by Hai Huang, Huang, Hai
Computer Science · #Context-Aware Activity Recognition Systems #Speech and dialogue systems #Semantic Web and Ontologies

paper · pdf · doi:10.48550/arxiv.2510.00079

Abstract

We introduce Directed Information γ-covering, a simple but general framework for redundancy-aware context engineering. Directed information (DI), a causal analogue of mutual information, measures asymmetric predictiveness between chunks. If DI_{i → j} ≥ H(C_j) - γ, then C_i suffices to represent C_j up to γ bits. Building on this criterion, we formulate context selection as a γ-cover problem and propose a greedy algorithm with provable guarantees: it preserves query information within bounded slack, inherits (1+ln n) and (1-1/e) approximations from submodular set cover, and enforces a diversity margin. Importantly, building the γ-cover is query-agnostic: it incurs no online cost and can be computed once offline and amortized across all queries. Experiments on HotpotQA show that γ-covering consistently improves over BM25, a competitive baseline, and provides clear advantages in hard-decision regimes such as context compression and single-slot prompt selection. These results establish DI γ-covering as a principled, self-organizing backbone for modern LLM pipelines.

Citations

• A Survey of Context Engineering for Large Language Models
• Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directions
• Pointwise Mutual Information as a Performance Gauge for Retrieval-Augmented Generation
• SirLLM: Streaming Infinite Retentive LLM
• Information-Theoretic Distillation for Reference-less Summarization
• LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression
• LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
• LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
• Compressing Context to Enhance Inference Efficiency of Large Language Models
• A Cookbook of Self-Supervised Learning
• Language Models are Few-Shot Learners
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Passage Re-ranking with BERT
• HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
• Information-theoretic analysis of generalization capability of learning algorithms
• A Coding Theorem for a Class of Stationary Channels with Feedback
• The information bottleneck method
• An analysis of approximations for maximizing submodular set functions—I

Related

• QUITO-X: A New Perspective on Context Compression from the Information Bottleneck Theory
• InfoFlow KV: Information-Flow-Aware KV Recomputation for Long Context
• Retaining Key Information under High Compression Ratios: Query-Guided Compressor for LLMs
• DAST: Context-Aware Compression in LLMs via Dynamic Allocation of Soft Tokens
• Exploring Information Processing in Large Language Models: Insights from Information Bottleneck Theory