Structured Distillation for Personalized Agent Memory: 11x Token Reduction with Retrieval Preservation

2026/03/13 by Sydney Lewis · 1 voice
Computer Science · #Context (archaeology) #Distillation #Information Retrieval and Search Behavior #Object (grammar) #Pipeline (software) #Recall #Reduction (mathematics) #Security token #Simple (philosophy) #Structured prediction #Topic Modeling #Web Data Mining and Analysis #cs.AI #cs.CL #cs.IR

paper · pdf · doi:10.48550/arxiv.2603.13017

openalex publication_date 2026/03/13 · arxiv published 2026/03/13 · arxiv updated 2026/03/13 · openalex created_date 2026/03/17 · openalex updated_date 2026/07/28

Abstract

Long conversations with an AI agent create a simple problem for one user: the history is useful, but carrying it verbatim is expensive. We study personalized agent memory: one user's conversation history with an agent, distilled into a compact retrieval layer for later search. Each exchange is compressed into a compound object with four fields (exchange_core, specific_context, thematic room_assignments, and regex-extracted files_touched). The searchable distilled text averages 38 tokens per exchange. Applied to 4,182 conversations (14,340 exchanges) from 6 software engineering projects, the method reduces average exchange length from 371 to 38 tokens, yielding 11x compression. We evaluate whether personalized recall survives that compression using 201 recall-oriented queries, 107 configurations spanning 5 pure and 5 cross-layer search modes, and 5 LLM graders (214,519 consensus-graded query-result pairs). The best pure distilled configuration reaches 96% of the best verbatim MRR (0.717 vs 0.745). Results are mechanism-dependent. All 20 vector search configurations remain non-significant after Bonferroni correction, while all 20 BM25 configurations degrade significantly (effect sizes |d|=0.031-0.756). The best cross-layer setup slightly exceeds the best pure verbatim baseline (MRR 0.759). Structured distillation compresses single-user agent memory without uniformly sacrificing retrieval quality. At 1/11 the context cost, thousands of exchanges fit within a single prompt while the verbatim source remains available for drill-down. We release the implementation and analysis pipeline as open-source software.

Citations

• Qwen3 Technical Report
• Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
• InternLM2 Technical Report
• Yi: Open Foundation Models by 01.AI
• MemGPT: Towards LLMs as Operating Systems
• Mistral 7B
• LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
• A Survey on Dialogue Summarization: Recent Advances and New Frontiers
• BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• Billion-scale similarity search with GPUs
• Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs
• The Kappa Statistic in Reliability Studies: Use, Interpretation, and Sample Size Requirements
• Cumulated gain-based evaluation of IR techniques
• The Measurement of Observer Agreement for Categorical Data

Discussions

• In my morning recap from Lares: "Did a deep dive on this one. 11x conversation compression, 96% retrieval quality preserved. Their architecture validates our Hebbian + QMD approach. Structured extract [bsky, 1 points, 1 comments]

Related

• SimpleMem: Efficient Lifelong Memory for LLM Agents
• Deployment-Time Memorization in Foundation-Model Agents
• Are We Ready For An Agent-Native Memory System?
• ENGRAM: Effective, Lightweight Memory Orchestration for Conversational Agents
• Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontiers