Learning Stateful Predictive Knowledge From Experience

2026/05/19 by Yan Song, Xidong Feng, Bo Liu +7
Computer Science · #cs.CL #cs.LG

paper · pdf

arxiv created 2026/05/19 · arxiv updated 2026/08/03

Abstract

As large language model (LLM) agents increasingly learn from experience, they primarily rely on trajectory-level reflection to extract insights. Viewed through the lens of predictive knowledge, we argue that this approach operates on episodic hindsight rather than predictive foresight, yielding brittle, path-dependent heuristics. To address this, we propose Stateful Knowledge Learning (SKL). SKL shifts the agent's focus from trajectory-level summarization to maintaining Stateful Knowledge: explicit, declarative predictive assessments anchored to state. We first demonstrate a motivating example showing how stateful knowledge provides granularity, enhances generalization, and enables knowledge bootstrapping. To further scale up the idea, we introduce two algorithms via self-distillation (SKL-SD) and reinforcement learning (SKL-RL), training agents to autonomously extract state-grounded predictive knowledge from experience and learn to leverage it for policy making. Experiments on interactive environments (WebShop, ScienceWorld) and a complex reasoning task (ChessPuzzles) demonstrate that equipping models with the inherent ability to learn stateful predictive knowledge significantly outpaces current reflection-based training paradigms.

Citations

• Reinforcement Learning via Self-Distillation
• AgentEvolver: Towards Efficient Self-Evolving Agent System
• Agent Learning via Early Experience
• ChessArena: A Chess Testbed for Evaluating Strategic Reasoning Capabilities of Large Language Models
• Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friends
• Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning
• Memento: Fine-tuning LLM Agents without Fine-tuning LLMs
• ComoRAG: A Cognitive-Inspired Memory-Organized RAG for Stateful Long Narrative Reasoning
• Group Sequence Policy Optimization
• Can Large Language Models Develop Strategic Reasoning? Post-training Insights from Learning Chess
• The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity
• Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback
• Dyna-Think: Synergizing Reasoning, Acting, and World Model Simulation in AI Agents
• Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning
• Trinity-RFT: A General-Purpose and Unified Framework for Reinforcement Fine-Tuning of Large Language Models
• Qwen3 Technical Report
• RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning
• ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learning
• Efficient Reinforcement Learning with Large Language Model Priors
• Natural Language Reinforcement Learning
• Amortized Planning with Large-Scale Transformers: A Case Study on Chess
• LLF-Bench: Benchmark for Interactive Learning from Language Feedback
• On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
• Large Language Models Are Semi-Parametric Reinforcement Learning Agents
• AdaPlanner: Adaptive Planning from Feedback with Language Models
• RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment
• Self-Refine: Iterative Refinement with Self-Feedback
• Reflexion: Language Agents with Verbal Reinforcement Learning
• Distributing Accountability, Not Capability: Phase Separation and the LLM Workflow Quadrant in Autonomous AI Agent Architectures
• WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
• ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
• World Modelling Improves Language Model Agents