MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memory

2026/01/06 by Shengtao Zhang, Jiaqian Wang, Ruiwen Zhou +11 · 1 voice · 3 citations
Computer Science · #cs.CL

paper · pdf · doi:10.48550/arxiv.2601.03192

arxiv published 2026/01/06 · arxiv updated 2026/02/12

Abstract

The hallmark of human intelligence is the self-evolving ability to master new skills by learning from past experiences. However, current AI agents struggle to emulate this self-evolution: fine-tuning is computationally expensive and prone to catastrophic forgetting, while existing memory-based methods rely on passive semantic matching that often retrieves noise. To address these challenges, we propose MemRL, a non-parametric approach that evolves via reinforcement learning on episodic memory. By decoupling stable reasoning from plastic memory, MemRL employs a Two-Phase Retrieval mechanism to filter noise and identify high-utility strategies through environmental feedback. Extensive experiments on HLE, BigCodeBench, ALFWorld, and Lifelong Agent Bench demonstrate that MemRL significantly outperforms state-of-the-art baselines, confirming that MemRL effectively reconciles the stability-plasticity dilemma, enabling continuous runtime improvement without weight updates. Code is available at https://github.com/MemTensor/MemRL.

Citations

• Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory
• LiCoMemory: Lightweight and Cognitive Agentic Memory for Efficient Long-Term Reasoning
• Retrieval Feedback Memory Enhancement Large Model Retrieval Generation Method
• Memento: Fine-tuning LLM Agents without Fine-tuning LLMs
• Learn to Memorize: Optimizing LLM-based Agents with Adaptive Memory Framework
• Memp: Exploring Agent Procedural Memory
• MemOS: An Operating System for Memory-Augmented Generation (MAG) in Large Language Models
• Task Memory Engine: Spatial Memory for Robust Multi-Step LLM Agents
• LifelongAgentBench: Evaluating LLM Agents as Lifelong Learners
• Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learning
• Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory
• MemInsight: Autonomous Memory Augmentation for LLM Agents
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• A-MEM: Agentic Memory for LLM Agents
• BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
• Revisiting Catastrophic Forgetting in Large Language Model Tuning
• Similarity is Not All You Need: Endowing Retrieval Augmented Generation with Multi Layered Thoughts
• TRAD: Enhancing LLM Agents with Step-Wise Thought Retrieval and Aligned Decision
• Retrieval-Augmented Generation for Large Language Models: A Survey
• Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Voyager: An Open-Ended Embodied Agent with Large Language Models
• A Comprehensive Survey on Test-Time Adaptation Under Distribution Shifts
• Reflexion: Language Agents with Verbal Reinforcement Learning
• Toolformer: Language Models Can Teach Themselves to Use Tools
• Distributing Accountability, Not Capability: Phase Separation and the LLM Workflow Quadrant in Autonomous AI Agent Architectures
• Training language models to follow instructions with human feedback
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
• Learning to summarize from human feedback
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Dense Passage Retrieval for Open-Domain Question Answering
• Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Continual Lifelong Learning with Neural Networks: A Review
• Value-Decomposition Networks For Cooperative Multi-Agent Learning
• Gradient Episodic Memory for Continual Learning
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Neural Episodic Control
• Overcoming catastrophic forgetting in neural networks
• Learning without Forgetting
• Analogical problem solving
• Scalable Real-Time Recurrent Learning Using Columnar-Constructive Networks
• Reinforcement Learning: An Introduction

Cited by

• SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning
• SkillClaw: Let Skills Evolve Collectively with Agentic Evolver
• Contextual Agentic Memory is a Memo, Not True Memory

Discussions

• 今日読んだ論文 MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memory www.arxiv.org/abs/2601.03192 RAGの弱点を受動的なところだと見て、RAGに保存する情報に意図や結果、強化学習における行動価値のようなものを追加する。記憶の検索も類似度と価値を半分ずつ混ぜてラ [bsky, 0 points, 1 comments]

Related

• MemVerse: Multimodal Memory for Lifelong Learning Agents
• Mem-α: Learning Memory Construction via Reinforcement Learning
• Memento: Fine-tuning LLM Agents without Fine-tuning LLMs
• MemGen: Weaving Generative Latent Memory for Self-Evolving Agents
• Memory Gym: Towards Endless Tasks to Benchmark Memory Capabilities of Agents