Truly Self-Improving Agents Require Intrinsic Metacognitive Learning

2025/06/05 by Tennison Liu, Mihaela van der Schaar, Liu, Tennison +1 · 6 citations
Computer Science · Neuroscience · Psychology · #Artificial Intelligence (cs.AI) #Embodied and Extended Cognition #FOS: Computer and information sciences #Face (sociological concept) #Key (lock) #Metacognition #Perception #Reinforcement Learning in Robotics #Scale (ratio) #Scope (computer science) #Social Robot Interaction and HRI

paper · pdf · doi:10.48550/arxiv.2506.05109

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/06/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Self-improving agents aim to continuously acquire new capabilities with minimal supervision. However, current approaches face two key limitations: their self-improvement processes are often rigid, fail to generalize across tasks domains, and struggle to scale with increasing agent capabilities. We argue that effective self-improvement requires intrinsic metacognitive learning, defined as an agent's intrinsic ability to actively evaluate, reflect on, and adapt its own learning processes. Drawing inspiration from human metacognition, we introduce a formal framework comprising three components: metacognitive knowledge (self-assessment of capabilities, tasks, and learning strategies), metacognitive planning (deciding what and how to learn), and metacognitive evaluation (reflecting on learning experiences to improve future learning). Analyzing existing self-improving agents, we find they rely predominantly on extrinsic metacognitive mechanisms, which are fixed, human-designed loops that limit scalability and adaptability. Examining each component, we contend that many ingredients for intrinsic metacognition are already present. Finally, we explore how to optimally distribute metacognitive responsibilities between humans and agents, and robustly evaluate and improve intrinsic metacognitive learning, key challenges that must be addressed to enable truly sustained, generalized, and aligned self-improvement.

Citations

• Imagining and building wise machines: The centrality of AI metacognition
• WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning
• The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery
• Metacognitive AI: Framework and the Case for a Neurosymbolic Approach
• DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning
• Open-Endedness is Essential for Artificial Superhuman Intelligence
• OMNI-EPIC: Open-endedness via Models of human Notions of Interestingness with Environments Programmed in Code
• Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solving
• AlphaMath Almost Zero: Process Supervision without Process
• A Survey on Self-Evolution of Large Language Models
• OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
• Teaching Large Language Models to Reason with Reinforcement Learning
• Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
• Empowering Large Language Model Agents through Action Learning
• AgentOhana: Design Unified Data and Training Pipeline for Effective Agent Learning
• Genie: Generative Interactive Environments
• How to Train Data-Efficient LLMs
• OS-Copilot: Towards Generalist Computer Agents with Self-Improvement
• The Metacognitive Demands and Opportunities of Generative AI
• ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agent
• Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models
• Failures Pave the Way: Enhancing Large Language Models through Tuning-free Rule Accumulation
• Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning
• Eureka: Human-Level Reward Design via Coding Large Language Models
• Quality-Diversity through AI Feedback
• Learning Interactive Real-World Simulators
• The Rise and Potential of Large Language Model Based Agents: A Survey
• Large Language Models as Optimizers
• Cognitive Architectures for Language Agents
• From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning
• ExpeL: LLM Agents Are Experiential Learners
• MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework
• A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis
• Large Language Models Are Semi-Parametric Reinforcement Learning Agents
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• SelfEvolve: A Code Evolution Framework via Large Language Models
• OMNI: Open-endedness via Models of human Notions of Interestingness
• Voyager: An Open-Ended Embodied Agent with Large Language Models
• Reasoning with Language Model is Planning with World Model
• RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment
• Teaching Large Language Models to Self-Debug
• Generative Agents: Interactive Simulacra of Human Behavior
• Self-Refine: Iterative Refinement with Self-Feedback
• Reflexion: Language Agents with Verbal Reinforcement Learning
• Toolformer: Language Models Can Teach Themselves to Use Tools
• Human-Timescale Adaptation in an Open-Ended Task Space
• Emotional Intelligence of Large Language Models
• Self-Instruct: Aligning Language Models with Self-Generated Instructions
• Affective Coherence Monitoring for Transformer-Based Language Models
• Scaling Instruction-Finetuned Language Models
• Distributing Accountability, Not Capability: Phase Separation and the LLM Workflow Quadrant in Autonomous AI Agent Architectures
• Language Models (Mostly) Know What They Know
• PaLM: Scaling Language Modeling with Pathways
• Training language models to follow instructions with human feedback
• WebGPT: Browser-assisted question-answering with human feedback
• Training Verifiers to Solve Math Word Problems
• On the Dangers of Stochastic Parrots
• Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design
• Prioritized Level Replay
• Measuring Massive Multitask Language Understanding
• Language Models are Few-Shot Learners
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• The Role of Metacognitive Knowledge in Learning, Teaching, and Assessing
• Open-Ended Artificial Evolution
• Principles of metareasoning
• EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizers
• I Think, Therefore I am: Benchmarking Awareness of Large Language Models Using AwareBench
• Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry.

Cited by

• When AI Agents Compete for Jobs: Strategic Capabilities and Economic Dynamics of AI Labour Markets
• Verifying Meta-Awareness via Predictive Rewards in Reasoning Models
• Galaxy: A Cognition-Centered Framework for Proactive, Privacy-Preserving, and Self-Evolving LLM Agents
• From Skills to Talent: Organising Heterogeneous Agents as a Real-World Company
• Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learning
• Metacognition in LLMs: Foundations, Progress, and Opportunities

Related

• Metacognition is all you need? Using Introspection in Generative Agents to Improve Goal-directed Behavior
• Competence-Aware AI Agents with Metacognition for Unknown Situations and Environments (MUSE)
• Metacognition
• What are the mechanisms underlying metacognitive learning?
• Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs