Foundation World Models for Agents that Learn, Verify, and Adapt Reliably Beyond Static Environments

2026/02/27 by Florent Delgrange · 1 voice
Computer Science · #Abstraction #Advanced Software Engineering Methodologies #Autonomous agent #Correctness #Formal verification #Foundation (evidence) #Machine Learning and Algorithms #Reinforcement Learning in Robotics #Reinforcement learning #Reliability (semiconductor) #Verifiable secret sharing #cs.AI #cs.LG

paper · pdf · doi:10.65109/wcei7331

arxiv published 2026/02/27 · arxiv updated 2026/02/27 · openalex publication_date 2026/05/24 · openalex created_date 2026/05/25 · openalex updated_date 2026/07/29

Abstract

The next generation of autonomous agents must not only learn efficiently but also act reliably and adapt their behavior in open worlds. Standard approaches typically assume fixed tasks and environments with little or no novelty, which limits world models' ability to support agents that must evolve their policies as conditions change. This paper outlines a vision for foundation world models: persistent, compositional representations that unify reinforcement learning, reactive/program synthesis, and abstraction mechanisms. We propose an agenda built around four components: (i) learnable reward models from specifications to support optimization with clear objectives; (ii) adaptive formal verification integrated throughout learning; (iii) online abstraction calibration to quantify the reliability of the model's predictions; and (iv) test-time synthesis and world-model generation guided by verifiers. Together, these components enable agents to synthesize verifiable programs, derive new policies from a small number of interactions, and maintain correctness while adapting to novelty. The resulting framework positions foundation world models as a substrate for learning, reasoning, and adaptation, laying the groundwork for agents that not only act well but can explain and justify the behavior they adopt.

Citations

• ARM-FM: Automated Reward Machines via Foundation Models for Compositional Reinforcement Learning
• Training Agents Inside of Scalable World Models
• Assessing Adaptive World Models in Machines with Novel Games
• Supermartingale Certificates for Quantitative Omega-regular Verification and Control
• ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems
• PoE-World: Compositional World Modeling with Products of Programmatic Experts
• Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Models
• Fast Adaptation with Behavioral Foundation Models
• Neural Control and Certificate Repair via Runtime Monitoring
• Proto Successor Measure: Representing the Behavior Space of an RL Agent
• Tulu 3: Pushing Frontiers in Open Language Model Post-Training
• Neural Model Checking
• Policy Verification in Stochastic Dynamical Systems Using Logarithmic Neural Certificates
• Formally Verifying Deep Reinforcement Learning Controllers with Lyapunov Barrier Certificates
• Compositional Policy Learning in Stochastic Control Systems with Formal Guarantees
• RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
• Policy Synthesis and Reinforcement Learning for Discounted LTL
• More for Less: Safe Policy Improvement With Stronger Performance Guarantees
• Wasserstein Auto-encoded MDPs: Formal Verification of Efficiently Distilled RL Policies with Many-sided Guarantees
• The Wasserstein Believer: Learning Belief Updates for Partially Observable Environments through Reliable Latent Space Models
• Safe Reinforcement Learning via Probabilistic Logic Shields
• Learning Control Policies for Stochastic Systems with Reach-avoid Guarantees
• Optimistic Linear Support and Successor Features as a Basis for Optimal Policy Transfer
• Training language models to follow instructions with human feedback
• On the (In)Tractability of Reinforcement Learning for LTL Objectives
• A Framework for Transforming Specifications in Reinforcement Learning
• MICo: Improved representations via sampling-based state similarity for Markov decision processes
• Learning One Representation to Optimize All Rewards
• Mastering Atari with Discrete World Models
• Learning Invariant Representations for Reinforcement Learning without Reconstruction
• Probabilistic Guarantees for Safe Deep Reinforcement Learning
• Plannable Approximations to MDP Homomorphisms: Equivariance under Actions
• Control Synthesis from Linear Temporal Logic Specifications using\n Model-Free Reinforcement Learning
• Safe Policy Improvement with an Estimated Baseline Policy
• DeepMDP: Learning Continuous Latent Space Models for Representation Learning
• Safe Reinforcement Learning via Shielding
• Deep reinforcement learning from human preferences
• Safe Policy Improvement by Minimizing Robust Baseline Regret
• Safety-Constrained Reinforcement Learning for MDPs
• Human-level control through deep reinforcement learning
• Metrics for labelled Markov processes
• On the synthesis of a reactive module

Discussions

• Glad to share that my paper, “Foundation World Models for Agents that Learn, Verify, and Adapt Reliably Beyond Static Environments” has been accepted at AAMAS 2026! 📄 Paper: arxiv.org/abs/2602.23997 [bsky, 0 points, 1 comments]

Related

• Foundation Models as World Models: A Foundational Study in Text-Based GridWorlds
• Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning
• General agents contain world models
• Agentic World Modeling: Foundations, Capabilities, Laws, and Beyond
• From Word to World: Can Large Language Models be Implicit Text-based World Models?