Learn from What We HAVE: History-Aware VErifier that Reasons about Past Interactions Online

2025/08/29 by Yishu Li, Li, Yishu, Ying Yuan +7 · 3 citations
Computer Science · Engineering · #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Reinforcement Learning in Robotics #Robot Manipulation and Learning #Robotics (cs.RO)

paper · pdf · doi:10.48550/arxiv.2509.00271

openalex publication_date 2025/08/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

We introduce a novel History-Aware VErifier (HAVE) to disambiguate uncertain scenarios online by leveraging past interactions. Robots frequently encounter visually ambiguous objects whose manipulation outcomes remain uncertain until physically interacted with. While generative models alone could theoretically adapt to such ambiguity, in practice they obtain suboptimal performance in ambiguous cases, even when conditioned on action history. To address this, we propose explicitly decoupling action generation from verification: we use an unconditional diffusion-based generator to propose multiple candidate actions and employ our history-aware verifier to select the most promising action by reasoning about past interactions. Through theoretical analysis, we demonstrate that employing a verifier significantly improves expected action quality. Empirical evaluations and analysis across multiple simulated and real-world environments including articulated objects, multi-modal doors, and uneven object pick-up confirm the effectiveness of our method and improvements over baselines. Our project website is available at: https://liy1shu.github.io/HAVE_CoRL25/

Citations

• All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuning
• Scaling Test-Time Compute Without Verification or RL is Suboptimal
• AdaManip: Adaptive Articulated Object Manipulation Environments and Policy Learning
• DualGuard MPPI: Safe and Performant Optimal Control by Combining Sampling-Based MPC and Hamilton-Jacobi Reachability
• From Foresight to Forethought: VLM-In-the-Loop Policy Steering via Latent Alignment
• Generalizing Safety Beyond Collision-Avoidance via Latent-Space Reachability Analysis
• Inference-Time Policy Steering through Human Interactions
• DELTA: Dense Efficient Long-range 3D Tracking for any video
• Dynamics as Prompts: In-Context Learning for Sim-to-Real System Identifications
• Non-rigid Relative Placement through 3D Dense Diffusion
• Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning
• FlowBotHD: History-Aware Diffuser Handling Ambiguities in Articulated Objects Manipulation
• Robots that Suggest Safe Alternatives
• Interactive Perception for Deformable Object Manipulation
• V-STaR: Training Verifiers for Self-Taught Reasoners
• Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models
• MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations
• What Went Wrong? Closing the Sim-to-Real Gap via Differentiable Causal Discovery
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Scalable Diffusion Models with Transformers
• In-context Reinforcement Learning with Algorithm Distillation
• Classifier-Free Diffusion Guidance
• Prompting Decision Transformer for Few-Shot Policy Generalization
• FlowBot3D: Learning 3D Articulation Flow to Manipulate Articulated Objects
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• Training language models to follow instructions with human feedback
• Training Verifiers to Solve Math Word Problems
• Generative Adversarial Networks
• Diffusion Models Beat GANs on Image Synthesis
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• SAPIEN: A SimulAted Part-based Interactive ENvironment
• TuneNet: One-Shot Residual Tuning for System Identification and Sim-to-Real Robot Task Transfer
• Control Barrier Functions: Theory and Applications
• Deep reinforcement learning from human preferences
• PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space

Cited by

• EVE: A Generator-Verifier System for Generative Policies
• Implicit State Estimation via Video Replanning
• History-Aware Visuomotor Policy Learning via Point Tracking

Related

• Rethinking Progression of Memory State in Robotic Manipulation: An Object-Centric Perspective
• HAMLET: Switch your Vision-Language-Action Model into a History-Aware Policy
• Visually Grounding Language Instruction for History-Dependent Manipulation
• Task Success is not Enough: Investigating the Use of Video-Language Models as Behavior Critics for Catching Undesirable Agent Behaviors
• Instruction-driven history-aware policies for robotic manipulations