Prompt-responsive Object Retrieval with Memory-augmented Student-Teacher Learning

2025/05/04 by Malte Mosbach, Mosbach, Malte, Sven Behnke +1
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Intelligent Tutoring Systems and Adaptive Learning #Machine Learning (cs.LG) #Robotics (cs.RO) #Robotics and Automated Systems

paper · pdf · doi:10.48550/arxiv.2505.02232

openalex publication_date 2025/05/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Building models responsive to input prompts represents a transformative shift in machine learning. This paradigm holds significant potential for robotics problems, such as targeted manipulation amidst clutter. In this work, we present a novel approach to combine promptable foundation models with reinforcement learning (RL), enabling robots to perform dexterous manipulation tasks in a prompt-responsive manner. Existing methods struggle to link high-level commands with fine-grained dexterous control. We address this gap with a memory-augmented student-teacher learning framework. We use the Segment-Anything 2 (SAM 2) model as a perception backbone to infer an object of interest from user prompts. While detections are imperfect, their temporal sequence provides rich information for implicit state estimation by memory-augmented models. Our approach successfully learns prompt-responsive policies, demonstrated in picking objects from cluttered scenes. Videos and code are available at https://memory-student-teacher.github.io

Citations

• SAM 2: Segment Anything in Images and Videos
• Grasp Anything: Combining Teacher-Augmented Policy Gradient Learning with Instance Segmentation to Grasp Arbitrary Objects
• Eureka: Human-Level Reward Design via Coding Large Language Models
• Resilient Legged Local Navigation: Learning to Traverse with Compromised Perception End-to-End
• Lifelong Robot Learning with Human Assisted Language Planners
• RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
• Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation
• DexPBT: Scaling up Dexterous Manipulation for Hand-Arm Systems with Population Based Training
• Micrograph segmentations for DDEVD
• Segment Anything
• GPT-4 Technical Report
• Emergence of Maps in the Memories of Blind Navigation Agents
• RT-1: Robotics Transformer for Real-World Control at Scale
• Rapid Locomotion via Reinforcement Learning
• Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
• Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents
• A System for General In-Hand Object Re-Orientation
• RMA: Rapid Motor Adaptation for Legged Robots
• Language Models are Few-Shot Learners
• Learning by Cheating
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Proximal Policy Optimization Algorithms
• End-to-End Training of Deep Visuomotor Policies
• Real-Time Grasp Detection Using Convolutional Neural Networks
• A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
• Long Short-Term Memory

Related

• Grasp Anything: Combining Teacher-Augmented Policy Gradient Learning with Instance Segmentation to Grasp Arbitrary Objects
• Versatile and Generalizable Manipulation via Goal-Conditioned Reinforcement Learning with Grounded Object Detection
• SOLD: Slot Object-Centric Latent Dynamics Models for Relational Manipulation Learning from Pixels
• MAP-VLA: Memory-Augmented Prompting for Vision-Language-Action Model in Robotic Manipulation
• Acceleration of Actor-Critic Deep Reinforcement Learning for Visual Grasping in Clutter by State Representation Learning Based on Disentanglement of a Raw Input Image