Affordance-Guided Reinforcement Learning via Visual Prompting

2024/07/14 by Olivia Y. Lee, Annie Xie, Lee, Olivia Y. +7 · 14 citations
Computer Science · Psychology · #Affordance #Artificial Intelligence (cs.AI) #Artificial intelligence #Cognitive psychology #Cognitive science #Computer science #FOS: Computer and information sciences #Human–computer interaction #Machine Learning (cs.LG) #Psychology #Reinforcement #Reinforcement Learning in Robotics #Reinforcement learning #Robotics (cs.RO) #Social psychology

paper · pdf · doi:10.48550/arxiv.2407.10341

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2024/07/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Robots equipped with reinforcement learning (RL) have the potential to learn a wide range of skills solely from a reward signal. However, obtaining a robust and dense reward signal for general manipulation tasks remains a challenge. Existing learning-based approaches require significant data, such as human demonstrations of success and failure, to learn task-specific reward functions. Recently, there is also a growing adoption of large multi-modal foundation models for robotics that can perform visual reasoning in physical contexts and generate coarse robot motions for manipulation tasks. Motivated by this range of capability, in this work, we present Keypoint-based Affordance Guidance for Improvements (KAGI), a method leveraging rewards shaped by vision-language models (VLMs) for autonomous RL. State-of-the-art VLMs have demonstrated impressive zero-shot reasoning about affordances through keypoints, and we use these to define dense rewards that guide autonomous robotic learning. On diverse real-world manipulation tasks specified by natural language descriptions, KAGI improves the sample efficiency of autonomous RL and enables successful task completion in 30K online fine-tuning steps. Additionally, we demonstrate the robustness of KAGI to reductions in the number of in-domain demonstrations used for pre-training, reaching similar performance in 45K online fine-tuning steps. Project website: https://sites.google.com/view/affordance-guided-rl

Cited by

• An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challenges
• Affordance-Guided Coarse-to-Fine Exploration for Base Placement in Open-Vocabulary Mobile Manipulation
• Vision-Language-Action Models for Robotics: A Review Towards Real-World Applications
• VLA Model Post-Training via Action-Chunked PPO and Self Behavior Cloning
• RoboSeek: You Need to Interact with Your Objects
• O³Afford: One-Shot 3D Object-to-Object Affordance Grounding for Generalizable Robotic Manipulation
• Manipulate-to-Navigate: Reinforcement Learning with Visual Affordances and Manipulability Priors
• VPN: Visual Prompt Navigation
• DEMONSTRATE: Zero-shot Language to Robotic Control via Multi-task Demonstration Learning
• VLM-TDP: VLM-guided Trajectory-conditioned Diffusion Policy for Robust Long-Horizon Manipulation
• Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trends
• Struct2D: A Perception-Guided Framework for Spatial Reasoning in MLLMs
• APEX: Empowering LLMs with Physics-Based Task Planning for Real-time Insight
• DORA: Object Affordance-Guided Reinforcement Learning for Dexterous Robotic Manipulation
• Robotic Visual Instruction

Related

• End-to-End Affordance Learning for Robotic Manipulation
• Affordance-R1: Reinforcement Learning for Generalizable Affordance Reasoning in Multimodal Large Language Model
• ManipLVM-R1: Reinforcement Learning for Reasoning in Embodied Manipulation with Large Vision-Language Models
• RT-Affordance: Affordances are Versatile Intermediate Representations for Robot Manipulation
• KALIE: Fine-Tuning Vision-Language Models for Open-World Manipulation without Robot Data