Affogato: Open-Vocabulary Affordance Grounding with Automated Data Generation at Scale

2025/06/13 by Jun-Ha Lee, Eunha Park, Lee, Junha +7
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Intelligent Tutoring Systems and Adaptive Learning #Natural Language Processing Techniques #Text Readability and Simplification

paper · pdf · doi:10.48550/arxiv.2506.12009

openalex publication_date 2025/06/13 · openalex created_date 2025/10/11 · openalex updated_date 2026/07/28

Abstract

Affordance grounding aims to localize where to interact with an object, a fundamental capability for embodied agents. Yet progress is bottlenecked by data: manual annotation is prohibitively expensive and confines existing datasets to a narrow set of predefined object and affordance categories. We introduce Affogato, a framework for open-vocabulary affordance grounding centered on Affogato-750K, a large-scale dataset of 750K 3D affordance heatmaps paired with natural language queries. We build it with a fully automated pipeline that orchestrates foundation models to generate them at scale without human labeling. It covers significantly more diverse categories than any existing dataset. For reliable evaluation, we further provide 5K human-verified test pairs. We also present Espresso-3D and Espresso-2D, simple yet effective models with a unified architecture across both modalities. Pretraining on Affogato-750K improves both Espresso and prior methods and yields the largest gains on unseen object and affordance categories, showing that it provides broadly transferable supervision across architectures.

Citations

• Task-Aware 3D Affordance Segmentation via 2D Guidance and Geometric Refinement
• Closed-Loop Transfer for Weakly-supervised Affordance Grounding
• Unlocking 3D Affordance Segmentation with 2D Semantic Knowledge
• RAGNet: Large-scale Reasoning-based Affordance Segmentation Benchmark towards General Grasping
• Weakly-Supervised Affordance Grounding Guided by Part-Level Semantic Priors
• PARTFIELD: Learning 3D Feature Fields for Part Segmentation and Beyond
• Grounding 3D Object Affordance with Language Instructions, Visual Observations and Interactions
• Gemma 3 Technical Report
• MMR: A Large-scale Benchmark Dataset for Multi-target and Multi-granularity Reasoning Segmentation
• 2HandedAfforder: Learning Precise Actionable Bimanual Affordances from Human Videos
• 3D-AffordanceLLM: Harnessing Large Language Models for Open-Vocabulary Affordance Detection in 3D Worlds
• Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentation
• SeqAfford: Sequential 3D Affordance Reasoning via Multimodal Large Language Model
• Functionality understanding and segmentation in 3D scenes
• SAMPart3D: Segment Any Part in 3D Objects
• Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
• In Defense of Lazy Visual Grounding for Open-Vocabulary Semantic Segmentation
• SAM 2: Segment Anything in Images and Videos
• What If We Recaption Billions of Web Images with LLaMA-3?
• What does CLIP know about peeling a banana?
• Beyond the Contact: Discovering Comprehensive Affordance for 3D Objects from Pre-trained 2D Diffusion Models
• Robo-ABC: Affordance Generalization Beyond Categories via Semantic Correspondence for Robot Manipulation
• AffordanceLLM: Grounding Affordance from Vision Language Models
• One-Shot Open Affordance Learning with Foundation Models
• OneFormer3D: One Transformer for Unified Point Cloud Segmentation
• LABELMAKER: Automatic Semantic Label Generation from RGB-D Trajectories
• Open X-Embodiment: Robotic Learning Datasets and RT-X Models
• Multi-label affordance mapping from egocentric vision
• PointLLM: Empowering Large Language Models to Understand Point Clouds
• LISA: Reasoning Segmentation via Large Language Model
• Faster Segment Anything: Towards Lightweight SAM for Mobile Applications
• GRES: Generalized Referring Expression Segmentation
• Understanding 3D Object Interaction from a Single Image
• Affordances from Human Videos as a Versatile Representation for Robotics
• DINOv2: Learning Robust Visual Features without Supervision
• RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understanding
• Grounding 3D Object Affordance from 2D Interactions in Images
• LOCATE: Localize and Transfer Object Parts for Weakly Supervised Affordance Grounding
• Open-Vocabulary Affordance Detection in 3D Point Clouds
• Objaverse: A Universe of Annotated 3D Objects
• OneFormer: One Transformer to Rule Universal Image Segmentation
• Mask3D: Mask Transformer for 3D Semantic Instance Segmentation
• One-Shot Transfer of Affordance Regions? AffCorrs!
• Grounded Affordance from Exocentric View
• Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videos
• Learning Affordance Grounding from Exocentric Images
• PartAfford: Part-level Affordance Discovery from 3D Objects
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Masked-attention Mask Transformer for Universal Image Segmentation
• Per-Pixel Classification is Not All You Need for Semantic Segmentation
• Referring Transformer: A One-step Approach to Multi-task Visual Grounding
• 3D AffordanceNet: A Benchmark for Visual Object Affordance Understanding
• Learning Transferable Visual Models From Natural Language Supervision
• Affordances in Robotic Tasks -- A Survey
• Point-Voxel CNN for Efficient 3D Deep Learning
• Grounded Human-Object Interaction Hotspots from Video
• PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical\n Part-level 3D Object Understanding
• Learning to Act Properly: Predicting and Explaining Affordances from Images
• Modeling Context in Referring Expressions
• V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation
• What do different evaluation metrics tell us about saliency models?

Related

• AffordanceLLM: Grounding Affordance from Vision Language Models
• WorldAfford: Affordance Grounding based on Natural Language Instructions
• Learning 2D Invariant Affordance Knowledge for 3D Affordance Grounding
• OVAL-Prompt: Open-Vocabulary Affordance Localization for Robot Manipulation through LLM Affordance-Grounding
• INTRA: Interaction Relationship-aware Weakly Supervised Affordance Grounding