Object pop-up: Can we infer 3D objects and their poses from human interactions alone?

2023/06/01 by Ilya A. Petrov, Riccardo Marin, Petrov, Ilya A. +5 · 5 citations
Computer Science · Engineering · #3D Shape Modeling and Analysis #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Face recognition and analysis #Human Pose and Action Recognition

paper · pdf · doi:10.48550/arxiv.2306.00777

openalex publication_date 2023/06/01 · openalex created_date 2023/06/04 · openalex updated_date 2026/07/28

Abstract

The intimate entanglement between objects affordances and human poses is of large interest, among others, for behavioural sciences, cognitive psychology, and Computer Vision communities. In recent years, the latter has developed several object-centric approaches: starting from items, learning pipelines synthesizing human poses and dynamics in a realistic way, satisfying both geometrical and functional expectations. However, the inverse perspective is significantly less explored: Can we infer 3D objects and their poses from human interactions alone? Our investigation follows this direction, showing that a generic 3D human point cloud is enough to pop up an unobserved object, even when the user is just imitating a functionality (e.g., looking through a binocular) without involving a tangible counterpart. We validate our method qualitatively and quantitatively, with synthetic data and sequences acquired for the task, showing applicability for XR/VR. The code is available at https://github.com/ptrvilya/object-popup.

Cited by

• H2OFlow: Grounding Human-Object Affordances with 3D Generative Models and Dense Diffused Flows
• InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation
• Category-Aware 3D Object Composition with Disentangled Texture and Shape Multi-view Diffusion
• ECHO: Ego-Centric modeling of Human-Object interactions
• CoopDiff: Anticipating 3D Human-object Interactions via Contact-consistent Decoupled Diffusion

Related

• LEMON: Learning 3D Human-Object Interaction Relation from 2D Images
• CHORUS: Learning Canonicalized 3D Human-Object Spatial Relations from Unbounded Synthesized Images
• InteractAnything: Zero-shot Human Object Interaction Synthesis via LLM Feedback and Object Affordance Parsing
• Understanding 3D Object Interaction from a Single Image
• PickScan: Object discovery and reconstruction from handheld interactions