pix2gestalt: Amodal Segmentation by Synthesizing Wholes

2024/01/25 by Ege Ozguroglu, Ruoshi Liu, Ozguroglu, Ege +11 · 12 citations
Computer Science · Engineering · #Anatomy and Medical Technology #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Human Pose and Action Recognition #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2401.14398

openalex publication_date 2024/01/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We introduce pix2gestalt, a framework for zero-shot amodal segmentation, which learns to estimate the shape and appearance of whole objects that are only partially visible behind occlusions. By capitalizing on large-scale diffusion models and transferring their representations to this task, we learn a conditional diffusion model for reconstructing whole objects in challenging zero-shot cases, including examples that break natural and physical priors, such as art. As training data, we use a synthetically curated dataset containing occluded objects paired with their whole counterparts. Experiments show that our approach outperforms supervised baselines on established benchmarks. Our model can furthermore be used to significantly improve the performance of existing object recognition and 3D reconstruction methods in the presence of occlusions.

Cited by

• Reasoning-Driven Amodal Completion: Collaborative Agents and Perceptual Evaluation
• Object Reconstruction under Occlusion with Generative Priors and Contact-induced Constraints
• Efficient and Scalable Monocular Human-Object Interaction Motion Reconstruction
• GENA3D: Generative Amodal 3D Modeling by Bridging 2D Priors and 3D Coherence
• CSF-Net: Context-Semantic Fusion Network for Large Mask Inpainting
• Walk through Paintings: Egocentric World Models from Internet Priors
• Extreme Amodal Face Detection
• CObL: Toward Zero-Shot Ordinal Layering without User Prompting
• Beyond the Visible: Benchmarking Occlusion Perception in Multimodal Large Language Models
• Single Point, Full Mask: Velocity-Guided Level Set Evolution for End-to-End Amodal Segmentation
• Shape Distribution Matters: Shape-specific Mixture-of-Experts for Amodal Segmentation under Diverse Occlusions
• Contact-Aware Amodal Completion for Human-Object Interaction via Multi-Regional Inpainting

Related

• Sequential Amodal Segmentation via Cumulative Occlusion Learning
• HoloPart: Generative 3D Part Amodal Segmentation
• Using Diffusion Priors for Video Amodal Segmentation
• Amodal3R: Amodal 3D Reconstruction from Occluded 2D Images
• Amodal Ground Truth and Completion in the Wild