2025/03/18 by Vihaan Misra, Misra, Vihaan, Peter Schaldenbrand +3 · 2 citations
Computer Science · Engineering · #3D Shape Modeling and Analysis #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Handwritten Text Recognition Techniques #Image Processing and 3D Reconstruction
paper · pdf · doi:10.48550/arxiv.2503.14720
openalex publication_date 2025/03/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/31
We present ShapeShift, a method for arranging rigid objects into configurations that visually convey semantic concepts specified by natural language. While pretrained diffusion models provide powerful semantic guidance, such as Score Distillation Sampling, enforcing physical validity poses a fundamental challenge. Naive overlap resolution disrupts semantic structure -- separating overlapping shapes along geometrically optimal directions (minimum translation vectors) often destroys the very arrangements that make concepts recognizable. Our intuition is that diffusion model features encode not just what a concept looks like, but its geometric, directional structure -- how it is oriented and shaped -- which we leverage to make overlap resolution semantically aware. We introduce a deformable boundary represented as a phase field that expands anisotropically, guided by intermediate features from the diffusion model, creating space along semantically coherent directions. Experiments demonstrate that ShapeShift, by coupling semantic guidance and feasibility constraint resolution, produces arrangements achieving both semantic clarity and overlap-free validity, significantly outperforming baselines that treat these objectives independently.