2025/12/13 by Benjamin Beilharz, Thomas S. A. Wallis, Beilharz, Benjamin +1 · 2 voices · 1 citation
Computer Science · Engineering · #3D Shape Modeling and Analysis #Computer Graphics and Visualization Techniques #Generative Adversarial Networks and Image Synthesis #cs.CV #cs.GR
paper · pdf · doi:10.48550/arxiv.2512.12307
v5: Accepted version at Journal of Vision. Note: v2/v3 had a false citation (citation key 16) which was fixed in v4 and was already correct in v1. Code is available here: https://github.com/ag-perception-wallis-lab/MRD
openalex publication_date 2025/12/13 · openalex created_date 2025/12/17 · openalex updated_date 2026/07/28 · arxiv created 2026/07/30 · arxiv updated 2026/07/31
While deep learning methods have achieved impressive success in many vision benchmarks, it remains difficult to understand and explain the representations and decisions of these models. Though vision models are typically trained on 2D inputs, they are often assumed to develop an implicit representation of the underlying 3D scene (for example, showing tolerance to partial occlusion, or the ability to reason about relative depth). Here, we introduce MRD (metamers rendered differentiably), an approach that uses physically based differentiable rendering to probe vision models' implicit understanding of generative 3D scene properties, by finding 3D scene parameters that are physically different but produce the same model activation (i.e. are model metamers). Unlike previous pixel-based methods for evaluating model representations, these reconstruction results are always grounded in physical scene descriptions. This means we can, for example, probe a model's sensitivity to object shape while holding material and lighting constant. As a proof-of-principle, we assess multiple models in their ability to recover scene parameters of geometry (shape) and bidirectional reflectance distribution function (material). The results show high similarity in model activation between target and optimized scenes, with varying visual results. Qualitatively, these reconstructions help investigate the physical scene attributes to which models are sensitive or invariant. MRD holds promise for advancing our understanding of both computer and human vision by enabling analysis of how physical scene parameters drive changes in model responses.