Repurposing Marigold for Zero-Shot Metric Depth Estimation via Defocus Blur Cues

2025/05/23 by Talegaonkar, Chinmay, Suresh, Nikhil Gandudi, Novack, Zachary +3
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2505.17358

Abstract

Recent monocular metric depth estimation (MMDE) methods have made notable progress towards zero-shot generalization. However, they still exhibit a significant performance drop on out-of-distribution datasets. We address this limitation by injecting defocus blur cues at inference time into Marigold, a pre-trained diffusion model for zero-shot, scale-invariant monocular depth estimation (MDE). Our method effectively turns Marigold into a metric depth predictor in a training-free manner. To incorporate defocus cues, we capture two images with a small and a large aperture from the same viewpoint. To recover metric depth, we then optimize the metric depth scaling parameters and the noise latents of Marigold at inference time using gradients from a loss function based on the defocus-blur image formation model. We compare our method against existing state-of-the-art zero-shot MMDE methods on a self-collected real dataset, showing quantitative and qualitative improvements.

Citations

• Survey on Monocular Metric Depth Estimation
• FoundationStereo: Zero-Shot Stereo Matching
• DepthMaster: Taming Diffusion Models for Monocular Depth Estimation
• Marigold-DC: Zero-Shot Monocular Depth Completion with Guided Diffusion
• Depth Pro: Sharp Monocular Metric Depth in Less Than a Second
• A Survey on Diffusion Models for Inverse Problems
• Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction
• Depth Anything V2
• ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimization
• DITTO-2: Distilled Diffusion Inference-Time T-Optimization for Music Generation
• TiNO-Edit: Timestep and Noise Optimization for Robust Diffusion-Based Image Editing
• UniDepth: Universal Monocular Metric Depth Estimation
• Osmosis: RGBD Diffusion Prior for Underwater Image Restoration
• GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image
• Passive Snapshot Coded Aperture Dual-Pixel RGB-D Imaging
• DITTO: Diffusion Inference-Time T-Optimization for Music Generation
• Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data
• Zero-Shot Metric Depth with a Field-of-View Conditioned Diffusion Model
• Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation
• Rapid Network Adaptation: Learning to Adapt Neural Networks Using Test-Time Feedback
• Dr.Bokeh: DiffeRentiable Occlusion-aware Bokeh Rendering
• MiDaS v3.1 -- A Model Zoo for Robust Monocular Relative Depth Estimation
• Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Image
• Solving Inverse Problems with Latent Diffusion Models via Hard Data Consistency
• Solving Linear Inverse Problems Provably via Posterior Sampling with Latent Diffusion Models
• Norm-guided latent space exploration for text-to-image generation
• The Surprising Effectiveness of Diffusion Models for Optical Flow and Monocular Depth Estimation
• End-to-End Diffusion Latent Optimization Improves Classifier Guidance
• Fully Self-Supervised Depth Estimation from Defocus Clue
• DiffusionDepth: Diffusion Denoising Approach for Monocular Depth Estimation
• ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth
• Diffusion Posterior Sampling for General Noisy Inverse Problems
• An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion
• Learning Depth from Focus in the Wild
• Improving Diffusion Models for Inverse Problems using Manifold Constraints
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• Come-Closer-Diffuse-Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contraction
• Defocus Map Estimation and Deblurring from a Single Dual-Pixel Image
• Bridging Unsupervised and Supervised Depth from Focus via All-in-Focus Supervision
• Dual Pixel Exploration: Simultaneous Depth Estimation and Image Restoration
• Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding
• Focus on defocus: bridging the synthetic to real domain gap for depth estimation
• Defocus Deblurring Using Dual-Pixel Data
• Joint Image and Depth Estimation with Mask-Based Lensless Cameras
• Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer
• Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer
• Single Image Depth Estimation Trained via Depth from Defocus Cues
• Learning Single Camera Depth Estimation using Dual-Pixels
• Deep Depth from Defocus: how can defocus blur improve 3D estimation using dense neural networks?
• Aperture Supervision for Monocular Depth Estimation
• DiffuserCam: Lensless Single-exposure 3D Imaging
• Deep Depth From Focus
• Central limit theorems for Gaussian polytopes
• Depth from defocus: A spatial domain approach
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Related

• Marigold-DC: Zero-Shot Monocular Depth Completion with Guided Diffusion
• Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation
• BetterDepth: Plug-and-Play Diffusion Refiner for Zero-Shot Monocular Depth Estimation
• Zero-Shot Metric Depth with a Field-of-View Conditioned Diffusion Model
• SharpDepth: Sharpening Metric Depth Predictions Using Diffusion Distillation