More Than Generation: Unifying Generation and Depth Estimation via Text-to-Image Diffusion Models

2025/10/27 by Lin, Hongkai, Liang, Dingkang, Du, Mingyang +2 · 1 citation
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2510.23574

Abstract

Generative depth estimation methods leverage the rich visual priors stored in pre-trained text-to-image diffusion models, demonstrating astonishing zero-shot capability. However, parameter updates during training lead to catastrophic degradation in the image generation capability of the pre-trained model. We introduce MERGE, a unified model for image generation and depth estimation, starting from a fixed pre-trained text-to-image model. MERGE demonstrates that the pre-trained text-to-image model can do more than image generation, but also expand to depth estimation effortlessly. Specifically, MERGE introduces a play-and-plug framework that enables seamless switching between image generation and depth estimation modes through simple and pluggable converters. Meanwhile, we propose a Group Reuse Mechanism to encourage parameter reuse and improve the utilization of the additional learnable parameters. MERGE unleashes the powerful depth estimation capability of the pre-trained text-to-image model while preserving its original image generation ability. Compared to other unified models for image generation and depth estimation, MERGE achieves state-of-the-art performance across multiple depth estimation benchmarks. The code will be made available at https://github.com/H-EmbodVis/MERGE

Citations

• A Unified Image-Dense Annotation Generation Model for Underwater Scenes
• One Diffusion to Generate Them All
• A Simple Approach to Unifying Diffusion-based Conditional Generation
• Parameter-Efficient Fine-Tuning in Spectral Domain for Point Cloud Learning
• Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction
• DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videos
• IGEV++: Iterative Multi-range Geometry Encoding Volumes for Stereo Matching
• BetterDepth: Plug-and-Play Diffusion Refiner for Zero-Shot Monocular Depth Estimation
• SOOD++: Leveraging Unlabeled Data to Boost Oriented Object Detection
• StableNormal: Reducing Diffusion Variance for Stable and Sharp Normal
• Depth Anything V2
• DepthFM: Fast Monocular Depth Estimation with Flow Matching
• GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image
• PixArt-Σ: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation
• Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
• DoRA: Weight-Decomposed Low-Rank Adaptation
• Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data
• A Simple Latent Diffusion Approach for Panoptic Segmentation and Mask Inpainting
• Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation
• Wonder3D: Single Image to 3D using Cross-Domain Diffusion
• JointNet: Extending Text-to-Image Diffusion for Dense Distribution Modeling
• SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
• Visual Instruction Tuning
• Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models
• Unleashing Text-to-Image Diffusion Models for Visual Perception
• Open-vocabulary Object Segmentation with Diffusion Models
• Scalable Diffusion Models with Transformers
• Hierarchical Normalization for Robust Monocular Depth Estimation
• Flow Matching for Generative Modeling
• Accurate and Efficient Stereo Matching via Attention Concatenation Volume
• Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
• Vector Quantized Diffusion Model for Text-to-Image Synthesis
• Vision Transformers for Dense Prediction
• Learning Transferable Visual Models From Natural Language Supervision
• Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding
• Denoising Diffusion Probabilistic Models
• Virtual KITTI 2
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• DIODE: A Dense Indoor and Outdoor DEpth Dataset
• Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer
• Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer
• Evaluation of CNN-based Single-Image Depth Estimation Methods
• Neural Discrete Representation Learning
• ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
• Auto-Encoding Variational Bayes

Cited by

• GaussianDWM: 3D Gaussian Driving World Model for Unified Scene Understanding and Multi-Modal Generation

Related

• Monocular Depth Estimation using Diffusion Models
• DepthMaster: Taming Diffusion Models for Monocular Depth Estimation
• Iris: Integrating Language into Diffusion-based Monocular Depth Estimation
• DepthFM: Fast Monocular Depth Estimation with Flow Matching
• DiffuseHigh: Training-free Progressive High-Resolution Image Synthesis through Structure Guidance