TR2M: Transferring Monocular Relative Depth to Metric Depth with Language Descriptions and Dual-Level Scale-Oriented Contrast

2025/06/16 by Beilei Cui, Yiming Huang, Cui, Beilei +5
Computer Science · #Advanced Image and Video Retrieval Techniques #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Handwritten Text Recognition Techniques #Image Processing and 3D Reconstruction

paper · pdf · doi:10.48550/arxiv.2506.13387

openalex publication_date 2025/06/16 · openalex created_date 2025/10/13 · openalex updated_date 2026/08/04

Abstract

This work presents a generalizable framework to transfer relative depth to metric depth. Current monocular depth estimation methods are mainly divided into metric depth estimation (MMDE) and relative depth estimation (MRDE). MMDEs estimate depth in metric scale but are often limited to a specific domain. MRDEs generalize well across different domains, but with uncertain scales which hinders downstream applications. To this end, we aim to build up a framework to solve scale uncertainty and transfer relative depth to metric depth. Previous methods used language as input and estimated two factors for conducting rescaling. Our approach, TR2M, utilizes both text description and image as inputs and estimates two rescale maps to transfer relative depth to metric depth at pixel level. Features from two modalities are fused with a cross-modality attention module to better capture scale information. A strategy is designed to construct and filter confident pseudo metric depth for more comprehensive supervision. We also develop scale-oriented contrastive learning to utilize depth distribution as guidance to enforce the model learning about intrinsic knowledge aligning with the scale distribution. TR2M only exploits a small number of trainable parameters to train on datasets in various domains and experiments not only demonstrate TR2M's great performance in seen datasets but also reveal superior zero-shot capabilities on five unseen datasets. We show the huge potential in pixel-wise transferring relative depth to metric depth with language assistance. (Code is available at: https://github.com/BeileiCui/TR2M)

Citations

• Hybrid-grained Feature Aggregation with Coarse-to-fine Language Guidance for Self-supervised Monocular Depth Estimation
• DepthLM: Metric Depth From Vision Language Models
• UniK3D: Universal Camera Monocular 3D Estimation
• RSA: Resolving Scale Ambiguities in Monocular Depth Estimators through Language Descriptions
• ScaleDepth: Decomposing Metric Depth Estimation into Scale Prediction and Relative Depth Estimation
• Depth Anything V2
• EndoDAC: Efficient Adapting Foundation Model for Self-Supervised Depth Estimation from Any Endoscopic Camera
• WorDepth: Variational Language Prior for Monocular Depth Estimation
• ECoDepth: Effective Conditioning of Diffusion Models for Monocular Depth Estimation
• UniDepth: Universal Monocular Metric Depth Estimation
• Region-aware Distribution Contrast: A Novel Approach to Multi-Task Partially Supervised Learning
• Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations
• Binding Touch to Everything: Learning Unified Multimodal Tactile Representations
• Endo-4DGS: Endoscopic Monocular Scene Reconstruction with 4D Gaussian Splatting
• Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data
• Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation
• Learning to Adapt CLIP for Few-Shot Monocular Depth Estimation
• Improved Baselines with Visual Instruction Tuning
• GasMono: Geometry-Aided Self-Supervised Monocular Depth Estimation for Indoor Scenes
• NDDepth: Normal-Distance Assisted Monocular Depth Estimation
• GEDepth: Ground Embedding for Monocular Depth Estimation
• Self-Supervised Monocular Depth Estimation by Direction-aware Cumulative Convolution Network
• Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Image
• Towards Zero-Shot Scale-Aware Monocular Depth Estimation
• Surgical-VQLA: Transformer with Gated Vision-Language Embedding for Visual Question Localized-Answering in Robotic Surgery
• DINOv2: Learning Robust Visual Features without Supervision
• iDisc: Internal Discretization for Monocular Depth Estimation
• ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth
• PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learning
• Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation
• BinsFormer: Revisiting Adaptive Bins for Monocular Depth Estimation
• Exploring Intra- and Inter-Video Relation for Surgical Semantic Scene Segmentation
• LocalBins: Improving Depth Estimation by Learning Local Distributions
• NeW CRFs: Neural Window Fully-connected CRFs for Monocular Depth Estimation
• BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
• Toward Practical Monocular Indoor Depth Estimation
• PointCLIP: Point Cloud Understanding by CLIP
• Towards Real-Time Monocular Depth Estimation for Robotics: A Survey
• Emerging Properties in Self-Supervised Vision Transformers
• Vision Transformers for Dense Prediction
• Transformer-Based Attention Networks for Continuous Pixel-Wise Prediction
• Learning Transferable Visual Models From Natural Language Supervision
• Exploring Cross-Image Pixel Contrast for Semantic Segmentation
• Contrastive Learning for Label-Efficient Semantic Segmentation
• Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learning
• Dense Contrastive Learning for Self-Supervised Visual Pre-Training
• Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding
• Targeted Adversarial Perturbations for Monocular Depth Prediction
• Improved Baselines with Momentum Contrastive Learning
• A Simple Framework for Contrastive Learning of Visual Representations
• Momentum Contrast for Unsupervised Visual Representation Learning
• Momentum Contrast for Unsupervised Visual Representation Learning
• DIODE: A Dense Indoor and Outdoor DEpth Dataset
• Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer
• Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer
• Unsupervised Depth Completion from Visual Inertial Odometry
• CAM-Convs: Camera-Aware Multi-Scale Convolutions for Single-View Depth
• FastDepth: Fast Monocular Depth Estimation on Embedded Systems
• Dense Depth Estimation in Monocular Endoscopy with Self-supervised Learning Methods
• Deep Ordinal Regression Network for Monocular Depth Estimation
• Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination
• Evaluation of CNN-based Single-Image Depth Estimation Methods
• Unsupervised Monocular Depth Estimation with Left-Right Consistency
• Deeper Depth Prediction with Fully Convolutional Residual Networks
• DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network

Related

• VGLD: Visually-Guided Linguistic Disambiguation for Monocular Depth Scale Recovery
• RSA: Resolving Scale Ambiguities in Monocular Depth Estimators through Language Descriptions
• ScaleDepth: Decomposing Metric Depth Estimation into Scale Prediction and Relative Depth Estimation
• Survey on Monocular Metric Depth Estimation
• DepthLM: Metric Depth From Vision Language Models