ColonCrafter: A Depth Estimation Model for Colonoscopy Videos Using Diffusion Priors

2025/09/16 by Romain Hardy, Tyler M. Berzin, Hardy, Romain +3
Computer Science · Medicine · #Advanced Image and Video Retrieval Techniques #Advanced Vision and Imaging #Artificial Intelligence (cs.AI) #Colorectal Cancer Screening and Detection #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2509.13525

openalex publication_date 2025/09/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Three-dimensional (3D) scene understanding in colonoscopy presents significant challenges that necessitate automated methods for accurate depth estimation. However, existing depth estimation models for endoscopy struggle with temporal consistency across video sequences, limiting their applicability for 3D reconstruction. We present ColonCrafter, a diffusion-based depth estimation model that generates temporally consistent depth maps from monocular colonoscopy videos. Our approach learns robust geometric priors from synthetic colonoscopy sequences to generate temporally consistent depth maps. We also introduce a style transfer technique that preserves geometric structure while adapting real clinical videos to match our synthetic training domain. ColonCrafter achieves state-of-the-art zero-shot performance on the C3VD dataset, outperforming both general-purpose and endoscopy-specific approaches. Although full trajectory 3D reconstruction remains a challenge, we demonstrate clinically relevant applications of ColonCrafter, including 3D point cloud generation and surface coverage assessment.

Citations

• C3VDv2 -- Colonoscopy 3D video dataset with enhanced realism
• PPS-Ctrl: Controllable Sim-to-Real Translation for Colonoscopy Depth Estimation
• GeometryCrafter: Consistent Geometry Estimation for Open-world Videos with Diffusion Priors
• Advancing Dense Endoscopic Reconstruction with Gaussian Splatting-driven Surface Normal-aware Tracking and Mapping
• EndoOmni: Zero-Shot Cross-Dataset Depth Estimation in Endoscopy by Robust Self-Learning from Noisy Labels
• DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videos
• Structure-preserving Image Translation for Depth Estimation in Colonoscopy Video
• Depth Anything V2
• EndoDAC: Efficient Adapting Foundation Model for Self-Supervised Depth Estimation from Any Endoscopic Camera
• SpatialTracker: Tracking Any 2D Pixels in 3D Space
• Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data
• Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfer
• WS-SfMLearner: Self-supervised Monocular Depth and Ego-motion Estimation on Surgical Videos with Unknown Camera Parameters
• StyleDiffusion: Controllable Disentangled Style Transfer via Diffusion Models
• Inversion-Based Style Transfer with Diffusion Models
• Elucidating the Design Space of Diffusion-Based Generative Models
• Automatic Generation of Synthetic Colonoscopy Videos for Domain Randomization
• FoldIt: Haustral Folds Detection and Segmentation in Colonoscopy Videos
• LoRA: Low-Rank Adaptation of Large Language Models
• Accelerating 3D Deep Learning with PyTorch3D
• SLAM Endoscopy enhanced by adversarial depth prediction
• Deep Learning and Conditional Random Fields-based Depth Estimation and\n Topographical Reconstruction from Conventional Endoscopy

Related

• ColDE: A Depth Estimation Framework for Colonoscopy Reconstruction
• ToDER: Towards Colonoscopy Depth Estimation and Reconstruction with Geometry Constraint Adaptation
• ColonAdapter: Geometry Estimation Through Foundation Model Adaptation for Colonoscopy
• C3VDv2 -- Colonoscopy 3D video dataset with enhanced realism
• Multi-task learning with cross-task consistency for improved depth estimation in colonoscopy