Visual Autoregressive Modelling for Monocular Depth Estimation

2025/12/27 by Amir El-Ghoussani, El-Ghoussani, Amir, André Kaup +7
Computer Science · #Advanced Image and Video Retrieval Techniques #Advanced Vision and Imaging #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Video Coding and Compression Technologies

paper · doi:10.48550/arxiv.2512.22653

openalex publication_date 2025/12/27 · openalex created_date 2025/12/31 · openalex updated_date 2026/07/28

Abstract

We propose a monocular depth estimation method based on visual autoregressive (VAR) priors, offering an alternative to diffusion-based approaches. Our method adapts a large-scale text-to-image VAR model and introduces a scale-wise conditional upsampling mechanism with classifier-free guidance. Our approach performs inference in ten fixed autoregressive stages, requiring only 74K synthetic samples for fine-tuning, and achieves competitive results. We report state-of-the-art performance in indoor benchmarks under constrained training conditions, and strong performance when applied to outdoor datasets. This work establishes autoregressive priors as a complementary family of geometry-aware generative models for depth estimation, highlighting advantages in data scalability, and adaptability to 3D vision tasks. Code available at "https://github.com/AmirMaEl/VAR-Depth".

Citations

• Marigold: Affordable Adaptation of Diffusion-Based Image Generators for Image Analysis
• Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis
• Switti: Designing Scale-Wise Transformers for Text-to-Image Synthesis
• Fine-Tuning Image-Conditional Diffusion Models is Easier than You Think
• Scalable Autoregressive Image Generation with Mamba
• Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretraining
• Depth Anything V2
• Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation
• Consistency Regularisation for Unsupervised Domain Adaptation in Monocular Depth Estimation
• Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction
• GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image
• Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation
• The Surprising Effectiveness of Diffusion Models for Optical Flow and Monocular Depth Estimation
• DINOv2: Learning Robust Visual Features without Supervision
• Vision-Language Models for Vision Tasks: A Survey
• Vision-Language Models for Vision Tasks: A Survey
• Monocular Depth Estimation using Diffusion Models
• LLaMA: Open and Efficient Foundation Language Models
• VA-DepthNet: A Variational Approach to Single Image Depth Prediction
• Muse: Text-To-Image Generation via Masked Generative Transformers
• Hierarchical Normalization for Robust Monocular Depth Estimation
• LAION-5B: An open large-scale dataset for training next generation image-text models
• Classifier-Free Diffusion Guidance
• Scaling Autoregressive Models for Content-Rich Text-to-Image Generation
• Lightweight Monocular Depth Estimation through Guided Decoding
• Autoregressive Image Generation using Residual Quantization
• MaskGIT: Masked Generative Image Transformer
• High-Resolution Image Synthesis with Latent Diffusion Models
• LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
• Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans
• Vision Transformers for Dense Prediction
• Taming Transformers for High-Resolution Image Synthesis
• Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• DIODE: A Dense Indoor and Outdoor DEpth Dataset
• Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer
• Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer
• Generating Diverse High-Fidelity Images with VQ-VAE-2
• CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features
• GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose
• AdaDepth: Unsupervised Content Congruent Adaptation for Depth Estimation
• Occlusion Aware Unsupervised Learning of Optical Flow
• Neural Discrete Representation Learning
• Unsupervised Learning of Depth and Ego-Motion from Video
• ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
• Unsupervised Monocular Depth Estimation with Left-Right Consistency
• Conditional Image Generation with PixelCNN Decoders
• Deeper Depth Prediction with Fully Convolutional Residual Networks
• Virtual Worlds as Proxy for Multi-Object Tracking Analysis
• Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue
• U-Net: Convolutional Networks for Biomedical Image Segmentation
• Adam: A Method for Stochastic Optimization
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• Vision meets robotics: The KITTI dataset

Related

• DepthART: Monocular Depth Estimation as Autoregressive Refinement Task
• Scalable Autoregressive Monocular Depth Estimation
• Stealing Stable Diffusion Prior for Robust Monocular Depth Estimation
• Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation
• Deep Learning based Monocular Depth Prediction: Datasets, Methods and Applications