Unified Category-Level Object Detection and Pose Estimation from RGB Images using 3D Prototypes

2025/08/04 by Fischer, Tom, Zhang, Xiaojie, Ilg, Eddy
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2508.02157

Abstract

Recognizing objects in images is a fundamental problem in computer vision. Although detecting objects in 2D images is common, many applications require determining their pose in 3D space. Traditional category-level methods rely on RGB-D inputs, which may not always be available, or employ two-stage approaches that use separate models and representations for detection and pose estimation. For the first time, we introduce a unified model that integrates detection and pose estimation into a single framework for RGB images by leveraging neural mesh models with learned features and multi-model RANSAC. Our approach achieves state-of-the-art results for RGB category-level pose estimation on REAL275, improving on the current state-of-the-art by 22.9% averaged across all scale-agnostic metrics. Finally, we demonstrate that our unified method exhibits greater robustness compared to single-stage baselines. Our code and models are available at https://github.com/Fischer-Tom/unified-detection-and-pose-estimation.

Citations

• LaPose: Laplacian Mixture Shape Modeling for RGB-Based Category-Level Object Pose Estimation
• iNeMo: Incremental Neural Mesh Models for Robust Class-Incremental Learning
• Deep Learning-Based Object Pose Estimation: A Comprehensive Survey
• Instance-Adaptive and Geometric-Aware Keypoint Learning for Category-Level 6D Object Pose Estimation
• SecondPose: SE(3)-Consistent Dual-Stream Feature Fusion for Category-Level Pose Estimation
• RGB-based Category-level Object Pose Estimation via Decoupled Metric Scale Recovery
• Deep Fusion Transformer Network with Weighted Vector-Wise Keypoints Voting for Robust 6D Object Pose Estimation
• GenPose: Generative Category-level Object Pose Estimation via Diffusion Models
• DINOv2: Learning Robust Visual Features without Supervision
• SOCS: Semantically-aware Object Coordinate Space for Category-Level 6D Object Pose Estimation under Large Shape Variations
• COPE: End-to-end trainable Constant Runtime Object Pose Estimation
• Category-Level 6D Object Pose and Size Estimation using Self-Supervised Deep Prior Deformation Networks
• AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition
• Object Level Depth Reconstruction for Category Level 6D Object Pose Estimation From Monocular RGB Image
• ZebraPose: Coarse to Fine Surface Encoding for 6DoF Object Pose Estimation
• CenterSnap: Single-Shot Multi-Object 3D Shape Reconstruction and Categorical 6D Pose and Size Estimation
• Deep ViT Features as Dense Visual Descriptors
• CaTGrasp: Learning Category-Level Task-Relevant Grasping in Clutter from Simulation
• Single-Stage Keypoint-Based Category-Level Object Pose Estimation from an RGB Image
• DualPoseNet: Category-level 6D Object Pose and Size Estimation Using Dual Pose Network with Refined Learning of Pose Consistency
• NeMo: Neural Mesh Models of Contrastive Features for Robust 3D Pose Estimation
• EfficientPose: An efficient, accurate and scalable end-to-end 6D multi object pose estimation approach
• CoKe: Localized Contrastive Learning for Robust Keypoint Detection
• CosyPose: Consistent multi-view multi-object 6D pose estimation
• Shape Prior Deformation for Categorical 6D Object Pose and Size Estimation
• End-to-End Object Detection with Transformers
• Single-Stage 6D Object Pose Estimation
• Accelerating 3D Deep Learning with PyTorch3D
• PVN3D: A Deep Point-wise 3D Keypoints Voting Network for 6DoF Pose Estimation
• Progressive-X: Efficient, Anytime, Multi-Model Fitting Algorithm
• DPOD: 6D Pose Object Detector and Refiner
• Normalized Object Coordinate Space for Category-Level 6D Object Pose and\n Size Estimation
• Segmentation-driven 6D Object Pose Estimation
• Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations
• DeepIM: Deep Iterative Matching for 6D Pose Estimation
• Real-Time Seamless Single Shot 6D Object Pose Prediction
• PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes
• Graph-Cut RANSAC
• Mask R-CNN
• Real-time Perception meets Reactive Motion Generation
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal\n Networks
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Random sample consensus
• Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey and Benchmark
• ShAPO: Implicit Representations for Multi-Object Shape, Appearance, and Pose Optimization

Related

• RCGNet: RGB-based Category-Level 6D Object Pose Estimation with Geometric Guidance
• Inferring 3D Object Pose in RGB-D Images
• RGB-based Category-level Object Pose Estimation via Decoupled Metric Scale Recovery
• Single-Stage Keypoint-Based Category-Level Object Pose Estimation from an RGB Image
• 3D Object Recognition By Corresponding and Quantizing Neural 3D Scene Representations