Simultaneous Detection and Segmentation

2014/07/07 by Bharath Hariharan, Pablo Arbeláez, Hariharan, Bharath +5 · 19 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #cs.CV

paper · pdf · doi:10.48550/arxiv.1407.1808

To appear in the European Conference on Computer Vision (ECCV), 2014

arxiv created 2014/07/07 · openalex publication_date 2014/07/07 · arxiv updated 2014/07/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We aim to detect all instances of a category in an image and, for each instance, mark the pixels that belong to it. We call this task Simultaneous Detection and Segmentation (SDS). Unlike classical bounding box detection, SDS requires a segmentation and not just a box. Unlike classical semantic segmentation, we require individual object instances. We build on recent work that uses convolutional neural networks to classify category-independent region proposals (R-CNN [16]), introducing a novel architecture tailored for SDS. We then use category-specific, top- down figure-ground predictions to refine our bottom-up proposals. We show a 7 point boost (16% relative) over our baselines on SDS, a 5 point boost (10% relative) over state-of-the-art on semantic segmentation, and state-of-the-art performance in object detection. Finally, we provide diagnostic tools that unpack performance and provide directions for future work.

Citations

• The Pascal Visual Object Classes (VOC) Challenge
• Backpropagation Applied to Handwritten Zip Code Recognition
• Distinctive Image Features from Scale-Invariant Keypoints
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition

Cited by

• What Holds Back Open-Vocabulary Segmentation?
• Rethinking Query-based Transformer for Continual Image Segmentation
• What Demands Attention in Urban Street Scenes? From Scene Understanding towards Road Safety: A Survey of Vision-driven Datasets and Studies
• Inter- and Intra-image Refinement for Few Shot Segmentation
• Unlocking Constraints: Source-Free Occlusion-Aware Seamless Segmentation
• IPFormer: Visual 3D Panoptic Scene Completion with Context-Adaptive Instance Proposals
• Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
• Efficient piecewise training of deep structured models for semantic segmentation
• Unlocking the Power of SAM 2 for Few-Shot Segmentation
• Simple Does It: Weakly Supervised Instance and Semantic Segmentation
• Applications of the Streaming Networks
• Multiple-Human Parsing in the Wild
• Category-Specific Object Reconstruction from a Single Image
• Semantic Instance Segmentation via Deep Metric Learning
• Instance Segmentation by Deep Coloring
• Straight to Shapes: Real-time Detection of Encoded Shapes
• Streaming Networks: Enable A Robust Classification of Noise-Corrupted Images
• Real-World Image Datasets for Federated Learning
• Image Labeling with Markov Random Fields and Conditional Random Fields

Related

• Rich feature hierarchies for accurate object detection and semantic segmentation
• Bridging Category-level and Instance-level Semantic Image Segmentation
• Object Detection based on Region Decomposition and Assembly
• R-CNN minus R
• Learning to decompose for object detection and instance segmentation