MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding
2021/04/26 by Kamath, Aishwarya, Singh, Mannat, LeCun, Yann +3 · 63 citations
#Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG)
paper · doi:10.48550/arxiv.2104.12763
Abstract
Multi-modal reasoning systems rely on a pre-trained object detector to extract regions of interest from the image. However, this crucial module is typically used as a black box, trained independently of the downstream task and on a fixed vocabulary of objects and attributes. This makes it challenging for such systems to capture the long tail of visual concepts expressed in free form text. In this paper we propose MDETR, an end-to-end modulated detector that detects objects in an image conditioned on a raw text query, like a caption or a question. We use a transformer-based architecture to reason jointly over text and image by fusing the two modalities at an early stage of the model. We pre-train the network on 1.3M text-image pairs, mined from pre-existing multi-modal datasets having explicit alignment between phrases in text and objects in the image. We then fine-tune on several downstream tasks such as phrase grounding, referring expression comprehension and segmentation, achieving state-of-the-art results on popular benchmarks. We also investigate the utility of our model as an object detector on a given label set when fine-tuned in a few-shot setting. We show that our pre-training approach provides a way to handle the long tail of object categories which have very few labelled instances. Our approach can be easily extended for visual question answering, achieving competitive performance on GQA and CLEVR. The code and models are available at https://github.com/ashkamath/mdetr.
Cited by
- What CLIP Knows but Cannot Say: Recovering Negation from Frozen Intermediate Features
- SuperCLIP: CLIP with Simple Classification Supervision
- WeDetect: Fast Open-Vocabulary Object Detection as Retrieval
- No Labels, No Problem: Training Visual Reasoners with Multimodal Verifiers
- START: Spatial and Textual Learning for Chart Understanding
- Omni-Referring Image Segmentation
- E3AD: An Emotion-Aware Vision-Language-Action Model for Human-Centric End-to-End Autonomous Driving
- Think Before You Drive: World Model-Inspired Multimodal Grounding for Autonomous Vehicles
- Making Dialogue Grounding Data Rich: A Three-Tier Data Synthesis Framework for Generalized Referring Expression Comprehension
- Artemis: Structured Visual Reasoning for Perception Policy Learning
- SceneProp: Combining Neural Network and Markov Random Field for Scene-Graph Grounding
- Generalized Medical Phrase Grounding
- Advanced Data Collection Techniques in Cloud Security: A Multi-Modal Deep Learning Autoencoder Approach
- Thinking With Bounding Boxes: Enhancing Spatio-Temporal Video Grounding via Reinforcement Fine-Tuning
- AudioScene: Integrating Object-Event Audio into 3D Scenes
- Percept-WAM: Perception-Enhanced World-Awareness-Action Model for Robust End-to-End Autonomous Driving
- QUILL: An Algorithm-Architecture Co-Design for Cache-Local Deformable Attention
- Backdoor Attacks on Open Vocabulary Object Detectors via Multi-Modal Prompt Tuning
- LIHE: Linguistic Instance-Split Hyperbolic-Euclidean Framework for Generalized Weakly-Supervised Referring Expression Comprehension
- Draft and Refine with Visual Experts
- TubeRMC: Tube-conditioned Reconstruction with Mutual Constraints for Weakly-supervised Spatio-Temporal Video Grounding
- Semantic-Guided Natural Language and Visual Fusion for Cross-Modal Interaction Based on Tiny Object Detection
- SIMS-V: Simulated Instruction-Tuning for Spatial Video Understanding
- Vision-TL-Action: Neuro-Symbolic Trajectory Generation from Visual Observations and Temporal Logic
- GranViT: A Fine-Grained Vision Model With Autoregressive Perception For MLLMs
- MOBIUS: Big-to-Mobile Universal Instance Segmentation via Multi-modal Bottleneck Fusion and Calibrated Decoder Pruning
- CaptionFormer: Unified Segmentation, Tracking, and Captioning for Spatio-Temporal Objects
- Spatial Preference Rewarding for MLLMs Spatial Understanding
- What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging
- Detect Anything via Next Point Prediction
- Image-to-Video Transfer Learning based on Image-Language Foundation Models: A Comprehensive Survey
- Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMs
- Synthetic Object Compositions for Scalable and Accurate Learning in Detection, Segmentation, and Grounding
- A Multimodal Depth-Aware Method For Embodied Reference Understanding
- Referring Expression Comprehension for Small Objects
- CoT Referring: Improving Referring Expression Tasks with Grounded Reasoning
- TimeScope: Towards Task-Oriented Temporal Grounding In Long Videos
- NePTune: A Neuro-Pythonic Framework for Tunable Compositional Reasoning on Vision-Language
- VLM-FO1: Bridging the Gap Between High-Level Reasoning and Fine-Grained Perception in VLMs
- Talk in Pieces, See in Whole: Disentangling and Hierarchical Aggregating Representations for Language-based Object Detection
- VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perception
- Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Grounding
- Improving Generalized Visual Grounding with Instance-aware Joint Learning
- MARS2 2025 Challenge on Multimodal Reasoning: Datasets, Methods, Results, Discussion, and Outlook
- TFANet: Three-Stage Image-Text Feature Alignment Network for Robust Referring Image Segmentation
- Multi-animal tracking in Transition: Comparative Insights into Established and Emerging Methods
- Towards Understanding Visual Grounding in Visual Language Models
- Visual Grounding from Event Cameras
- WAVE-DETR Multi-Modal Visible and Acoustic Real-Life Drone Detector
- Light-Weight Cross-Modal Enhancement Method with Benchmark Construction for UAV-based Open-Vocabulary Object Detection
- PropVG: End-to-End Proposal-Driven Visual Grounding with Multi-Granularity Discrimination
- GENNAV: Polygon Mask Generation for Generalized Referring Navigable Regions
- MRFD: Multi-Region Fusion Decoding with Self-Consistency for Mitigating Hallucinations in LVLMs
- IAG: Input-aware Backdoor Attack on VLM-based Visual Grounding
- DocThinker: Explainable Multimodal Large Language Models with Rule-based Reinforcement Learning for Document Understanding
- Text-guided Visual Prompt DINO for Generic Segmentation
- Textual Inversion for Efficient Adaptation of Open-Vocabulary Object Detectors Without Forgetting
- Latent Expression Generation for Referring Image Segmentation and Grounding
- Referring Remote Sensing Image Segmentation with Cross-view Semantics Interaction Network
- Multimodal Referring Segmentation: A Survey
- Beyond Gloss: A Hand-Centric Framework for Gloss-Free Sign Language Translation
- CAPE: A CLIP-Aware Pointing Ensemble of Complementary Heatmap Cues for Embodied Reference Understanding
- Interpretable Open-Vocabulary Referring Object Detection with Reverse Contrast Attention
Related