LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding
2026/05/26 by Shihao Wang, Shilong Liu, Yuanguo Kuang +10 · 1 voice · 2 citations
Computer Science · #Advanced Neural Network Applications #Bottleneck #Coherence (philosophical gambling strategy) #Decoding methods #Domain Adaptation and Few-Shot Learning #Inference #Multimodal Machine Learning Applications #Scalability #Sequential decoding #Throughput #cs.AI #cs.CV #cs.LG #cs.RO
paper · pdf · doi:10.48550/arxiv.2605.27365
openalex publication_date 2026/05/26 · arxiv published 2026/05/26 · arxiv updated 2026/05/27 · openalex created_date 2026/05/28 · openalex updated_date 2026/07/28
Abstract
Vision-language models (VLMs) commonly formulate visual grounding and detection as a coordinate-token generation problem, serializing each 2D box into multiple 1D tokens that are learned and decoded largely independently. This token-by-token decoding mismatches the coupled structure of box geometry and creates a practical inference bottleneck due to strictly sequential generation. We introduce LocateAnything, a unified generative grounding and detection framework based on Parallel Box Decoding (PBD). By decoding geometric elements such as bounding boxes and points as atomic units in a single step, LocateAnything preserves intra-box geometric coherence and unlocks substantial parallelism. We show that PBD improves both decoding throughput and localization accuracy. We further develop a scalable data engine and curate LocateAnything-Data, a large-scale dataset with more than 138 million training samples, substantially increasing data diversity for high-precision localization. Extensive evaluations show that LocateAnything advances the speed-accuracy frontier, achieving significantly higher decoding throughput while improving high-IoU localization quality across diverse benchmarks. The results highlight the complementary benefits of Parallel Box Decoding and large-scale training data in enabling efficient and precise unified visual grounding and detection.
Citations
- WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference
- MAI-UI Technical Report: Real-World Centric Foundation GUI Agents
- DiffusionVL: Translating Any Autoregressive Models into Diffusion Vision Language Models
- Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speed
- WeDetect: Fast Open-Vocabulary Object Detection as Retrieval
- CoT4Det: A Chain-of-Thought Framework for Perception-Oriented Vision-Language Tasks
- Qwen3-VL Technical Report
- LocateAnything3D: Vision-Language 3D Detection with Chain-of-Sight
- TiDAR: Think in Diffusion, Talk in Autoregression
- Grounding Computer Use Agents on Human Demonstrations
- PixelRefer: A Unified Framework for Spatio-Temporal Object Referring with Arbitrary Granularity
- Beyond Multi-Token Prediction: Pretraining LLMs with Future Summaries
- Detect Anything via Next Point Prediction
- Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMs
- Fast-dLLM v2: Efficient Block-Diffusion LLM
- AdaBlock-dLLM: Semantic-Aware Diffusion LLM Inference via Adaptive Block Size
- Sequential Diffusion Language Models
- ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data
- Kwai Keye-VL 1.5 Technical Report
- InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
- Mobile-Agent-v3: Fundamental Agents for GUI Automation
- Dream 7B: Diffusion Large Language Models
- Ovis2.5 Technical Report
- Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing
- LMM-Det: Make Large Multimodal Models Excel in Object Detection
- Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potential
- GTA1: GUI Test-time Scaling Agent
- PaddleOCR 3.0 Technical Report
- Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videos
- MiMo-VL Technical Report
- Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding
- UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learning
- Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis
- Seed1.5-VL Technical Report
- Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Models
- InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasoners
- Perception-R1: Pioneering Perception Policy with Reinforcement Learning
- ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use
- Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learning
- UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction
- Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning
- Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models
- Referring to Any Person
- OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preference
- Qwen2.5-VL Technical Report
- Large Language Diffusion Models
- LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Models
- Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Models
- DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding
- RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Models
- Harnessing Webpage UIs for Text-Rich Visual Understanding
- DocLayout-YOLO: Enhancing Document Layout Analysis through Diverse Synthetic Data and Global-to-Local Adaptive Perception
- LLaVA-Critic: Learning to Evaluate Multimodal Models
- OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data
- Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
- YesBut: A High-Quality Annotated Multimodal Dataset for evaluating Satire Comprehension capability of Vision-Language Models
- SPARK: Multi-Vision Sensor Perception and Reasoning Benchmark for Large-scale Vision-Language Models
- Building and better understanding vision-language models: insights and future directions
- ColPali: Efficient Document Retrieval with Vision Language Models
- Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs
- MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs
- WeatherQA: Can Multimodal Language Models Reason about Severe Weather?
- WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences
- Multimodal Table Understanding
- Image Textualization: An Automatic Framework for Creating Accurate and Detailed Image Descriptions
- SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models
- MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answering
- Grounding DINO 1.5: Advance the "Edge" of Open-Set Object Detection
- What matters when building vision-language models?
- Better & Faster Large Language Models via Multi-token Prediction
- Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Models
- Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Models
- Advancing LLM Reasoning Generalists with Preference Trees
- Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want
- TableLLM: Enabling Tabular Data Manipulation by LLMs in Real Office Usage Scenarios
- Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset
- Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering
- Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models
- OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement
- ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models
- Orca-Math: Unlocking the potential of SLMs in Grade School Math
- Visually Dehallucinative Instruction Generation: Know What You Don't Know
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
- InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
- DriveLM: Driving with Graph Visual Question Answering
- AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into One
- LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models
- PixelLM: Pixel Reasoning with Large Multimodal Model
- ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts
- How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMs
- Griffon: Spelling out All Object Locations at Any Granularity with Large Language Models
- MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning
- To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuning
- Ferret: Refer and Ground Anything Anywhere at Any Granularity
- UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model
- Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs
- AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZ
- MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models
- MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning
- Far3D: Expanding the Horizon for Surround-view 3D Object Detection
- LISA: Reasoning Segmentation via Large Language Model
- LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding
- VisText: A Benchmark for Semantically Rich Chart Captioning
- Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic
- Kosmos-2: Grounding Multimodal Large Language Models to the World
- RobuT: A Systematic Study of Table QA Robustness Against Human-Annotated Adversarial Perturbations
- ICDAR 2023 Competition on Structured Text Extraction from Visually-Rich Document Images
- UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning
- TheoremQA: A Theorem-driven Question Answering dataset
- Evaluating Object Hallucination in Large Vision-Language Models
- PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering
- ICDAR 2023 Competition on Hierarchical Text Detection and Recognition
- M6Doc: A Large-Scale Multi-Format, Multi-Type, Multi-Layout, Multi-Language, Multi-Annotation Category Dataset for Modern Document Layout Analysis
- CHIC: Corporate Document for Visual question Answering
- Visual Instruction Tuning
- PDFVQA: A New Dataset for Real-World VQA on PDF Documents
- Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection
- Annotated Point Clouds and Images from a Spatial-Semantic Perception Pipeline: Robotized Deconstruction at the Reference Construction Site, Aachen
- SlideVQA: A Dataset for Document Visual Question Answering on Multiple Images
- UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expression
- Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoning
- MapQA: A Dataset for Question Answering on Choropleth Maps
- Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning
- Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
- ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshots
- CLEVR-Math: A Dataset for Compositional Language, Visual and Mathematical Reasoning
- Toward Understanding WordArt: Corner-Guided Transformer for Scene Text Recognition
- MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data
- A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge
- Diffusion-LM Improves Controllable Text Generation
- ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
- Chart-to-Text: A Large-Scale Benchmark for Chart Summarization
- DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection
- OCR-free Document Understanding Transformer
- IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning
- Pix2seq: A Language Modeling Framework for Object Detection
- TextStyleBrush: Transfer of Text Aesthetics from a Single Example
- TextStyleBrush: Transfer of Text Aesthetics From a Single Example
- VSR: A Unified Framework for Document Layout Analysis combining Vision, Semantics and Relations
- Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning
- InfographicVQA
- Learning To Count Everything
- MOTChallenge: A Benchmark for Single-Camera Multiple Target Tracking
- Deformable DETR: Deformable Transformers for End-to-End Object Detection
- End-to-End Object Detection with Transformers
- The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
- Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrieval
- PathVQA: 30000+ Questions for Medical Visual Question Answering
- On the General Value of Evidence, and Bilingual Scene-Text Visual Question Answering
- Connecting Vision and Language with Localized Narratives
- KonIQ-10k: An ecologically valid database for deep learning of blind image quality assessment
- PlotQA: Reasoning over Scientific Plots
- ICDAR 2019 Competition on Large-scale Street View Text with Partial Labeling -- RRC-LSVT
- ICDAR2019 Robust Reading Challenge on Arbitrary-Shaped Text (RRC-ArT)
- ICDAR 2019 Robust Reading Challenge on Reading Chinese Text on Signboard
- SpatialSense: An Adversarially Crowdsourced Benchmark for Spatial Relation Recognition
- LVIS: A Dataset for Large Vocabulary Instance Segmentation
- Scene Text Visual Question Answering
- FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents
- Towards VQA Models That Can Read
- Precise Detection in Densely Packed Scenes
- RAVEN: A Dataset for Relational and Analogical Visual rEasoNing
- TallyQA: Answering Complex Counting Questions
- Learning to Describe Differences Between Pairs of Similar Images
- VizWiz Grand Challenge: Answering Visual Questions from Blind People
- DVQA: Understanding Data Visualizations via Question Answering
- Simple and Effective Multi-Paragraph Reading Comprehension
- Total-Text: A Comprehensive Dataset for Scene Text Detection and Recognition
- FigureQA: An Annotated Figure Dataset for Visual Reasoning
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
- Modeling Context in Referring Expressions
- A Diagram Is Worth A Dozen Images
- COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Images
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
- Exploring Models and Data for Image Question Answering
- Microsoft COCO: Common Objects in Context
- MMRA: A Benchmark for Evaluating Multi-Granularity and Multi-Image Relational Association Capabilities in Large Visual Language Models
- Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning
- WizardLM: Empowering large pre-trained language models to follow complex instructions
Cited by
Discussions
Related