Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Grounding

2025/09/18 by Yang, Zaiquan, Yuhao Liu, Gerhard Hancke +4 · 1 citation
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Human Pose and Action Recognition #Multimodal Machine Learning Applications #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2509.15178

openalex publication_date 2025/09/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Spatio-temporal video grounding (STVG) aims at localizing the spatio-temporal tube of a video, as specified by the input text query. In this paper, we utilize multimodal large language models (MLLMs) to explore a zero-shot solution in STVG. We reveal two key insights about MLLMs: (1) MLLMs tend to dynamically assign special tokens, referred to as grounding tokens, for grounding the text query; and (2) MLLMs often suffer from suboptimal grounding due to the inability to fully integrate the cues in the text query (e.g., attributes, actions) for inference. Based on these insights, we propose a MLLM-based zero-shot framework for STVG, which includes novel decomposed spatio-temporal highlighting (DSTH) and temporal-augmented assembling (TAS) strategies to unleash the reasoning ability of MLLMs. The DSTH strategy first decouples the original query into attribute and action sub-queries for inquiring the existence of the target both spatially and temporally. It then uses a novel logit-guided re-attention (LRA) module to learn latent variables as spatial and temporal prompts, by regularizing token predictions for each sub-query. These prompts highlight attribute and action cues, respectively, directing the model's attention to reliable spatial and temporal related visual regions. In addition, as the spatial grounding by the attribute sub-query should be temporally consistent, we introduce the TAS strategy to assemble the predictions using the original video frames and the temporal-augmented frames as inputs to help improve temporal consistency. We evaluate our method on various MLLMs, and show that it outperforms SOTA methods on three common STVG benchmarks. The code will be available at https://github.com/zaiquanyang/LLaVA_Next_STVG.

Citations

• Decoupled Entropy Minimization
• FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Driving
• Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding
• See What You Are Told: Visual Attention Sink in Large Multimodal Models
• STPro: Spatial and Temporal Progressive Learning for Weakly Supervised Spatio-Temporal Grounding
• MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs
• Contextual Self-paced Learning for Weakly Supervised Spatio-Temporal Video Grounding
• LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understanding
• Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens
• Historical Test-time Prompt Tuning for Vision Foundation Models
• Emergent Visual Grounding in Large Multimodal Models Without Grounding Supervision
• Boosting Weakly-Supervised Referring Image Segmentation via Progressive Comprehension
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• LLaVA-OneVision: Easy Visual Task Transfer
• SAM 2: Segment Anything in Images and Videos
• ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models
• OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding
• ShareGPT4Video: Improving Video Understanding and Generation with Better Captions
• Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Models
• Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Models
• C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersion
• Context-Guided Spatio-Temporal Video Grounding
• Video-GroundingDINO: Towards Open-Vocabulary Spatio-Temporal Video Grounding
• GroundVLP: Harnessing Zero-shot Visual Grounding from Vision-Language Pre-training and Open-Vocabulary Object Detection
• InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
• Osprey: Pixel Understanding with Visual Instruction Tuning
• Zero-shot Referring Expression Comprehension via Structural Similarity Between Images and Captions
• PG-Video-LLaVA: Pixel Grounding Large Video-Language Models
• Video-LLaVA: Learning United Visual Representation by Alignment Before Projection
• GLaMM: Pixel Grounding Large Multimodal Model
• Ferret: Refer and Ground Anything Anywhere at Any Granularity
• Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models
• Efficient Streaming Language Models with Attention Sinks
• Vision Transformers Need Registers
• Investigating the Catastrophic Forgetting in Multimodal Large Language Models
• Can I Trust Your Answer? Visually Grounded Video Question Answering
• Referring Image Segmentation Using Text Supervision
• LISA: Reasoning Segmentation via Large Language Model
• Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic
• GRES: Generalized Referring Expression Segmentation
• Visual Instruction Tuning
• What does CLIP know about a red circle? Visual prompt engineering for VLMs
• GPT-4 Technical Report
• Annotated Point Clouds and Images from a Spatial-Semantic Perception Pipeline: Robotized Deconstruction at the Reference Construction Site, Aachen
• Embracing Consistency: A One-Stage Approach for Spatio-Temporal Video Grounding
• Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models
• Learning Prototype via Placeholder for Zero-shot Recognition
• BoT-SORT: Robust Associations Multi-Pedestrian Tracking
• ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension
• TubeDETR: Spatio-Temporal Video Grounding with Transformers
• Visual Prompt Tuning
• Scaling Open-Vocabulary Image Segmentation with Image-Level Labels
• LAVT: Language-Aware Vision Transformer for Referring Image Segmentation
• Extract Free Dense Labels from CLIP
• ByteTrack: Multi-Object Tracking by Associating Every Detection Box
• ByteTrack: Multi-object Tracking by Associating Every Detection Box
• Learning to Prompt for Vision-Language Models
• Vision-Language Transformer and Query Generation for Referring Segmentation
• MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding
• TransVG: End-to-End Visual Grounding with Transformers
• TS-CAM: Token Semantic Coupled Attention Map for Weakly Supervised Object Localization
• Learning Transferable Visual Models From Natural Language Supervision
• Training data-efficient image transformers & distillation through attention
• Human-centric Spatio-Temporal Video Grounding With Visual Transformers
• Where Does It Exist: Spatio-Temporal Video Grounding for Multi-Form Sentences
• Segmentation from Natural Language Expressions
• Learning Deep Features for Discriminative Localization

Cited by

• Decoupled Entropy Minimization

Related

• Detector-Empowered Video Large Language Model for Efficient Spatio-Temporal Grounding
• Thinking With Bounding Boxes: Enhancing Spatio-Temporal Video Grounding via Reinforcement Fine-Tuning
• Enrich and Detect: Video Temporal Grounding with Multimodal LLMs
• A Survey on Video Temporal Grounding with Multimodal Large Language Model
• OmniGround: A Comprehensive Spatio-Temporal Grounding Benchmark for Real-World Complex Scenarios