Agentic Spatio-Temporal Grounding via Collaborative Reasoning
Abstract
Spatio-Temporal Video Grounding (STVG) aims to retrieve the spatio-temporal tube of a target object or person in a video given a text query. Most existing approaches perform frame-wise spatial localization within a predicted temporal span, resulting in redundant computation, heavy supervision requirements, and limited generalization. Weakly-supervised variants mitigate annotation costs but remain constrained by the dataset-level train-and-fit paradigm with an inferior performance. To address these challenges, we propose the Agentic Spatio-Temporal Grounder (ASTG) framework for the task of STVG in an open-world and zero-shot setting. Specifically, two specialized agents SRA (Spatial Reasoning Agent) and TRA (Temporal Reasoning Agent) constructed leveraging modern Multi-modal Large Language Models (MLLMs) work collaboratively to retrieve the target tube in an autonomous and self-guided manner. Following a propose-and-evaluation paradigm, ASTG duly decouples spatio-temporal reasoning and automates the tube extraction, verification and temporal localization processes. With a dedicated visual memory and dialogue context, ASTG achieves architectural efficiency by minimizing the number of reasoning calls compared to exhaustive per-frame reasoning, eliminating the logical redundancy inherent in joint-reasoning systems. Experiments on popular benchmarks demonstrate the superiority of the proposed approach where it outperforms existing weakly-supervised and zero-shot approaches by a margin and is comparable to some of the fully-supervised methods.
Assigned reviewers
| # | Reviewer | Match | Why (closest evidence publications in the taxonomy) | |
|---|---|---|---|---|
| 1 | Linchao Zhu | 80 | 69 ActBERT: Learning Global-Local Video-Text Representations (2020) Video and Moment Retrieval 69 Variational Cross-Graph Reasoning and Adaptive Structured Semantics Learning for Compositional Temporal Grounding (2023) Video and Moment Retrieval 69 Stitching Segments and Sentences towards Generalization in Video-Text Pre-training (2024) Video and Moment Retrieval |
|
| 2 | Haiyang Xu 0001 | 63 | 69 Learning Semantics-Grounded Vocabulary Representation for Video-Text Retrieval (2023) Video and Moment Retrieval 69 Unifying Latent and Lexicon Representations for Effective Video-Text Retrieval (2024) Video and Moment Retrieval 59 mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models (2025) Video and Temporal Understanding |
Area chair
| Area chair | Match | |
|---|---|---|
| Xiaoye Qu | 57 |
Area-chair candidates (8)
| Xiaoye Qu assigned | 57 | 15 papers | |
| Xun Yang 0001 | 51 | 4 papers | |
| Hao Fei 0001 | 36 | 4 papers | |
| Hongjin Qian | 35 | 12 papers | |
| Kam-Fai Wong | 34 | 6 papers | |
| Chenyan Xiong | 33 | 15 papers | |
| Xiaoxi Li 0005 | 33 | 15 papers | |
| Yi R. Fung 0001 | 32 | 10 papers |
Candidates from the panel ranked by match score
| # | Reviewer | Match | Load | Why | |
|---|---|---|---|---|---|
| 1 | Linchao Zhu assigned | 80 | 3 | 69 ActBERT: Learning Global-Local Video-Text Representations (2020) Video and Moment Retrieval 69 Variational Cross-Graph Reasoning and Adaptive Structured Semantics Learning for Compositional Temporal Grounding (2023) Video and Moment Retrieval |
|
| 2 | Jianfeng Dong | 63 | 3 | 69 IVCR-200K: A Large-Scale Multi-turn Dialogue Benchmark for Interactive Video Corpus Retrieval (2025) Conversational Search and Retrieval 69 Temporal Sentence Grounding with Relevance Feedback in Videos (2024) Video and Moment Retrieval |
|
| 3 | Haiyang Xu 0001 assigned | 63 | 3 | 69 Learning Semantics-Grounded Vocabulary Representation for Video-Text Retrieval (2023) Video and Moment Retrieval 69 Unifying Latent and Lexicon Representations for Effective Video-Text Retrieval (2024) Video and Moment Retrieval |
|
| 4 | Zihao Yue | 58 | 3 | 69 Time-R1: Post-Training Large Vision Language Model for Temporal Video Grounding (2025) Multimodal Reasoning and Grounding 69 VideoOrion: Tokenizing Object Dynamics in Videos (2025) Multimodal Reasoning and Grounding |
|
| 7 | Huaiwen Zhang | 46 | 3 | 69 Sequence-Event Semantic Consistent Learning for Text-to-Motion Retrieval (2025) Video and Moment Retrieval 69 Robust Video-Text Retrieval Via Noisy Pair Calibration (2023) Video and Moment Retrieval |
|
| 8 | Zhe Xu 0009 | 45 | 3 | 69 Point-Supervised Video Temporal Grounding (2023) Video and Moment Retrieval 69 HiSA: Hierarchically Semantic Associating for Video Temporal Grounding (2022) Video and Moment Retrieval |
|
| 9 | Junjie Zhou 0001 | 45 | 3 | 69 MomentSeeker: A Task-Oriented Benchmark For Long-Video Moment Retrieval (2025) Video and Moment Retrieval 59 Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding (2025) Video and Temporal Understanding |
|
| 10 | Qingyi Si | 43 | 3 | 52 Combo of Thinking and Observing for Outside-Knowledge VQA (2023) Retrieval-Augmented and Open-Domain QA 51 Object Attribute Matters in Visual Question Answering (2024) Multimodal Reasoning and Grounding |
|
| 11 | Liwei Wang 0009 | 41 | 3 | 59 AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruning (2025) Video and Temporal Understanding 42 RAST: Domain-Robust Dialogue Rewriting as Sequence Tagging (2021) Conversational Search and Retrieval |
|
| 12 | Minghui Fang 0002 | 40 | 3 | 44 Vela: Scalable Embeddings with Voice Large Language Models for Multimodal Retrieval (2025) Universal Multimodal Embedding and MLLM Retrieval 38 VoxpopuliTTS: a large-scale multilingual TTS corpus for zero-shot speech generation (2025) Multimodal Retrieval and Embedding |
|
| 13 | Haokun Wen | 38 | 3 | 41 Differential-Perceptive and Retrieval-Augmented MLLM for Change Captioning (2024) Multimodal and Graph RAG 38 HUD: Hierarchical Uncertainty-Aware Disambiguation Network for Composed Video Retrieval (2025) Universal Multimodal Embedding and MLLM Retrieval |
|
| 14 | Tingyu Song | 38 | 3 | 55 VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC Videos (2025) Video and Temporal Understanding 43 Rethinking Composed Image Retrieval Evaluation: A Fine-Grained Benchmark from Image Editing (2026) Universal Multimodal Embedding and MLLM Retrieval |
|
| 15 | Jinpeng Wang 0002 | 37 | 3 | 69 Enhancing Partially Relevant Video Retrieval with Hyperbolic Learning (2025) Video and Moment Retrieval 44 Imagine with Layout and Sketch: Enhancing Vision-Language Retrieval with Dual-Stream Multi-Modal Query Refinement (2026) Universal Multimodal Embedding and MLLM Retrieval |
|
| 17 | Navve Wasserman | 35 | 3 | 42 REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmark (2025) Universal Multimodal Embedding and MLLM Retrieval 42 DocReRank: Single-Page Hard Negative Query Generation for Training Multi-Modal RAG Rerankers (2025) Universal Multimodal Embedding and MLLM Retrieval |
|
| 18 | Jian Lang | 35 | 3 | 44 REDEEMing Modality Information Loss: Retrieval-Guided Conditional Generation for Severely Modality Missing Learning (2025) Universal Multimodal Embedding and MLLM Retrieval 38 Generative Thinking, Corrective Action: User-Friendly Composed Image Retrieval via Automatic Multi-Agent Collaboration (2025) Universal Multimodal Embedding and MLLM Retrieval |
|
| 19 | Sheng-Chieh Lin | 35 | 3 | 44 Mm-Embed: Universal Multimodal Retrieval with Multimodal LLMS (2025) Universal Multimodal Embedding and MLLM Retrieval 43 Unifying Multimodal Retrieval via Document Screenshot Embedding (2024) Universal Multimodal Embedding and MLLM Retrieval |
|
| 20 | Yanzhao Zhang | 35 | 3 | 44 Bridging Modalities: Improving Universal Multimodal Retrieval by Multimodal Large Language Models (2025) Universal Multimodal Embedding and MLLM Retrieval 44 Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs (2025) Universal Multimodal Embedding and MLLM Retrieval |
|
| 21 | Xinying Qian | 35 | 3 | 38 Hyperbolic Multimodal Generative Representation Learning for Generalized Zero-Shot Multimodal Information Extraction (2026) Universal Multimodal Embedding and MLLM Retrieval 38 SMIR: Span-based multi-grained information refinement for joint multimodal entity-relation extraction (2026) Universal Multimodal Embedding and MLLM Retrieval |
|
| 24 | Archiki Prasad | 34 | 3 | 51 Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models (2024) Multimodal Reasoning and Grounding 48 GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMs (2026) Language Model Pre-training and Adaptation |
|
| 25 | Peitian Zhang | 34 | 3 | 59 Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding (2025) Video and Temporal Understanding 34 INTERS: Unlocking the Power of Large Language Models in Search with Instruction Tuning (2024) LLM-Based Reranking |
|
| 26 | Xuhui Sui | 33 | 3 | 38 Coarse-to-fine multimodal prototype network for few-shot multimodal relation extraction (2025) Universal Multimodal Embedding and MLLM Retrieval 38 SMIR: Span-based multi-grained information refinement for joint multimodal entity-relation extraction (2026) Universal Multimodal Embedding and MLLM Retrieval |
|
| 27 | Huiqiang Jiang | 33 | 3 | 42 SeCom: On Memory Construction and Retrieval for Personalized Conversational Agents (2025) Conversational Search and Retrieval 38 Attentive Mask CLIP (2023) Universal Multimodal Embedding and MLLM Retrieval |
|
| 30 | Xingyu Tan 0001 | 32 | 3 | 37 PRoH: Dynamic Planning and Reasoning over Knowledge Hypergraphs for Retrieval-Augmented Generation (2026) Multimodal and Graph RAG 34 MemoTime: Memory-Augmented Temporal Knowledge Graph Enhanced Large Language Model Reasoning (2026) KG-Augmented LLM Reasoning |
|
| 32 | Weipeng Chen | 32 | 3 | 55 Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs (2025) Video and Temporal Understanding 36 AITQE: An Adaptive Image-Text Quality Enhancer for Scalable MLLM Pretraining (2026) Multimodal Evaluation and Benchmarks |
|
| 33 | Weiwei Sun 0001 | 32 | 3 | 42 DukeNet: A Dual Knowledge Interaction Network for Knowledge-Grounded Conversation (2020) Conversational Search and Retrieval 32 ReasonRank: Empowering Passage Ranking with Strong Reasoning Ability (2026) LLM-Based Reranking |
|
| 34 | Jinheon Baek | 32 | 3 | 44 UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularities (2026) Multimodal and Graph RAG 39 Realistic Conversational Question Answering with Answer Selection based on Calibrated Confidence and Uncertainty Measurement (2023) Conversational Search and Retrieval |
|
| 35 | Qinggang Zhang | 32 | 3 | 38 Facilitating Generative Retrieval with Logical Denoising for Interpretable Conversational Search (2026) Personalized and Conversational RAG 37 You Don't Need Pre-Built Graphs for RAG: Retrieval Augmented Generation with Adaptive Reasoning Structures (2026) Multi-Hop and Iterative Retrieval Reasoning |
|
| 36 | Shengyao Zhuang | 32 | 3 | 40 Embark on DenseQuest: A System for Selecting the Best Dense Retriever for a Custom Collection (2024) Zero-Shot and Domain Generalization for Retrieval 34 A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Models (2024) LLM-Based Reranking |
|
| 37 | Xinyu Ma 0001 | 32 | 3 | 33 Pre-training with Aspect-Content Text Mutual Prediction for Multi-Aspect Dense Retrieval (2023) Sparse and Dense Retrieval Encoders 32 A Contrastive Pre-training Approach to Discriminative Autoencoder for Dense Retrieval (2022) Sparse and Dense Retrieval Encoders |
|
| 38 | Yujia Zhou 0002 | 32 | 3 | 34 Cognitive Personalized Search Integrating Large Language Models with an Efficient Memory Mechanism (2024) Conversational Search and Retrieval 34 Grounding Language Model with Chunking-Free In-Context Retrieval (2024) Generative Retrieval |