Think with Grounding: Curriculum Reinforced Reasoning with Video Grounding for Long Video Understanding
Abstract
Long video understanding (LVU) is challenging due to rich and complicated multimodal clues in long temporal range. Current methods adopt reasoning to improve the model's ability to analyze complex video clues in long videos via text-form reasoning. However, the existing literature suffers from the fact that the text-only reasoning under fixed video context may exacerbate hallucinations since detailed crucial clues are often ignored under limited video context length due to the temporal redundancy of long videos. To address this gap, we propose Video-TwG, a curriculum reinforced framework that employs a novel Think-with-Grounding paradigm, enabling video LLMs to actively decide when to perform on-demand grounding during interleaved text–video reasoning, selectively zooming into question-relevant clips only when necessary. Video-TwG can be trained end-to-end in a straightforward manner, without relying on complex auxiliary modules or heavily annotated reasoning traces. In detail, we design a Two-stage Reinforced Curriculum Strategy, where the model first learns think-with-grounding behavior on a small short-video GQA dataset with grounding labels, and then scales to diverse general QA data with videos of diverse domains to encourage generalization. Further, to handle complex think-with-grounding reasoning for various kinds of data, we propose the TwG-GRPO algorithm, which features the fine-grained grounding reward, self-confirmed pseudo reward, and accuracy-gated mechanism. Finally, we propose to construct a new TwG-51K dataset that facilitates training. Experiments on Video-MME, LongVideoBench, and MLVU show that Video-TwG consistently outperforms strong LVU baselines. Further ablation validates the necessity of our Two-stage Reinforced Curriculum Strategy and shows our TwG-GRPO better leverages diverse unlabeled data to improve grounding quality and reduce redundant groundings without sacrificing QA performance. https://github.com/hlchen23/Video-TwG
Assigned reviewers
| # | Reviewer | Match | Why (closest evidence publications in the taxonomy) | |
|---|---|---|---|---|
| 1 | Zihao Yue | 61 | 100 Time-R1: Post-Training Large Vision Language Model for Temporal Video Grounding (2025) Multimodal Reasoning and Grounding 100 VideoOrion: Tokenizing Object Dynamics in Videos (2025) Multimodal Reasoning and Grounding 73 Unified Multimodal Understanding via Byte-Pair Visual Encoding (2025) Multimodal Reasoning and Grounding |
|
| 2 | Qingyi Si | 45 | 73 Object Attribute Matters in Visual Question Answering (2024) Multimodal Reasoning and Grounding 73 Check It Again: Progressive Visual Question Answering via Visual Entailment (2021) Multimodal Reasoning and Grounding 66 Cross-modality Multiple Relations Learning for Knowledge-based Visual Question Answering (2024) Knowledge Graph Question Answering |
Area chair
| Area chair | Match | |
|---|---|---|
| Xiaoye Qu | 43 |
Area-chair candidates (7)
| Xiaoye Qu assigned | 43 | 15 papers | |
| Hao Fei 0001 | 35 | 4 papers | |
| Yi R. Fung 0001 | 33 | 10 papers | |
| Dacheng Tao | 33 | 5 papers | |
| Kam-Fai Wong | 32 | 6 papers | |
| Xiaoxi Li 0005 | 32 | 15 papers | |
| Hongjin Qian | 32 | 12 papers |
Candidates from the panel ranked by match score
| # | Reviewer | Match | Load | Why | |
|---|---|---|---|---|---|
| 1 | Zihao Yue assigned | 61 | 3 | 100 Time-R1: Post-Training Large Vision Language Model for Temporal Video Grounding (2025) Multimodal Reasoning and Grounding 100 VideoOrion: Tokenizing Object Dynamics in Videos (2025) Multimodal Reasoning and Grounding |
|
| 2 | Haiyang Xu 0001 | 59 | 3 | 73 mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models (2025) Video and Temporal Understanding 73 SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimization (2025) Multimodal Reasoning and Grounding |
|
| 3 | Linchao Zhu COI: co-author | 53 | 3 | 73 From Trial to Triumph: Advancing Long Video Understanding via Visual Context Sample Scaling and Self-Reward Alignment (2025) Video and Temporal Understanding 73 MIST : Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answering (2023) Video and Temporal Understanding |
|
| 4 | Qingyi Si assigned | 45 | 3 | 73 Object Attribute Matters in Visual Question Answering (2024) Multimodal Reasoning and Grounding 73 Check It Again: Progressive Visual Question Answering via Visual Entailment (2021) Multimodal Reasoning and Grounding |
|
| 6 | Minghui Fang 0002 | 38 | 3 | 42 Mitigating Hallucinations in LM-Based TTS Models via Distribution Alignment Using GFlowNets (2025) Multimodal Retrieval and Embedding 42 SyncTalklip: Highly Synchronized Lip-Readable Speaker Generation with Multi-Task Learning (2024) Multimodal Retrieval and Embedding |
|
| 7 | Archiki Prasad | 37 | 3 | 73 Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models (2024) Multimodal Reasoning and Grounding 63 GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMs (2026) Language Model Pre-training and Adaptation |
|
| 8 | Junjie Zhou 0001 | 37 | 3 | 73 Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding (2025) Video and Temporal Understanding 71 MLVU: Benchmarking Multi-task Long Video Understanding (2025) Video and Temporal Understanding |
|
| 9 | Tingyu Song | 37 | 3 | 71 VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC Videos (2025) Video and Temporal Understanding 36 Rethinking Composed Image Retrieval Evaluation: A Fine-Grained Benchmark from Image Editing (2026) Universal Multimodal Embedding and MLLM Retrieval |
|
| 10 | Liwei Wang 0009 | 36 | 3 | 73 AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruning (2025) Video and Temporal Understanding 38 RAST: Domain-Robust Dialogue Rewriting as Sequence Tagging (2021) Conversational Search and Retrieval |
|
| 12 | Xingyu Tan 0001 | 34 | 3 | 38 PRoH: Dynamic Planning and Reasoning over Knowledge Hypergraphs for Retrieval-Augmented Generation (2026) Multimodal and Graph RAG 36 MemoTime: Memory-Augmented Temporal Knowledge Graph Enhanced Large Language Model Reasoning (2026) KG-Augmented LLM Reasoning |
|
| 13 | Jiapu Wang | 34 | 3 | 65 Large-Small Model Synergy with Multimodal Fine-Grained Heuristics for Knowledge-Based Visual Question Answering (2025) Retrieval-Augmented and Open-Domain QA 33 Decomposing and Revising What Language Models Generate (2025) Multi-Hop and Iterative Retrieval Reasoning |
|
| 14 | Jianfeng Dong | 34 | 3 | 40 IVCR-200K: A Large-Scale Multi-turn Dialogue Benchmark for Interactive Video Corpus Retrieval (2025) Conversational Search and Retrieval 37 Temporal Sentence Grounding with Relevance Feedback in Videos (2024) Video and Moment Retrieval |
|
| 16 | Weipeng Chen | 33 | 3 | 71 Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs (2025) Video and Temporal Understanding 38 AITQE: An Adaptive Image-Text Quality Enhancer for Scalable MLLM Pretraining (2026) Multimodal Evaluation and Benchmarks |
|
| 18 | Mariano Barone | 33 | 3 | 38 AMICO: A Semantic and Multimodal Framework for AI-Assisted Clinical Reporting (2025) Multimodal Evaluation and Benchmarks 36 A Multilingual Multimodal Medical Examination Dataset for Visual Question Answering in Healthcare (2025) Retrieval-Augmented and Open-Domain QA |
|
| 19 | Qinggang Zhang | 33 | 3 | 38 KnowGPT: Knowledge Graph based Prompting for Large Language Models (2024) Multimodal and Graph RAG 38 Augmenting Intra-Modal Understanding in MLLMs for Robust Multimodal Keyphrase Generation (2026) Multimodal Evaluation and Benchmarks |
|
| 20 | Xinying Qian | 33 | 3 | 36 TimeR⁴ : Time-aware Retrieval-Augmented Large Language Models for Temporal Knowledge Graph Question Answering (2024) Multi-Hop and Iterative Retrieval Reasoning 34 Beyond Timestamps: Bridging Forward and Backward Reasoning in Temporal Numerical and Relational Understanding (2026) Multi-hop and Complex Reasoning QA |
|
| 21 | Kaixin Ma | 32 | 3 | 67 COLUMBUS: Evaluating COgnitive Lateral Understanding Through Multiple-Choice reBUSes (2025) Multimodal Reasoning and Grounding 38 Open Domain Question Answering with A Unified Knowledge Interface (2022) Multimodal and Graph RAG |
|
| 22 | Nancy F. Chen | 32 | 3 | 42 Multimodal Dialogue State Tracking (2022) Conversational Search and Retrieval 42 Rescoring Hypothesized Detections of Out-of-Vocabulary Keywords Using Subword Samples (2016) Multimodal Retrieval and Embedding |
|
| 26 | Di Wu 0054 | 31 | 3 | 37 VisRet: Visualization Improves Knowledge-Intensive Text-to-Image Retrieval (2026) Multimodal and Graph RAG 36 LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory (2025) Conversational Search and Retrieval |
|
| 27 | Jian Lang | 31 | 3 | 37 REDEEMing Modality Information Loss: Retrieval-Guided Conditional Generation for Severely Modality Missing Learning (2025) Universal Multimodal Embedding and MLLM Retrieval 33 TAMEing Long Contexts in Personalization: Towards Training-Free and State-Aware MLLM Personalized Assistant (2026) Personalized Question Answering and RAG |
|
| 28 | Navve Wasserman | 31 | 3 | 35 REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmark (2025) Universal Multimodal Embedding and MLLM Retrieval 35 DocReRank: Single-Page Hard Negative Query Generation for Training Multi-Modal RAG Rerankers (2025) Universal Multimodal Embedding and MLLM Retrieval |
|
| 29 | Yu Gu 0016 | 31 | 3 | 36 HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models (2024) KG-Augmented LLM Reasoning 34 Few-shot In-context Learning on Knowledge Base Question Answering (2023) Multi-hop and Complex Reasoning QA |
|
| 30 | Armin Toroghi | 31 | 3 | 34 CommonWhy: A Dataset for Evaluating Entity-Based Causal Commonsense Reasoning in Large Language Models (2026) Knowledge Graph Question Answering 32 Bayesian Inference with Complex Knowledge Graph Evidence (2024) Knowledge Graph Question Answering |
|
| 31 | Peitian Zhang | 31 | 3 | 73 Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding (2025) Video and Temporal Understanding 33 INTERS: Unlocking the Power of Large Language Models in Search with Instruction Tuning (2024) LLM-Based Reranking |
|
| 32 | Yanzhao Zhang | 31 | 3 | 38 An End-to-End Model for Photo-Sharing Multi-Modal Dialogue Generation (2025) Multimodal Evaluation and Benchmarks 37 Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs (2025) Universal Multimodal Embedding and MLLM Retrieval |
|
| 33 | Jinheon Baek | 31 | 3 | 39 Realistic Conversational Question Answering with Answer Selection based on Calibrated Confidence and Uncertainty Measurement (2023) Conversational Search and Retrieval 37 UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularities (2026) Multimodal and Graph RAG |
|
| 34 | Idan Szpektor | 31 | 3 | 39 On the Robustness of Dialogue History Representation in Conversational Question Answering: A Comprehensive Study and a New Prompt-based Method (2023) Conversational Search and Retrieval 38 Bridging the Visual Gap: Fine-Tuning Multimodal Models with Knowledge-Adapted Captions (2025) Multimodal Evaluation and Benchmarks |
|
| 35 | Weiwei Sun 0001 | 31 | 3 | 38 DukeNet: A Dual Knowledge Interaction Network for Knowledge-Grounded Conversation (2020) Conversational Search and Retrieval 33 Direct Retrieval-augmented Optimization: Synergizing Knowledge Selection and Language Models (2026) Evidence Selection and Context Refinement |
|
| 36 | Ben Liu 0002 | 31 | 3 | 31 Plan Then Retrieve: Reinforcement Learning-Guided Complex Reasoning over Knowledge Graphs (2026) RL Training of Search Agents 30 Filter-then-Generate: Large Language Models with Structure-Text Adapter for Knowledge Graph Completion (2025) Knowledge Graph Embedding and Completion |
|
| 37 | Chen Zhao 0013 | 30 | 3 | 38 Are Multimodal LLMs Robust Against Adversarial Perturbations? RoMMath: A Systematic Evaluation on Multimodal Math Reasoning (2025) Multimodal Evaluation and Benchmarks 34 TaPERA: Enhancing Faithfulness and Interpretability in Long-Form Table QA by Content Planning and Execution-based Reasoning (2024) Multi-hop and Complex Reasoning QA |