Denoise and Align: Diffusion-Driven Foreground Knowledge Prompting for Open-Vocabulary Temporal Action Detection
Abstract
Open-Vocabulary Temporal Action Detection (OV-TAD) aims to localize and classify action segments of unseen categories in untrimmed videos, where effective alignment between action semantics and video representations is critical for accurate detection. However, existing methods struggle to mitigate the semantic imbalance between concise, abstract action labels and rich, complex video contents, inevitably introducing semantic noise and misleading cross-modal alignment. To address this challenge, we propose DFAlign, the first framework that leverages diffusion-based denoising to generate foreground knowledge for the guidance of action–video alignment. Following the 'conditioning, denoising and aligning' manner, we first introduce the Semantic-Unify Conditioning (SUC) module, which unifies action-shared and action-specific semantics as conditions for diffusion denoising. Then, the Background-Suppress Denoising (BSD) module generates foreground knowledge by progressively removing background redundancy from videos through denoising process. This foreground knowledge serves as effective intermediate semantic anchor between video and text representations, mitigating the semantic gap and enhancing the discriminability of action-relevant segments. Furthermore, we introduce the Foreground-Prompt Alignment (FPA) module to inject extracted foreground knowledge as prompt tokens into text representations, guiding model's attention towards action-relevant segments and enabling precise cross-modal alignment. Extensive experiments demonstrate that our method achieves state-of-the-art performance on two OV-TAD benchmarks. The code repository is provided as follows: https://github.com/Sasa77777779/DFAlign_SIGIR26.git.
Assigned reviewers
| # | Reviewer | Match | Why (closest evidence publications in the taxonomy) | |
|---|---|---|---|---|
| 1 | Linchao Zhu | 80 | 69 ActBERT: Learning Global-Local Video-Text Representations (2020) Video and Moment Retrieval 69 Variational Cross-Graph Reasoning and Adaptive Structured Semantics Learning for Compositional Temporal Grounding (2023) Video and Moment Retrieval 69 Stitching Segments and Sentences towards Generalization in Video-Text Pre-training (2024) Video and Moment Retrieval |
|
| 2 | Haiyang Xu 0001 | 63 | 69 Learning Semantics-Grounded Vocabulary Representation for Video-Text Retrieval (2023) Video and Moment Retrieval 69 Unifying Latent and Lexicon Representations for Effective Video-Text Retrieval (2024) Video and Moment Retrieval 59 mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models (2025) Video and Temporal Understanding |
Area chair
| Area chair | Match | |
|---|---|---|
| Xiaoye Qu | 57 |
Area-chair candidates (8)
| Xiaoye Qu assigned | 57 | 15 papers | |
| Xun Yang 0001 | 51 | 4 papers | |
| Hao Fei 0001 | 36 | 4 papers | |
| Hongjin Qian | 35 | 12 papers | |
| Kam-Fai Wong | 34 | 6 papers | |
| Chenyan Xiong | 33 | 15 papers | |
| Xiaoxi Li 0005 | 33 | 15 papers | |
| Yi R. Fung 0001 | 32 | 10 papers |
Candidates from the panel ranked by match score
| # | Reviewer | Match | Load | Why | |
|---|---|---|---|---|---|
| 1 | Linchao Zhu assigned | 80 | 3 | 69 ActBERT: Learning Global-Local Video-Text Representations (2020) Video and Moment Retrieval 69 Variational Cross-Graph Reasoning and Adaptive Structured Semantics Learning for Compositional Temporal Grounding (2023) Video and Moment Retrieval |
|
| 2 | Jianfeng Dong | 63 | 3 | 69 IVCR-200K: A Large-Scale Multi-turn Dialogue Benchmark for Interactive Video Corpus Retrieval (2025) Conversational Search and Retrieval 69 Temporal Sentence Grounding with Relevance Feedback in Videos (2024) Video and Moment Retrieval |
|
| 3 | Haiyang Xu 0001 assigned | 63 | 3 | 69 Learning Semantics-Grounded Vocabulary Representation for Video-Text Retrieval (2023) Video and Moment Retrieval 69 Unifying Latent and Lexicon Representations for Effective Video-Text Retrieval (2024) Video and Moment Retrieval |
|
| 4 | Zihao Yue | 58 | 3 | 69 VideoOrion: Tokenizing Object Dynamics in Videos (2025) Multimodal Reasoning and Grounding 69 Time-R1: Post-Training Large Vision Language Model for Temporal Video Grounding (2025) Multimodal Reasoning and Grounding |
|
| 7 | Huaiwen Zhang | 46 | 3 | 69 Debiased Video-Text Retrieval via Soft Positive Sample Calibration (2023) Video and Moment Retrieval 69 Sequence-Event Semantic Consistent Learning for Text-to-Motion Retrieval (2025) Video and Moment Retrieval |
|
| 8 | Zhe Xu 0009 | 45 | 3 | 69 Point-Supervised Video Temporal Grounding (2023) Video and Moment Retrieval 69 HiSA: Hierarchically Semantic Associating for Video Temporal Grounding (2022) Video and Moment Retrieval |
|
| 9 | Junjie Zhou 0001 | 45 | 3 | 69 MomentSeeker: A Task-Oriented Benchmark For Long-Video Moment Retrieval (2025) Video and Moment Retrieval 59 Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding (2025) Video and Temporal Understanding |
|
| 10 | Qingyi Si | 43 | 3 | 52 Combo of Thinking and Observing for Outside-Knowledge VQA (2023) Retrieval-Augmented and Open-Domain QA 51 Object Attribute Matters in Visual Question Answering (2024) Multimodal Reasoning and Grounding |
|
| 11 | Liwei Wang 0009 | 41 | 3 | 59 AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruning (2025) Video and Temporal Understanding 42 RAST: Domain-Robust Dialogue Rewriting as Sequence Tagging (2021) Conversational Search and Retrieval |
|
| 12 | Minghui Fang 0002 | 40 | 3 | 44 Vela: Scalable Embeddings with Voice Large Language Models for Multimodal Retrieval (2025) Universal Multimodal Embedding and MLLM Retrieval 38 OmniSep: Unified Omni-Modality Sound Separation with Query-Mixup (2025) Multimodal Retrieval and Embedding |
|
| 13 | Haokun Wen | 38 | 3 | 41 Differential-Perceptive and Retrieval-Augmented MLLM for Change Captioning (2024) Multimodal and Graph RAG 38 HUD: Hierarchical Uncertainty-Aware Disambiguation Network for Composed Video Retrieval (2025) Universal Multimodal Embedding and MLLM Retrieval |
|
| 14 | Tingyu Song | 38 | 3 | 55 VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC Videos (2025) Video and Temporal Understanding 43 Rethinking Composed Image Retrieval Evaluation: A Fine-Grained Benchmark from Image Editing (2026) Universal Multimodal Embedding and MLLM Retrieval |
|
| 15 | Jinpeng Wang 0002 | 37 | 3 | 69 Enhancing Partially Relevant Video Retrieval with Hyperbolic Learning (2025) Video and Moment Retrieval 44 Imagine with Layout and Sketch: Enhancing Vision-Language Retrieval with Dual-Stream Multi-Modal Query Refinement (2026) Universal Multimodal Embedding and MLLM Retrieval |
|
| 17 | Navve Wasserman | 35 | 3 | 42 REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmark (2025) Universal Multimodal Embedding and MLLM Retrieval 42 DocReRank: Single-Page Hard Negative Query Generation for Training Multi-Modal RAG Rerankers (2025) Universal Multimodal Embedding and MLLM Retrieval |
|
| 18 | Jian Lang | 35 | 3 | 44 REDEEMing Modality Information Loss: Retrieval-Guided Conditional Generation for Severely Modality Missing Learning (2025) Universal Multimodal Embedding and MLLM Retrieval 38 In-context Prompt-augmented Micro-video Popularity Prediction (2025) Universal Multimodal Embedding and MLLM Retrieval |
|
| 19 | Sheng-Chieh Lin | 35 | 3 | 44 Mm-Embed: Universal Multimodal Retrieval with Multimodal LLMS (2025) Universal Multimodal Embedding and MLLM Retrieval 43 Unifying Multimodal Retrieval via Document Screenshot Embedding (2024) Universal Multimodal Embedding and MLLM Retrieval |
|
| 20 | Yanzhao Zhang | 35 | 3 | 44 Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs (2025) Universal Multimodal Embedding and MLLM Retrieval 44 Bridging Modalities: Improving Universal Multimodal Retrieval by Multimodal Large Language Models (2025) Universal Multimodal Embedding and MLLM Retrieval |
|
| 21 | Xinying Qian | 35 | 3 | 38 Hyperbolic Multimodal Generative Representation Learning for Generalized Zero-Shot Multimodal Information Extraction (2026) Universal Multimodal Embedding and MLLM Retrieval 38 SMIR: Span-based multi-grained information refinement for joint multimodal entity-relation extraction (2026) Universal Multimodal Embedding and MLLM Retrieval |
|
| 24 | Archiki Prasad | 34 | 3 | 51 Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models (2024) Multimodal Reasoning and Grounding 48 GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMs (2026) Language Model Pre-training and Adaptation |
|
| 25 | Peitian Zhang | 34 | 3 | 59 Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding (2025) Video and Temporal Understanding 34 INTERS: Unlocking the Power of Large Language Models in Search with Instruction Tuning (2024) LLM-Based Reranking |
|
| 26 | Xuhui Sui | 33 | 3 | 38 Coarse-to-fine multimodal prototype network for few-shot multimodal relation extraction (2025) Universal Multimodal Embedding and MLLM Retrieval 38 SMIR: Span-based multi-grained information refinement for joint multimodal entity-relation extraction (2026) Universal Multimodal Embedding and MLLM Retrieval |
|
| 27 | Huiqiang Jiang | 33 | 3 | 42 SeCom: On Memory Construction and Retrieval for Personalized Conversational Agents (2025) Conversational Search and Retrieval 38 Attentive Mask CLIP (2023) Universal Multimodal Embedding and MLLM Retrieval |
|
| 30 | Xingyu Tan 0001 | 32 | 3 | 37 PRoH: Dynamic Planning and Reasoning over Knowledge Hypergraphs for Retrieval-Augmented Generation (2026) Multimodal and Graph RAG 34 MemoTime: Memory-Augmented Temporal Knowledge Graph Enhanced Large Language Model Reasoning (2026) KG-Augmented LLM Reasoning |
|
| 32 | Weipeng Chen | 32 | 3 | 55 Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs (2025) Video and Temporal Understanding 36 AITQE: An Adaptive Image-Text Quality Enhancer for Scalable MLLM Pretraining (2026) Multimodal Evaluation and Benchmarks |
|
| 33 | Weiwei Sun 0001 | 32 | 3 | 42 DukeNet: A Dual Knowledge Interaction Network for Knowledge-Grounded Conversation (2020) Conversational Search and Retrieval 32 ReasonRank: Empowering Passage Ranking with Strong Reasoning Ability (2026) LLM-Based Reranking |
|
| 34 | Jinheon Baek | 32 | 3 | 44 UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularities (2026) Multimodal and Graph RAG 39 Realistic Conversational Question Answering with Answer Selection based on Calibrated Confidence and Uncertainty Measurement (2023) Conversational Search and Retrieval |
|
| 35 | Qinggang Zhang COI: co-author | 32 | 3 | 38 Facilitating Generative Retrieval with Logical Denoising for Interpretable Conversational Search (2026) Personalized and Conversational RAG 37 You Don't Need Pre-Built Graphs for RAG: Retrieval Augmented Generation with Adaptive Reasoning Structures (2026) Multi-Hop and Iterative Retrieval Reasoning |
|
| 36 | Shengyao Zhuang | 32 | 3 | 40 Embark on DenseQuest: A System for Selecting the Best Dense Retriever for a Custom Collection (2024) Zero-Shot and Domain Generalization for Retrieval 34 A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Models (2024) LLM-Based Reranking |
|
| 37 | Xinyu Ma 0001 | 32 | 3 | 33 Pre-training with Aspect-Content Text Mutual Prediction for Multi-Aspect Dense Retrieval (2023) Sparse and Dense Retrieval Encoders 32 A Contrastive Pre-training Approach to Discriminative Autoencoder for Dense Retrieval (2022) Sparse and Dense Retrieval Encoders |
|
| 38 | Yujia Zhou 0002 | 32 | 3 | 34 Cognitive Personalized Search Integrating Large Language Models with an Efficient Memory Mechanism (2024) Conversational Search and Retrieval 34 Grounding Language Model with Chunking-Free In-Context Retrieval (2024) Generative Retrieval |