Demonstration venue · read-only. Every page can be browsed; the buttons that would change it are switched off. Create an account to run TaxoReview on your own data. A simulation from public publication records: the people listed were not invited and did not agree to review.
? Tutorial for this page

Towards Mixed-Modal Retrieval for Universal Retrieval-Augmented Generation

sigir26-192 · Full research paper · Chenghao Zhang, Guanting Dong, Xinyu Yang, Zhicheng Dou
Multimodal and Graph RAG · 50% Universal Multimodal Embedding and M · 50%
Abstract

Retrieval-Augmented Generation (RAG) has emerged as a powerful paradigm for enhancing large language models (LLMs) by retrieving relevant documents from an external corpus. However, existing RAG systems primarily focus on unimodal text documents, and often fall short in real-world scenarios where both queries and documents may contain mixed modalities (such as text and images). In this paper, we address the challenge of Universal Retrieval-Augmented Generation (URAG), which involves retrieving and reasoning over mixed-modal information to improve vision-language generation. To this end, we propose Nyx, a unified mixed-modal to mixed-modal retriever tailored for URAG scenarios. To mitigate the scarcity of realistic mixed-modal data, we introduce a four-stage automated pipeline for generation and filtering, leveraging web documents to construct NyxQA, a dataset comprising diverse mixed-modal question-answer pairs that better reflect real-world information needs. Building on this high-quality dataset, we adopt a two-stage training framework for Nyx: we first perform pre-training on NyxQA along with a variety of open-source retrieval datasets, followed by supervised fine-tuning using feedback from downstream vision-language models (VLMs) to align retrieval outputs with generative preferences. Experimental results demonstrate that Nyx not only performs competitively on standard text-only RAG benchmarks, but also excels in the more general and realistic URAG setting, significantly improving generation quality in vision-language tasks.

Assigned reviewers

#ReviewerMatchWhy (closest evidence publications in the taxonomy)
1 Navve Wasserman
63
87 DocReRank: Single-Page Hard Negative Query Generation for Training Multi-Modal RAG Rerankers (2025) Universal Multimodal Embedding and MLLM Retrieval
87 REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmark (2025) Universal Multimodal Embedding and MLLM Retrieval
17 Paint by Inpaint: Learning to Add Image Objects by Removing Them First (2025) Generative Models for Recommendation and Content › Others
2 Jian Lang
59
74 REDEEMing Modality Information Loss: Retrieval-Guided Conditional Generation for Severely Modality Missing Learning (2025) Universal Multimodal Embedding and MLLM Retrieval
72 Generative Thinking, Corrective Action: User-Friendly Composed Image Retrieval via Automatic Multi-Agent Collaboration (2025) Universal Multimodal Embedding and MLLM Retrieval
72 Echoes in the Feed: Evolution-aware Prompt-augmented Micro-video Popularity Prediction (2025) Universal Multimodal Embedding and MLLM Retrieval
3 Xuhui Sui
52
72 Coarse-to-fine multimodal prototype network for few-shot multimodal relation extraction (2025) Universal Multimodal Embedding and MLLM Retrieval
72 SMIR: Span-based multi-grained information refinement for joint multimodal entity-relation extraction (2026) Universal Multimodal Embedding and MLLM Retrieval
61 Hyperbolic Multimodal Generative Representation Learning for Generalized Zero-Shot Multimodal Information Extraction (2026) Universal Multimodal Embedding and MLLM Retrieval

Area chair

Area chairMatch
Xiaoye Qu 46
Area-chair candidates (3)
Xiaoye Qu assigned 4615 papers
Hao Fei 0001 374 papers
Hongjin Qian COI: co-author 3712 papers

Candidates from the panel ranked by match score

#ReviewerMatchLoadWhy
1 Haokun Wen 72 3
74 Differential-Perceptive and Retrieval-Augmented MLLM for Change Captioning (2024) Multimodal and Graph RAG
72 FiRE: Enhancing MLLMs with Fine-Grained Context Learning for Complex Image Retrieval (2025) Universal Multimodal Embedding and MLLM Retrieval
2 Navve Wasserman assigned 63 3
87 DocReRank: Single-Page Hard Negative Query Generation for Training Multi-Modal RAG Rerankers (2025) Universal Multimodal Embedding and MLLM Retrieval
87 REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmark (2025) Universal Multimodal Embedding and MLLM Retrieval
3 Liwei Wang 0009 60 3
72 Learning Two-Branch Neural Networks for Image-Text Matching Tasks (2019) Universal Multimodal Embedding and MLLM Retrieval
72 Learning Deep Structure-Preserving Image-Text Embeddings (2016) Universal Multimodal Embedding and MLLM Retrieval
4 Jian Lang assigned 59 3
74 REDEEMing Modality Information Loss: Retrieval-Guided Conditional Generation for Severely Modality Missing Learning (2025) Universal Multimodal Embedding and MLLM Retrieval
72 Generative Thinking, Corrective Action: User-Friendly Composed Image Retrieval via Automatic Multi-Agent Collaboration (2025) Universal Multimodal Embedding and MLLM Retrieval
5 Minghui Fang 0002 59 3
74 Vela: Scalable Embeddings with Voice Large Language Models for Multimodal Retrieval (2025) Universal Multimodal Embedding and MLLM Retrieval
72 Multimodal Conditional Retrieval with High Controllability (2025) Universal Multimodal Embedding and MLLM Retrieval
6 Xinying Qian 55 3
72 Hyperbolic Multimodal Generative Representation Learning for Generalized Zero-Shot Multimodal Information Extraction (2026) Universal Multimodal Embedding and MLLM Retrieval
72 SMIR: Span-based multi-grained information refinement for joint multimodal entity-relation extraction (2026) Universal Multimodal Embedding and MLLM Retrieval
7 Xuhui Sui assigned 52 3
72 Coarse-to-fine multimodal prototype network for few-shot multimodal relation extraction (2025) Universal Multimodal Embedding and MLLM Retrieval
72 SMIR: Span-based multi-grained information refinement for joint multimodal entity-relation extraction (2026) Universal Multimodal Embedding and MLLM Retrieval
8 Jinpeng Wang 0002 50 3
74 Imagine with Layout and Sketch: Enhancing Vision-Language Retrieval with Dual-Stream Multi-Modal Query Refinement (2026) Universal Multimodal Embedding and MLLM Retrieval
74 Suit the Remedy to the Retriever: Interpretable Query Optimization with Retriever Preference Alignment for Vision-Language Retrieval (2026) Universal Multimodal Embedding and MLLM Retrieval
9 Yanzhao Zhang 50 3
74 Bridging Modalities: Improving Universal Multimodal Retrieval by Multimodal Large Language Models (2025) Universal Multimodal Embedding and MLLM Retrieval
74 Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs (2025) Universal Multimodal Embedding and MLLM Retrieval
10 Junjie Zhou 0001 COI: co-author 48 3
72 MegaPairs: Massive Data Synthesis for Universal Multimodal Retrieval (2025) Universal Multimodal Embedding and MLLM Retrieval
72 VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval (2024) Universal Multimodal Embedding and MLLM Retrieval
11 Linchao Zhu 47 3
72 Knowledge-Enhanced Dual-Stream Zero-Shot Composed Image Retrieval (2024) Universal Multimodal Embedding and MLLM Retrieval
72 Attention as Selector: Unlocking VLM Attention for Long Document Page Retrieval (2026) Universal Multimodal Embedding and MLLM Retrieval
12 Meiyu Liang 47 3
72 Dynamic Self-adaptive Multiscale Distillation from Pre-trained Multimodal Large Model for Efficient Cross-modal Retrieval (2025) Universal Multimodal Embedding and MLLM Retrieval
72 Knowledge Graph Enhanced Multimodal Transformer for Image-Text Retrieval (2024) Universal Multimodal Embedding and MLLM Retrieval
14 Di Wu 0054 46 3
100 VisRet: Visualization Improves Knowledge-Intensive Text-to-Image Retrieval (2026) Multimodal and Graph RAG
42 LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory (2025) Conversational Search and Retrieval
15 Haiyang Xu 0001 44 3
50 Efficient and Effective In-context Demonstration Selection with Coreset (2026) Multimodal Retrieval and Embedding
50 Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation (2023) Multimodal Retrieval and Embedding
16 Xingyu Tan 0001 44 3
59 PRoH: Dynamic Planning and Reasoning over Knowledge Hypergraphs for Retrieval-Augmented Generation (2026) Multimodal and Graph RAG
46 HydraRAG: Structured Cross-Source Enhanced Large Language Model Reasoning (2025) KG-Augmented LLM Reasoning
17 Sheng-Chieh Lin 43 3
74 Mm-Embed: Universal Multimodal Retrieval with Multimodal LLMS (2025) Universal Multimodal Embedding and MLLM Retrieval
72 Unifying Multimodal Retrieval via Document Screenshot Embedding (2024) Universal Multimodal Embedding and MLLM Retrieval
18 Jianfeng Dong 42 3
68 Feature Re-Learning with Data Augmentation for Content-based Video Recommendation (2018) Universal Multimodal Embedding and MLLM Retrieval
66 Exploring Content-based Video Relevance for Video Click-Through Rate Prediction (2019) Universal Multimodal Embedding and MLLM Retrieval
19 Jiuqiang Li 0002 42 3
70 Multi-view Semantic Contrastive Alignment for Multimodal Recommendation (2026) Universal Multimodal Embedding and MLLM Retrieval
26 Graph Diffusive Self-Supervised Learning for Social Recommendation (2024) Social and Group Recommendation
20 Zihao Yue 42 3
43 OpenMMEgo: Enhancing Egocentric Understanding for LMMs with Open Weights and Data (2025) Video and Temporal Understanding
43 Movie101v2: Improved Movie Narration Benchmark (2025) Video and Temporal Understanding
21 Jinheon Baek 41 3
100 UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularities (2026) Multimodal and Graph RAG
43 Realistic Conversational Question Answering with Answer Selection based on Calibrated Confidence and Uncertainty Measurement (2023) Conversational Search and Retrieval
22 Huiqiang Jiang 41 3
72 Attentive Mask CLIP (2023) Universal Multimodal Embedding and MLLM Retrieval
40 SeCom: On Memory Construction and Retrieval for Personalized Conversational Agents (2025) Conversational Search and Retrieval
23 Tingyu Song 40 3
72 Rethinking Composed Image Retrieval Evaluation: A Fine-Grained Benchmark from Image Editing (2026) Universal Multimodal Embedding and MLLM Retrieval
43 VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC Videos (2025) Video and Temporal Understanding
24 Zheng Zhang 0006 40 3
40 Enhancing Cross-Modal Retrieval via Visual-Textual Prompt Hashing (2024) Cross-Modal Hashing and Quantization
40 Semantics-Reconstructing Hashing for Cross-Modal Retrieval (2020) Cross-Modal Hashing and Quantization
25 Ruqing Zhang 0001 40 3
42 PROP: Pre-training with Representative Words Prediction for Ad-hoc Retrieval (2021) Sparse and Dense Retrieval Encoders
42 B-PROP: Bootstrapped Pre-training with Representative Words Prediction for Ad-hoc Retrieval (2021) Sparse and Dense Retrieval Encoders
26 Xinyu Ma 0001 COI: co-author 40 3
42 PROP: Pre-training with Representative Words Prediction for Ad-hoc Retrieval (2021) Sparse and Dense Retrieval Encoders
42 B-PROP: Bootstrapped Pre-training with Representative Words Prediction for Ad-hoc Retrieval (2021) Sparse and Dense Retrieval Encoders
27 Fengling Li 0001 40 3
40 Noise-Robust Generative Hashing for Cross-Modal Retrieval (2026) Cross-Modal Hashing and Quantization
40 Prompt-Driven Bit Extension Hashing for Continual Cross-Modal Retrieval (2026) Cross-Modal Hashing and Quantization
28 Shu Zhou 0002 39 3
87 When MLLMs Meet ICH: A visual retrieval-augmented generation-based method for intangible cultural heritage image recognition-take Shadow Puppetry as a case (2026) Universal Multimodal Embedding and MLLM Retrieval
47 DynKGRAG: A framework for dynamic knowledge graph construction and multi-hop querying over numerical and textual data (2026) Multi-Hop and Iterative Retrieval Reasoning
29 Weipeng Chen COI: co-author 39 3
43 Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs (2025) Video and Temporal Understanding
42 AITQE: An Adaptive Image-Text Quality Enhancer for Scalable MLLM Pretraining (2026) Multimodal Evaluation and Benchmarks
30 Qingyi Si 38 3
74 Multimodal Hypothetical Summary for Retrieval-based Multi-image Question Answering (2025) Universal Multimodal Embedding and MLLM Retrieval
42 Towards Flexible Evaluation for Generative Visual Question Answering (2024) Multimodal Evaluation and Benchmarks
31 Qinggang Zhang 38 3
64 You Don't Need Pre-Built Graphs for RAG: Retrieval Augmented Generation with Adaptive Reasoning Structures (2026) Multi-Hop and Iterative Retrieval Reasoning
59 KnowGPT: Knowledge Graph based Prompting for Large Language Models (2024) Multimodal and Graph RAG