Mixture-of-Retrieval Experts for Reasoning-Guided Multimodal Knowledge Exploitation
sigir26-064 · Full research paper · Chunyi Peng, Zhipeng Xu, Zhenghao Liu, Yishan Li, Yukun Yan, Shuo Wang, Yu Jason Gu, Yu Jason Gu, Ge Yu, Maosong Sun
Multimodal and Graph RAG · 25% Agentic and Deep Search RAG · 25% Multimodal Reasoning and Grounding · 25% Multimodal Retrieval and Embedding · 25%
Abstract
Multimodal Retrieval-Augmented Generation (MRAG) has shown promise in mitigating hallucinations in Multimodal Large Language Models (MLLMs) by incorporating external knowledge. However, existing methods typically adhere to rigid retrieval paradigms by mimicking fixed retrieval trajectories and thus fail to fully exploit the knowledge of different retrieval experts through dynamic interaction based on the model's knowledge needs or evolving reasoning states. To overcome this limitation, we introduce Mixture-of-Retrieval Experts (MoRE), a novel framework that enables MLLMs to collaboratively interact with diverse retrieval experts for more effective knowledge exploitation. Specifically, MoRE learns to dynamically determine which expert to engage with, conditioned on the evolving reasoning state. To effectively train this capability, we propose Stepwise Group Relative Policy Optimization (Step-GRPO), which goes beyond sparse outcome-based supervision by encouraging MLLMs to interact with multiple retrieval experts and synthesize fine-grained rewards, thereby teaching the MLLM to fully coordinate all experts when answering a given query. Experimental results on diverse open-domain QA benchmarks demonstrate the effectiveness of MoRE, achieving average performance gains of over 7% compared to competitive baselines. Notably, MoRE exhibits strong adaptability by dynamically coordinating heterogeneous experts to precisely locate relevant information, validating its capability for robust, reasoning-driven expert collaboration. All codes and data are released on https://github.com/OpenBMB/MoRE.
Assigned reviewers
| # | Reviewer | Match | Why (closest evidence publications in the taxonomy) | |
|---|---|---|---|---|
| 1 | Qingyi Si | 54 | 58 Combo of Thinking and Observing for Outside-Knowledge VQA (2023) Retrieval-Augmented and Open-Domain QA 57 Object Attribute Matters in Visual Question Answering (2024) Multimodal Reasoning and Grounding 57 Check It Again: Progressive Visual Question Answering via Visual Entailment (2021) Multimodal Reasoning and Grounding |
|
| 2 | Archiki Prasad | 54 | 57 Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models (2024) Multimodal Reasoning and Grounding 54 An Investigation of End-to-End Models for Robust Speech Recognition (2021) Multimodal Retrieval and Embedding 54 How Accents Confound: Probing for Accent Information in End-to-End Speech Recognition Systems (2020) Multimodal Retrieval and Embedding |
Area chair
| Area chair | Match | |
|---|---|---|
| Xiaoye Qu | 48 |
Area-chair candidates (4)
| Xiaoye Qu assigned | 48 | 15 papers | |
| Xiaoxi Li 0005 | 45 | 15 papers | |
| Hao Fei 0001 | 41 | 4 papers | |
| Hongjin Qian | 38 | 12 papers |
Candidates from the panel ranked by match score
| # | Reviewer | Match | Load | Why | |
|---|---|---|---|---|---|
| 1 | Haiyang Xu 0001 | 63 | 3 | 57 SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimization (2025) Multimodal Reasoning and Grounding 57 mPLUG-OwI2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration (2024) Multimodal Reasoning and Grounding |
|
| 2 | Minghui Fang 0002 | 55 | 3 | 54 Mitigating Hallucinations in LM-Based TTS Models via Distribution Alignment Using GFlowNets (2025) Multimodal Retrieval and Embedding 54 Speech Token Prediction via Compressed-to-fine Language Modeling for Speech Generation (2025) Multimodal Retrieval and Embedding |
|
| 3 | Qingyi Si assigned | 54 | 3 | 58 Combo of Thinking and Observing for Outside-Knowledge VQA (2023) Retrieval-Augmented and Open-Domain QA 57 Object Attribute Matters in Visual Question Answering (2024) Multimodal Reasoning and Grounding |
|
| 4 | Archiki Prasad assigned | 54 | 3 | 57 Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models (2024) Multimodal Reasoning and Grounding 54 An Investigation of End-to-End Models for Robust Speech Recognition (2021) Multimodal Retrieval and Embedding |
|
| 5 | Zihao Yue | 51 | 3 | 57 Unified Multimodal Understanding via Byte-Pair Visual Encoding (2025) Multimodal Reasoning and Grounding 54 Time-R1: Post-Training Large Vision Language Model for Temporal Video Grounding (2025) Multimodal Reasoning and Grounding |
|
| 6 | Navve Wasserman | 48 | 3 | 57 DocReRank: Single-Page Hard Negative Query Generation for Training Multi-Modal RAG Rerankers (2025) Universal Multimodal Embedding and MLLM Retrieval 57 REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmark (2025) Universal Multimodal Embedding and MLLM Retrieval |
|
| 8 | Xingyu Tan 0001 | 48 | 3 | 57 PRoH: Dynamic Planning and Reasoning over Knowledge Hypergraphs for Retrieval-Augmented Generation (2026) Multimodal and Graph RAG 51 HydraRAG: Structured Cross-Source Enhanced Large Language Model Reasoning (2025) KG-Augmented LLM Reasoning |
|
| 9 | Linchao Zhu | 45 | 3 | 57 Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models (2024) Multimodal Reasoning and Grounding 52 Attention as Selector: Unlocking VLM Attention for Long Document Page Retrieval (2026) Universal Multimodal Embedding and MLLM Retrieval |
|
| 11 | Kaixin Ma | 45 | 3 | 57 COLUMBUS: Evaluating COgnitive Lateral Understanding Through Multiple-Choice reBUSes (2025) Multimodal Reasoning and Grounding 55 Open Domain Question Answering with A Unified Knowledge Interface (2022) Multimodal and Graph RAG |
|
| 12 | Tingyu Song | 43 | 3 | 52 Rethinking Composed Image Retrieval Evaluation: A Fine-Grained Benchmark from Image Editing (2026) Universal Multimodal Embedding and MLLM Retrieval 41 VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC Videos (2025) Video and Temporal Understanding |
|
| 13 | Xinying Qian | 43 | 3 | 51 TimeR⁴ : Time-aware Retrieval-Augmented Large Language Models for Temporal Knowledge Graph Question Answering (2024) Multi-Hop and Iterative Retrieval Reasoning 38 Hyperbolic Multimodal Generative Representation Learning for Generalized Zero-Shot Multimodal Information Extraction (2026) Universal Multimodal Embedding and MLLM Retrieval |
|
| 14 | Di Wu 0054 | 42 | 3 | 56 VisRet: Visualization Improves Knowledge-Intensive Text-to-Image Retrieval (2026) Multimodal and Graph RAG 40 LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory (2025) Conversational Search and Retrieval |
|
| 15 | Liwei Wang 0009 | 42 | 3 | 40 Eliciting Knowledge from Large Pre-Trained Models for Unsupervised Knowledge-Grounded Conversation (2022) Knowledge Conflict and Faithfulness 40 MVP-Tuning: Multi-View Knowledge Retrieval with Prompt Tuning for Commonsense Reasoning (2023) Retrieval-Augmented and Open-Domain QA |
|
| 16 | Mariano Barone | 42 | 3 | 44 A Multilingual Multimodal Medical Examination Dataset for Visual Question Answering in Healthcare (2025) Retrieval-Augmented and Open-Domain QA 40 SOLVE-Med: Specialized Orchestration for Leading Vertical Experts Across Medical Specialties (2025) Retrieval-Augmented and Open-Domain QA |
|
| 17 | Yanzhao Zhang | 42 | 3 | 53 Bridging Modalities: Improving Universal Multimodal Retrieval by Multimodal Large Language Models (2025) Universal Multimodal Embedding and MLLM Retrieval 53 Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs (2025) Universal Multimodal Embedding and MLLM Retrieval |
|
| 18 | Yingyi Zhang 0001 | 42 | 3 | 57 GeoArena: Evaluating Open-World Geographic Reasoning in Large Vision-Language Models (2026) Multimodal Reasoning and Grounding 50 MemSearch-o1: Empowering Large Language Models with Reasoning-Aligned Memory Growth in Agentic Search (2026) Agentic and Deep Search RAG |
|
| 19 | Haokun Wen | 42 | 3 | 71 Differential-Perceptive and Retrieval-Augmented MLLM for Change Captioning (2024) Multimodal and Graph RAG 38 Personalized Fashion Compatibility Modeling via Metapath-guided Heterogeneous Graph Learning (2022) Universal Multimodal Embedding and MLLM Retrieval |
|
| 20 | Qinggang Zhang | 41 | 3 | 57 KnowGPT: Knowledge Graph based Prompting for Large Language Models (2024) Multimodal and Graph RAG 56 You Don't Need Pre-Built Graphs for RAG: Retrieval Augmented Generation with Adaptive Reasoning Structures (2026) Multi-Hop and Iterative Retrieval Reasoning |
|
| 21 | Xuhui Sui | 41 | 3 | 47 TimeR⁴ : Time-aware Retrieval-Augmented Large Language Models for Temporal Knowledge Graph Question Answering (2024) RL Training of Search Agents 38 Coarse-to-fine multimodal prototype network for few-shot multimodal relation extraction (2025) Universal Multimodal Embedding and MLLM Retrieval |
|
| 23 | Nancy F. Chen | 41 | 3 | 54 Rescoring Hypothesized Detections of Out-of-Vocabulary Keywords Using Subword Samples (2016) Multimodal Retrieval and Embedding 43 Multimodal Dialogue State Tracking (2022) Conversational Search and Retrieval |
|
| 24 | Weipeng Chen | 40 | 3 | 49 ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning (2025) RL Training of Search Agents 41 Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs (2025) Video and Temporal Understanding |
|
| 25 | Jinheon Baek | 40 | 3 | 56 UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularities (2026) Multimodal and Graph RAG 41 Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Tasks (2023) Retrieval-Augmented and Open-Domain QA |
|
| 26 | Xinyu Ma 0001 | 40 | 3 | 47 Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learning (2025) Agentic and Deep Search RAG 44 Knowledge Graph Retrieval-Augmented Generation for LLM-based Recommendation (2025) LLM-Derived Knowledge Infusion |
|
| 27 | Weiwei Sun 0001 | 39 | 3 | 47 Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learning (2025) Agentic and Deep Search RAG 40 ReasonRank: Empowering Passage Ranking with Strong Reasoning Ability (2026) LLM-Based Reranking |
|
| 28 | Chen Zhao 0013 | 39 | 3 | 41 SportReason: Evaluating Retrieval-Augmented Reasoning across Tables and Text for Sports Question Answering (2025) Multi-Hop and Iterative Retrieval Reasoning 40 Are Multimodal LLMs Robust Against Adversarial Perturbations? RoMMath: A Systematic Evaluation on Multimodal Math Reasoning (2025) Multimodal Evaluation and Benchmarks |
|
| 29 | Farzad Shami | 39 | 3 | 40 From Single to Multi-agent Reasoning: Advancing GeneGPT for Genomics QA (2026) Retrieval-Augmented and Open-Domain QA 35 Benchmarking Large Language Models for Knowledge Graph Validation (2026) KG-Augmented LLM Reasoning |
|
| 30 | Sheng-Chieh Lin | 39 | 3 | 53 Mm-Embed: Universal Multimodal Retrieval with Multimodal LLMS (2025) Universal Multimodal Embedding and MLLM Retrieval 52 Unifying Multimodal Retrieval via Document Screenshot Embedding (2024) Universal Multimodal Embedding and MLLM Retrieval |
|
| 31 | Xi Ye 0003 | 39 | 3 | 40 Query-Focused Retrieval Heads Improve Long-Context Reasoning and Re-ranking (2025) Multi-hop and Complex Reasoning QA 36 Effective Large Language Model Adaptation for Improved Grounding and Citation Generation (2024) Knowledge Conflict and Faithfulness |
|
| 32 | Wenlin Zhang 0001 | 39 | 3 | 50 MemSearch-o1: Empowering Large Language Models with Reasoning-Aligned Memory Growth in Agentic Search (2026) Agentic and Deep Search RAG 49 To Search or Not to Search: Aligning the Decision Boundary of Deep Search Agents via Causal Intervention (2026) Agentic and Deep Search RAG |
|
| 33 | Jiapu Wang | 38 | 3 | 58 Large-Small Model Synergy with Multimodal Fine-Grained Heuristics for Knowledge-Based Visual Question Answering (2025) Retrieval-Augmented and Open-Domain QA 41 Decomposing and Revising What Language Models Generate (2025) Multi-Hop and Iterative Retrieval Reasoning |