mKG-RAG: Leveraging Multimodal Knowledge Graphs in Retrieval-Augmented Generation for Knowledge-intensive VQA
sigir26-156 · Full research paper · Xu Yuan, Liangbo Ning, Qingqing Ye, Wenqi Fan, Qing Li
KG-Augmented LLM Reasoning · 33% Multimodal and Graph RAG · 17% RAG for Domain Applications · 17% Multimodal Retrieval and Embedding · 17%
Abstract
Retrieval-Augmented Generation (RAG) has emerged as an effective paradigm for expanding the knowledge capacity of Multimodal Large Language Models (MLLMs) by incorporating external knowledge sources into the generation process, and has been widely adopted for knowledge-based Visual Question Answering (VQA). Despite impressive advancements, vanilla RAG-based VQA methods that rely on unstructured documents and overlook the structural relations among knowledge elements frequently introduce irrelevant or misleading content, degrading answer accuracy and reliability. To overcome these challenges, a promising solution is to integrate multimodal knowledge graphs (KGs) into RAG-based VQA frameworks, thereby enhancing generation through structured multimodal knowledge. To this end, this paper proposes mKG-RAG, a novel retrieval-augmented generation framework built upon multimodal KGs for knowledge-intensive VQA tasks. Specifically, mKG-RAG leverages MLLM-driven graph extraction and vision-text matching to distill semantically consistent, modality-complementary entities and relations from multimodal documents, constructing high-quality multimodal KGs as structured knowledge representations. Furthermore, a dual-stage retrieval strategy equipped with a query-aware multimodal retriever is introduced to improve retrieval efficiency while progressively refining precision. Comprehensive experiments demonstrate that our approach significantly outperforms existing approaches and sets new state-of-the-art results for knowledge-based VQA. The code is available at https://github.com/xandery-geek/mKG-RAG.
Assigned reviewers
| # | Reviewer | Match | Why (closest evidence publications in the taxonomy) | |
|---|---|---|---|---|
| 1 | Xingyu Tan 0001 | 65 | 73 PRoH: Dynamic Planning and Reasoning over Knowledge Hypergraphs for Retrieval-Augmented Generation (2026) Multimodal and Graph RAG 69 HydraRAG: Structured Cross-Source Enhanced Large Language Model Reasoning (2025) KG-Augmented LLM Reasoning 69 Paths-over-Graph: Knowledge Graph Empowered Large Language Model Reasoning (2025) KG-Augmented LLM Reasoning |
|
| 2 | Archiki Prasad | 52 | 48 Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models (2024) Multimodal Reasoning and Grounding 45 GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMs (2026) Language Model Pre-training and Adaptation 44 How Accents Confound: Probing for Accent Information in End-to-End Speech Recognition Systems (2020) Multimodal Retrieval and Embedding |
|
| 3 | Qingyi Si | 51 | 56 Cross-modality Multiple Relations Learning for Knowledge-based Visual Question Answering (2024) Knowledge Graph Question Answering 53 Combo of Thinking and Observing for Outside-Knowledge VQA (2023) Retrieval-Augmented and Open-Domain QA 48 Object Attribute Matters in Visual Question Answering (2024) Multimodal Reasoning and Grounding |
Area chair
| Area chair | Match | |
|---|---|---|
| Xiaoye Qu | 44 |
Area-chair candidates (3)
| Xiaoye Qu assigned | 44 | 15 papers | |
| Xiaoxi Li 0005 | 39 | 15 papers | |
| Hao Fei 0001 | 37 | 4 papers |
Candidates from the panel ranked by match score
| # | Reviewer | Match | Load | Why | |
|---|---|---|---|---|---|
| 1 | Xingyu Tan 0001 assigned | 65 | 3 | 73 PRoH: Dynamic Planning and Reasoning over Knowledge Hypergraphs for Retrieval-Augmented Generation (2026) Multimodal and Graph RAG 69 HydraRAG: Structured Cross-Source Enhanced Large Language Model Reasoning (2025) KG-Augmented LLM Reasoning |
|
| 2 | Farzad Shami | 57 | 3 | 57 Benchmarking Large Language Models for Knowledge Graph Validation (2026) KG-Augmented LLM Reasoning 57 Fact Verification in Knowledge Graphs Using LLMs (2025) KG-Augmented LLM Reasoning |
|
| 3 | Armin Toroghi | 55 | 3 | 57 CommonWhy: A Dataset for Evaluating Entity-Based Causal Commonsense Reasoning in Large Language Models (2026) Knowledge Graph Question Answering 57 LLM-based Typed Hyperresolution for Commonsense Reasoning with Knowledge Bases (2025) KG-Augmented LLM Reasoning |
|
| 4 | Haiyang Xu 0001 | 53 | 3 | 48 MaVEn: An Effective Multi-granularity Hybrid Visual Encoding Framework for Multimodal Large Language Model (2024) Multimodal Reasoning and Grounding 48 mPLUG-OwI2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration (2024) Multimodal Reasoning and Grounding |
|
| 5 | Archiki Prasad assigned | 52 | 3 | 48 Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models (2024) Multimodal Reasoning and Grounding 45 GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMs (2026) Language Model Pre-training and Adaptation |
|
| 6 | Qingyi Si assigned | 51 | 3 | 56 Cross-modality Multiple Relations Learning for Knowledge-based Visual Question Answering (2024) Knowledge Graph Question Answering 53 Combo of Thinking and Observing for Outside-Knowledge VQA (2023) Retrieval-Augmented and Open-Domain QA |
|
| 7 | Mariano Barone | 48 | 3 | 44 Combating Biomedical Misinformation through Multi-modal Claim Detection and Evidence-based Verification (2025) RAG for Domain Applications 43 Combining Evidence and Reasoning for Biomedical Fact-Checking (2025) Automated Fact Checking and Claim Verification |
|
| 8 | Qinggang Zhang COI: co-author | 47 | 3 | 73 KnowGPT: Knowledge Graph based Prompting for Large Language Models (2024) Multimodal and Graph RAG 53 Cost-efficient Knowledge-based Question Answering with Large Language Models (2024) Multi-hop and Complex Reasoning QA |
|
| 9 | Jiapu Wang | 47 | 3 | 54 Unifying Large Language Models and Knowledge Graphs: A Roadmap (2024) KG-Augmented LLM Reasoning 54 From implicit parameters to explicit knowledge graphs: A structured knowledge recall framework for bidirectional generalization in LLMs (2027) KG-Augmented LLM Reasoning |
|
| 10 | Xinying Qian | 47 | 3 | 64 TimeR⁴ : Time-aware Retrieval-Augmented Large Language Models for Temporal Knowledge Graph Question Answering (2024) Multi-Hop and Iterative Retrieval Reasoning 37 Beyond Timestamps: Bridging Forward and Backward Reasoning in Temporal Numerical and Relational Understanding (2026) Multi-hop and Complex Reasoning QA |
|
| 11 | Zihao Yue | 45 | 3 | 48 Unified Multimodal Understanding via Byte-Pair Visual Encoding (2025) Multimodal Reasoning and Grounding 47 Time-R1: Post-Training Large Vision Language Model for Temporal Video Grounding (2025) Multimodal Reasoning and Grounding |
|
| 12 | Navve Wasserman | 45 | 3 | 48 DocReRank: Single-Page Hard Negative Query Generation for Training Multi-Modal RAG Rerankers (2025) Universal Multimodal Embedding and MLLM Retrieval 48 REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmark (2025) Universal Multimodal Embedding and MLLM Retrieval |
|
| 13 | Kaixin Ma | 45 | 3 | 57 Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering (2021) Retrieval-Augmented and Open-Domain QA 53 Open Domain Question Answering with A Unified Knowledge Interface (2022) Multimodal and Graph RAG |
|
| 14 | Minghui Fang 0002 | 44 | 3 | 44 VoxpopuliTTS: a large-scale multilingual TTS corpus for zero-shot speech generation (2025) Multimodal Retrieval and Embedding 44 OmniSep: Unified Omni-Modality Sound Separation with Query-Mixup (2025) Multimodal Retrieval and Embedding |
|
| 16 | Di Wu 0054 | 43 | 3 | 48 VisRet: Visualization Improves Knowledge-Intensive Text-to-Image Retrieval (2026) Multimodal and Graph RAG 39 LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory (2025) Conversational Search and Retrieval |
|
| 17 | Yu Gu 0016 | 43 | 3 | 69 HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models (2024) KG-Augmented LLM Reasoning 53 Middleware for LLMs: Tools Are Instrumental for Language Agents in Complex Environments (2024) KG-Augmented LLM Reasoning |
|
| 18 | Ben Liu 0002 | 43 | 3 | 51 Plan Then Retrieve: Reinforcement Learning-Guided Complex Reasoning over Knowledge Graphs (2026) RL Training of Search Agents 50 SymAgent: A Neural-Symbolic Self-Learning Agent Framework for Complex Reasoning over Knowledge Graphs (2025) Reasoning Evaluation and Benchmarks |
|
| 19 | Jinheon Baek | 42 | 3 | 57 KALA: Knowledge-Augmented Language Model Adaptation (2022) KG-Augmented LLM Reasoning 51 Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Tasks (2023) Retrieval-Augmented and Open-Domain QA |
|
| 20 | Xuhui Sui | 41 | 3 | 57 TimeR⁴ : Time-aware Retrieval-Augmented Large Language Models for Temporal Knowledge Graph Question Answering (2024) RL Training of Search Agents 37 Multimodal Knowledge Graph Error Detection with Disentanglement VAE and Multi-Grained Triplet Confidence (2025) Knowledge Graph Embedding and Completion |
|
| 21 | Xiaolong Jin 0001 | 41 | 3 | 57 A New Pipeline for Knowledge Graph Reasoning Enhanced by Large Language Models Without Fine-Tuning (2024) KG-Augmented LLM Reasoning 40 Hierarchical Query Graph Generation for Complex Question Answering over Knowledge Graph (2020) Knowledge Graph Question Answering |
|
| 22 | Wenyu Huang | 40 | 3 | 60 Prompting large language models with knowledge graphs for question answering involving long-tail facts (2025) KG-Augmented LLM Reasoning 37 Masking in Multi-hop QA: An Analysis of How Language Models Perform with Context Permutation (2025) Multi-hop and Complex Reasoning QA |
|
| 23 | Shu Zhou 0002 | 40 | 3 | 64 DynKGRAG: A framework for dynamic knowledge graph construction and multi-hop querying over numerical and textual data (2026) Multi-Hop and Iterative Retrieval Reasoning 57 When MLLMs Meet ICH: A visual retrieval-augmented generation-based method for intangible cultural heritage image recognition-take Shadow Puppetry as a case (2026) Universal Multimodal Embedding and MLLM Retrieval |
|
| 24 | Linchao Zhu | 40 | 3 | 48 Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models (2024) Multimodal Reasoning and Grounding 43 Attention as Selector: Unlocking VLM Attention for Long Document Page Retrieval (2026) Universal Multimodal Embedding and MLLM Retrieval |
|
| 25 | Tingyu Song | 40 | 3 | 43 Rethinking Composed Image Retrieval Evaluation: A Fine-Grained Benchmark from Image Editing (2026) Universal Multimodal Embedding and MLLM Retrieval 40 LimRank: Less is More for Reasoning-Intensive Information Reranking (2025) Evidence Selection and Context Refinement |
|
| 26 | Liwei Wang 0009 | 39 | 3 | 41 MVP-Tuning: Multi-View Knowledge Retrieval with Prompt Tuning for Commonsense Reasoning (2023) Retrieval-Augmented and Open-Domain QA 40 Eliciting Knowledge from Large Pre-Trained Models for Unsupervised Knowledge-Grounded Conversation (2022) Knowledge Conflict and Faithfulness |
|
| 27 | Chen Zhao 0013 | 39 | 3 | 41 A dataset and baselines for sequential open-domain question answering (2018) Retrieval-Augmented and Open-Domain QA 41 SportReason: Evaluating Retrieval-Augmented Reasoning across Tables and Text for Sports Question Answering (2025) Multi-Hop and Iterative Retrieval Reasoning |
|
| 29 | Yuzhong Qu | 39 | 3 | 40 Enhancing Complex Question Answering over Knowledge Graphs through Evidence Pattern Retrieval (2024) Knowledge Graph Question Answering 40 Skeleton parsing for complex question answering over knowledge bases (2022) Knowledge Graph Question Answering |
|
| 30 | Nancy F. Chen | 38 | 3 | 44 Rescoring Hypothesized Detections of Out-of-Vocabulary Keywords Using Subword Samples (2016) Multimodal Retrieval and Embedding 39 Multimodal Dialogue State Tracking (2022) Conversational Search and Retrieval |
|
| 31 | Dan Luo 0004 | 38 | 3 | 57 From Anchors to Answers: A Novel Node Tokenizer for Integrating Graph Structure into Large Language Models (2025) KG-Augmented LLM Reasoning 53 PAGER: Proactive Monitoring Agent for Enterprise AI Assistant (2026) Reasoning Evaluation and Benchmarks |
|
| 32 | Jimeng Sun 0001 | 38 | 3 | 79 A Survey on Unifying Large Language Models and Knowledge Graphs for Biomedicine and Healthcare (2025) KG-Augmented LLM Reasoning 73 Reasoning-Enhanced Healthcare Predictions with Knowledge Graph Community Retrieval (2025) Multimodal and Graph RAG |