Towards Knowledgeable Deep Research: Framework and Benchmark
Abstract
Deep Research (DR) requires LLM agents to autonomously perform multi-step information seeking, processing, and reasoning to generate comprehensive reports. In contrast to existing studies that mainly focus on unstructured web content, a more challenging DR task should additionally utilize structured knowledge to provide a solid data foundation, facilitate quantitative computation, and lead to in-depth analyses. In this paper, we refer to this novel task as Knowledgeable Deep Research (KDR), which requires DR agents to generate reports with both structured and unstructured knowledge. Furthermore, we propose the Hybrid Knowledge Analysis framework (HKA), a multi-agent architecture that reasons over both kinds of knowledge and integrates the texts, figures, and tables into coherent multimodal reports. The key design is the Structured Knowledge Analyzer, which utilizes both coding and vision-language models to produce figures, tables, and corresponding insights. To support systematic evaluation, we construct KDR-Bench, which covers 9 domains, includes 41 expert-level questions, and incorporates a large number of structured knowledge resources (e.g., 1,252 tables). We further annotate the main conclusions and key points for each question and propose three categories of evaluation metrics including general-purpose, knowledge-centric, and vision-enhanced ones. Experimental results demonstrate that HKA consistently outperforms most existing DR agents on general-purpose and knowledge-centric metrics, and even surpasses the Gemini DR agent on vision-enhanced metrics, highlighting its effectiveness in deep, structure-aware knowledge analysis. Finally, we hope this work can serve as a new foundation for structured knowledge analysis in DR agents and facilitate future multimodal DR studies.
Assigned reviewers
| # | Reviewer | Match | Why (closest evidence publications in the taxonomy) | |
|---|---|---|---|---|
| 1 | Jialong Wu 0007 | 57 | 73 WebWalker: Benchmarking LLMs in Web Traversal (2025) Agentic and Deep Search RAG 72 OmniThink: Expanding Knowledge Boundaries in Machine Writing through Thinking (2025) Deep Research and Report Generation 71 WebDancer: Towards Autonomous Information Seeking Agency (2025) Deep Research and Report Generation |
|
| 2 | Wenlin Zhang 0001 | 49 | 87 MemSearch-o1: Empowering Large Language Models with Reasoning-Aligned Memory Growth in Agentic Search (2026) Agentic and Deep Search RAG 73 To Search or Not to Search: Aligning the Decision Boundary of Deep Search Agents via Causal Intervention (2026) Agentic and Deep Search RAG 69 Personalized Deep Research: A User-Centric Framework, Dataset, and Hybrid Evaluation for Knowledge Discovery (2026) Deep Research and Report Generation |
|
| 3 | Yingyi Zhang 0001 | 47 | 87 MemSearch-o1: Empowering Large Language Models with Reasoning-Aligned Memory Growth in Agentic Search (2026) Agentic and Deep Search RAG 71 Personalized Deep Research: A User-Centric Framework, Dataset, and Hybrid Evaluation for Knowledge Discovery (2026) Deep Research and Report Generation 60 To Search or Not to Search: Aligning the Decision Boundary of Deep Search Agents via Causal Intervention (2026) Agentic and Deep Search RAG |
Area chair
| Area chair | Match | |
|---|---|---|
| Xiaoxi Li 0005 | 59 |
Area-chair candidates (6)
| Xiaoxi Li 0005 assigned | 59 | 15 papers | |
| Hongjin Qian | 49 | 12 papers | |
| Jun Xu 0001 COI: co-author | 42 | 2 papers | |
| Xiaoxue Zang | 40 | 4 papers | |
| Fernando Diaz 0001 | 39 | 12 papers | |
| Pengyue Jia COI: co-author | 38 | 10 papers |
Candidates from the panel ranked by match score
| # | Reviewer | Match | Load | Why | |
|---|---|---|---|---|---|
| 2 | Jialong Wu 0007 assigned | 57 | 3 | 73 WebWalker: Benchmarking LLMs in Web Traversal (2025) Agentic and Deep Search RAG 72 OmniThink: Expanding Knowledge Boundaries in Machine Writing through Thinking (2025) Deep Research and Report Generation |
|
| 3 | Wenlin Zhang 0001 assigned | 49 | 3 | 87 MemSearch-o1: Empowering Large Language Models with Reasoning-Aligned Memory Growth in Agentic Search (2026) Agentic and Deep Search RAG 73 To Search or Not to Search: Aligning the Decision Boundary of Deep Search Agents via Causal Intervention (2026) Agentic and Deep Search RAG |
|
| 5 | Kaixin Ma | 47 | 3 | 60 Retrieval-augmented GUI Agents with Generative Guidelines (2025) Agentic and Deep Search RAG 55 Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Models (2024) Evidence Selection and Context Refinement |
|
| 6 | Yingyi Zhang 0001 assigned | 47 | 3 | 87 MemSearch-o1: Empowering Large Language Models with Reasoning-Aligned Memory Growth in Agentic Search (2026) Agentic and Deep Search RAG 71 Personalized Deep Research: A User-Centric Framework, Dataset, and Hybrid Evaluation for Knowledge Discovery (2026) Deep Research and Report Generation |
|
| 7 | Weiwei Sun 0001 | 45 | 3 | 60 Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learning (2025) Agentic and Deep Search RAG 56 Direct Retrieval-augmented Optimization: Synergizing Knowledge Selection and Language Models (2026) Evidence Selection and Context Refinement |
|
| 8 | Yujia Zhou 0002 | 43 | 3 | 57 RetroLLM: Empowering Large Language Models to Retrieve Fine-grained Evidence within Generation (2025) Evidence Selection and Context Refinement 56 CorpusLM: Towards a Unified Language Model on Corpus for Knowledge-Intensive Tasks (2024) Generative Retrieval |
|
| 9 | Navve Wasserman | 43 | 3 | 48 DocReRank: Single-Page Hard Negative Query Generation for Training Multi-Modal RAG Rerankers (2025) Universal Multimodal Embedding and MLLM Retrieval 48 REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmark (2025) Universal Multimodal Embedding and MLLM Retrieval |
|
| 10 | Jing Liu 0022 | 43 | 3 | 56 REAR: A Relevance-Aware Retrieval-Augmented Framework for Open-Domain Question Answering (2024) Evidence Selection and Context Refinement 55 Reinforced Informativeness Optimization for Long-Form Retrieval-Augmented Generation (2026) Evidence Selection and Context Refinement |
|
| 11 | Yuchen Li 0006 | 43 | 3 | 61 Retain to Refine: Adaptive Online Question Answering via Query Routing and Long-Short Memory (2026) Agentic and Deep Search RAG 56 Enhancing Retrieval-Augmented Generation via Evidence Tree Search (2025) Evidence Selection and Context Refinement |
|
| 12 | Farzad Shami | 42 | 3 | 45 GROKE: Vision-Free Navigation Instruction Evaluation via Graph Reasoning on OpenStreetMap (2026) Reasoning Evaluation and Benchmarks 41 From Single to Multi-agent Reasoning: Advancing GeneGPT for Genomics QA (2026) Retrieval-Augmented and Open-Domain QA |
|
| 13 | Wonduk Seo | 42 | 3 | 45 Automated Visualization Code Synthesis via Multi-path Reasoning and Feedback-Driven Optimizations (2026) Reasoning Evaluation and Benchmarks 44 SPIO: Ensemble and Selective Strategies via LLM-Based Multi-Agent Planning in Automated Data Science (2026) Multi-Agent and Tool-Use Orchestration |
|
| 14 | Yu Gu 0016 | 42 | 3 | 72 Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge (2025) Deep Research and Report Generation 45 AgentBench: Evaluating LLMs as Agents (2024) Reasoning Evaluation and Benchmarks |
|
| 15 | Shuaiqiang Wang COI: co-author | 42 | 3 | 56 Enhancing Retrieval-Augmented Generation via Evidence Tree Search (2025) Evidence Selection and Context Refinement 47 Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learning (2025) Multi-Hop and Iterative Retrieval Reasoning |
|
| 16 | Xinyu Ma 0001 COI: co-author | 42 | 3 | 60 Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learning (2025) Agentic and Deep Search RAG 56 Direct Retrieval-augmented Optimization: Synergizing Knowledge Selection and Language Models (2026) Evidence Selection and Context Refinement |
|
| 18 | Chen Zhao 0013 | 42 | 3 | 72 Beyond Single-shot Writing: Deep Research Agents are Unreliable at Multi-turn Report Revision (2026) Deep Research and Report Generation 57 SportReason: Evaluating Retrieval-Augmented Reasoning across Tables and Text for Sports Question Answering (2025) Multi-Hop and Iterative Retrieval Reasoning |
|
| 19 | Weipeng Chen | 41 | 3 | 61 ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning (2025) RL Training of Search Agents 57 RichRAG: Crafting Rich Responses for Multi-faceted Queries in Retrieval-Augmented Generation (2025) Evidence Selection and Context Refinement |
|
| 20 | Daiting Shi COI: co-author | 41 | 3 | 56 Utility-Focused LLM Annotation for Retrieval and Retrieval-Augmented Generation (2025) Evidence Selection and Context Refinement 44 Reinforced Efficient Reasoning via Semantically Diverse Exploration (2026) RL Training of Search Agents |
|
| 21 | Xingyu Tan 0001 | 40 | 3 | 42 Paths-over-Graph: Knowledge Graph Empowered Large Language Model Reasoning (2025) KG-Augmented LLM Reasoning 42 HydraRAG: Structured Cross-Source Enhanced Large Language Model Reasoning (2025) KG-Augmented LLM Reasoning |
|
| 22 | Tingyu Song | 40 | 3 | 56 LimRank: Less is More for Reasoning-Intensive Information Reranking (2025) Evidence Selection and Context Refinement 40 Rethinking Reasoning-Intensive Retrieval: Evaluating and Advancing Retrievers in Agentic Search Systems (2026) Reasoning-Intensive Retrieval and Ranking |
|
| 23 | Xi Ye 0003 | 40 | 3 | 72 Beyond Single-shot Writing: Deep Research Agents are Unreliable at Multi-turn Report Revision (2026) Deep Research and Report Generation 45 MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning (2024) Reasoning Evaluation and Benchmarks |
|
| 24 | Haritz Puerto | 40 | 3 | 45 MetaQA: Combining Expert Agents for Multi-Skill Question Answering (2023) Retrieval-Augmented and Open-Domain QA 41 Code Prompting Elicits Conditional Reasoning Abilities in Text+Code LLMs (2024) Reward Modeling and Process Supervision |
|
| 25 | Ori Yoran | 40 | 3 | 57 Making Retrieval-Augmented Language Models Robust to Irrelevant Context (2024) Evidence Selection and Context Refinement 46 Answering Questions by Meta-Reasoning over Multiple Chains of Thought (2023) Reasoning-Intensive Retrieval and Ranking |
|
| 27 | Qinggang Zhang | 40 | 3 | 44 You Don't Need Pre-Built Graphs for RAG: Retrieval Augmented Generation with Adaptive Reasoning Structures (2026) Multi-Hop and Iterative Retrieval Reasoning 42 FaithfulRAG: Fact-Level Conflict Modeling for Context-Faithful Retrieval-Augmented Generation (2025) Knowledge Conflict and Faithfulness |
|
| 29 | Han Li 0005 | 39 | 3 | 56 QE-RAG: A Robust Retrieval-Augmented Generation Benchmark for Query Entry Errors (2026) Evidence Selection and Context Refinement 44 Towards Context-aware Reasoning-enhanced Generative Searching in E-commerce (2026) Recommendation with Search and Ranking |
|
| 30 | Jinheon Baek | 39 | 3 | 57 Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Tasks (2023) Retrieval-Augmented and Open-Domain QA 47 Efficient Long Context Language Model Retrieval with Compression (2025) Learned Sparse Retrieval Efficiency |
|
| 31 | Peitian Zhang | 39 | 3 | 87 Search-o1: Agentic Search-Enhanced Large Reasoning Models (2025) Agentic and Deep Search RAG 50 A Multi-Task Embedder For Retrieval Augmented LLMs (2024) Evidence Selection and Context Refinement |
|
| 32 | Mariano Barone | 39 | 3 | 45 SOLVE-Med: Specialized Orchestration for Leading Vertical Experts Across Medical Specialties (2025) Retrieval-Augmented and Open-Domain QA 41 Combining Evidence and Reasoning for Biomedical Fact-Checking (2025) Automated Fact Checking and Claim Verification |
|
| 33 | Jiapu Wang | 39 | 3 | 57 Atomic Fact Decomposition Helps Attributed Question Answering (2025) Multi-Hop and Iterative Retrieval Reasoning 57 Decomposing and Revising What Language Models Generate (2025) Multi-Hop and Iterative Retrieval Reasoning |
|
| 34 | Venktesh V | 38 | 3 | 55 The Surprising Effectiveness of Rankers trained on Expanded Queries (2024) Evidence Selection and Context Refinement 52 Reproducing Adaptive Reranking for Reasoning-Intensive IR (2026) Evidence Selection and Context Refinement |
|
| 35 | Wenyu Huang | 38 | 3 | 42 Masking in Multi-hop QA: An Analysis of How Language Models Perform with Context Permutation (2025) Multi-hop and Complex Reasoning QA 41 Prompting large language models with knowledge graphs for question answering involving long-tail facts (2025) KG-Augmented LLM Reasoning |