Demonstration venue · read-only. Every page can be browsed; the buttons that would change it are switched off. Create an account to run TaxoReview on your own data. A simulation from public publication records: the people listed were not invited and did not agree to review.
? Tutorial for this page

From Existence to Exhaustiveness: Unveiling the Compounding Failures of LLMs in Multi-answer Event Temporal Reasoning

sigir26-135 · Full research paper · Shaojuan Wu
Multi-hop and Complex Reasoning QA · 100%
Abstract

Large Language Models (LLMs) have achieved remarkable success in temporal reasoning. However, existing benchmarks predominantly adopt a "single-answer" paradigm, focusing on verifying the existence of a specific fact while overlooking the challenge of exhaustiveness. In real-world scenarios, entities often simultaneously play multiple roles or exist in multiple states within the same timeframe. To bridge this gap, we introduce MulTR, a comprehensive benchmark designed for multi-answer temporal reasoning from long unstructured contexts. Specifically, MulTR integrates structured temporal facts from Wikidata and natural language text from Wikipedia MulTR integrates structured temporal facts from Wikidata and natural language text from Wikipedia through a logic-driven synthesis process. Notably, we formulate two distinct settings, question-dependent and document-dependent, based on the presence of cue words in the question. It is designed to systematically decouple temporal reasoning capabilities from the uncertainty of the number of answers. Experiment results demonstrate that state-of-the-art models suffer from retrieval laziness, terminating the search process prematurely after locating the first valid piece of evidence. Consequently, their performance drops sharply when evaluated on strict exact match metrics. MulTR, as a diagnostic testing platform, reveal these defects and establish the rigorous standard for future research in dynamic knowledge processing. The MulTR benchmark and evaluation prompt are publicly available at https://github.com/TemporalNLP/MulTR.

Assigned reviewers

#ReviewerMatchWhy (closest evidence publications in the taxonomy)
1 Wenyu Huang
56
100 Masking in Multi-hop QA: An Analysis of How Language Models Perform with Context Permutation (2025) Multi-hop and Complex Reasoning QA
59 Prompting large language models with knowledge graphs for question answering involving long-tail facts (2025) KG-Augmented LLM Reasoning
33 Retrieval Augmented Generation with Rich Answer Encoding (2023) Retrieval-Augmented Generation › Others
2 Adam Jatowt
52
100 Do Language Models Have a Common Sense regarding Time? Revisiting Temporal Commonsense Reasoning in the Era of Large Language Models (2023) Multi-hop and Complex Reasoning QA
100 Temporal Blind Spots in Large Language Models (2024) Multi-hop and Complex Reasoning QA
100 Inferential Question Answering (2026) Multi-hop and Complex Reasoning QA

Area chair

Area chairMatch
Hongjin Qian 37
Area-chair candidates (4)
Xiaoxi Li 0005 3815 papers
Hongjin Qian assigned 3712 papers
Kam-Fai Wong 346 papers
Jun Xu 0001 342 papers

Candidates from the panel ranked by match score

#ReviewerMatchLoadWhy
1 Wenyu Huang assigned 56 3
100 Masking in Multi-hop QA: An Analysis of How Language Models Perform with Context Permutation (2025) Multi-hop and Complex Reasoning QA
59 Prompting large language models with knowledge graphs for question answering involving long-tail facts (2025) KG-Augmented LLM Reasoning
2 Adam Jatowt assigned 52 3
100 Do Language Models Have a Common Sense regarding Time? Revisiting Temporal Commonsense Reasoning in the Era of Large Language Models (2023) Multi-hop and Complex Reasoning QA
100 Temporal Blind Spots in Large Language Models (2024) Multi-hop and Complex Reasoning QA
3 Chen Zhao 0013 49 3
100 TaPERA: Enhancing Faithfulness and Interpretability in Long-Form Table QA by Content Planning and Execution-based Reasoning (2024) Multi-hop and Complex Reasoning QA
100 RobuT: A Systematic Study of Table QA Robustness Against Human-Annotated Adversarial Perturbations (2023) Multi-hop and Complex Reasoning QA
4 Yu Gu 0016 46 3
74 ArcaneQA: Dynamic Program Induction and Contextualized Encoding for Knowledge Base Question Answering (2022) Knowledge Graph Question Answering
74 Don't Generate, Discriminate: A Proposal for Grounding Language Models to Real-World Environments (2023) Knowledge Graph Question Answering
5 Qisheng Hu 46 3
100 Just What You Desire: Constrained Timeline Summarization with Self-Reflection for Enhanced Relevance (2025) Multi-hop and Complex Reasoning QA
39 Decomposition Dilemmas: Does Claim Decomposition Boost or Burden Fact-Checking Performance? (2025) Automated Fact Checking and Claim Verification
6 Qinggang Zhang 45 3
100 Knapsack Optimization-Based Schema Linking for LLM-Based Text-to-SQL Generation (2026) Multi-hop and Complex Reasoning QA
74 Hierarchy-Aware Multi-Hop Question Answering over Knowledge Graphs (2023) Knowledge Graph Question Answering
7 Ori Yoran 45 3
73 Answering Questions by Meta-Reasoning over Multiple Chains of Thought (2023) Reasoning-Intensive Retrieval and Ranking
63 Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills (2022) Multi-hop and Complex Reasoning QA
8 Yuzhong Qu 43 3
74 Enhancing Complex Question Answering over Knowledge Graphs through Evidence Pattern Retrieval (2024) Knowledge Graph Question Answering
74 Skeleton parsing for complex question answering over knowledge bases (2022) Knowledge Graph Question Answering
9 Tianzhe Zhao 43 3
68 Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond (2025) Reasoning Evaluation and Benchmarks
64 PathReasoner: Modeling Reasoning Path with Equivalent Extension for Logical Question Answering (2024) Search Agents and LLM Reasoning › Others
10 Xinying Qian 42 3
100 Beyond Timestamps: Bridging Forward and Backward Reasoning in Temporal Numerical and Relational Understanding (2026) Multi-hop and Complex Reasoning QA
45 TimeR⁴ : Time-aware Retrieval-Augmented Large Language Models for Temporal Knowledge Graph Question Answering (2024) Multi-Hop and Iterative Retrieval Reasoning
11 Jiapu Wang 42 3
63 Atomic Fact Decomposition Helps Attributed Question Answering (2025) Multi-Hop and Iterative Retrieval Reasoning
63 Decomposing and Revising What Language Models Generate (2025) Multi-Hop and Iterative Retrieval Reasoning
12 Xingyu Tan 0001 40 3
44 MemoTime: Memory-Augmented Temporal Knowledge Graph Enhanced Large Language Model Reasoning (2026) KG-Augmented LLM Reasoning
44 Paths-over-Graph: Knowledge Graph Empowered Large Language Model Reasoning (2025) KG-Augmented LLM Reasoning
13 Idan Szpektor 40 3
63 MDCure: A Scalable Pipeline for Multi-Document Instruction-Following (2025) Multi-hop and Complex Reasoning QA
54 SIGIR 2016 Workshop WebQA II: Web Question Answering Beyond Factoids (2016) Retrieval-Augmented and Open-Domain QA
14 Farzad Shami 40 3
54 From Single to Multi-agent Reasoning: Advancing GeneGPT for Genomics QA (2026) Retrieval-Augmented and Open-Domain QA
40 Fact Verification in Knowledge Graphs Using LLMs (2025) KG-Augmented LLM Reasoning
15 Xi Ye 0003 39 3
100 Connecting Attributions and QA Model Behavior on Realistic Counterfactuals (2021) Multi-hop and Complex Reasoning QA
74 RNG-KBQA: Generation Augmented Iterative Ranking for Knowledge Base Question Answering (2022) Knowledge Graph Question Answering
16 Xiaolong Jin 0001 39 3
48 Stepwise Reasoning for Multi-Relation Question Answering over Knowledge Graph with Weak Supervision (2020) Knowledge Graph Question Answering
48 Hierarchical Query Graph Generation for Complex Question Answering over Knowledge Graph (2020) Knowledge Graph Question Answering
17 Armin Toroghi 39 3
48 Bayesian Inference with Complex Knowledge Graph Evidence (2024) Knowledge Graph Question Answering
44 CommonWhy: A Dataset for Evaluating Entity-Based Causal Commonsense Reasoning in Large Language Models (2026) Knowledge Graph Question Answering
18 Ben Liu 0002 38 3
40 SymAgent: A Neural-Symbolic Self-Learning Agent Framework for Complex Reasoning over Knowledge Graphs (2025) Reasoning Evaluation and Benchmarks
39 MoodAngels: A Retrieval-augmented Multi-agent Framework for Psychiatry Diagnosis (2025) RAG for Domain Applications
19 Jinheon Baek 38 3
64 An Empirical Study of Multilingual Reasoning Distillation for Question Answering (2024) Multi-hop and Complex Reasoning QA
51 Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity (2024) Multi-Hop and Iterative Retrieval Reasoning
21 Zhe Xu 0009 38 3
100 MindGYM: What Matters in Question Synthesis for Thinking-Centric Fine-Tuning? (2025) Multi-hop and Complex Reasoning QA
42 Dynamic Demonstration Retrieval and Cognitive Understanding for Emotional Support Conversation (2024) Conversational Search and Retrieval
23 Jing Liu 0022 37 3
54 A Regularized Competition Model for Question Difficulty Estimation in Community Question Answering Services (2014) Retrieval-Augmented and Open-Domain QA
47 Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation (2025) Knowledge Conflict and Faithfulness
24 Wonduk Seo 37 3
40 A New Query Expansion Approach for Enhancing Information Retrieval via Agent-Mediated Dialogic Inquiry (2026) Query Reformulation and Expansion
37 SPIO: Ensemble and Selective Strategies via LLM-Based Multi-Agent Planning in Automated Data Science (2026) Multi-Agent and Tool-Use Orchestration
25 Daiting Shi 37 3
100 A Question-Oriented Propagation Network for News Reading Comprehension (2022) Multi-hop and Complex Reasoning QA
42 Utility-Focused LLM Annotation for Retrieval and Retrieval-Augmented Generation (2025) Evidence Selection and Context Refinement
26 Shu Zhou 0002 36 3
60 Enhancing Extractive Question Answering in Multiparty Dialogues with Logical Inference Memory Network (2025) Conversational Search and Retrieval
49 SCORE-RAG: Self-Correcting Exploration-Exploitation Retrieval for Multi-hop Question Answering (2026) Multi-Hop and Iterative Retrieval Reasoning
27 Kaixin Ma 36 3
49 Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering (2021) Retrieval-Augmented and Open-Domain QA
48 Open Domain Question Answering with A Unified Knowledge Interface (2022) Multimodal and Graph RAG
28 Liwei Wang 0009 36 3
100 Making Long-Context Language Models Better Multi-Hop Reasoners (2024) Multi-hop and Complex Reasoning QA
54 MVP-Tuning: Multi-View Knowledge Retrieval with Prompt Tuning for Commonsense Reasoning (2023) Retrieval-Augmented and Open-Domain QA
29 Weizhi Xu 0002 36 3
39 Heterogeneous Graph Reasoning for Fact Checking over Texts and Tables (2024) Automated Fact Checking and Claim Verification
39 Evidence-aware Fake News Detection with Graph Neural Networks (2022) Automated Fact Checking and Claim Verification
30 Mariano Barone 36 3
46 SOLVE-Med: Specialized Orchestration for Leading Vertical Experts Across Medical Specialties (2025) Retrieval-Augmented and Open-Domain QA
41 A Multilingual Multimodal Medical Examination Dataset for Visual Question Answering in Healthcare (2025) Retrieval-Augmented and Open-Domain QA
31 Falk Scholer 36 3
77 WikiHowQA: A Comprehensive Benchmark for Multi-Document Non-Factoid Question Answering (2023) Multi-hop and Complex Reasoning QA
44 Document Summarization for Answering Non-Factoid Queries (2018) Retrieval-Augmented and Open-Domain QA
32 Qingyi Si 35 3
70 Are Large Language Models Table-based Fact-Checkers? (2024) Automated Fact Checking and Claim Verification
43 Combo of Thinking and Observing for Outside-Knowledge VQA (2023) Retrieval-Augmented and Open-Domain QA