Demonstration venue · read-only. Every page can be browsed; the buttons that would change it are switched off. Create an account to run TaxoReview on your own data. A simulation from public publication records: the people listed were not invited and did not agree to review.
? Tutorial for this page

FollowTable: A Benchmark for Instruction-Following Table Retrieval

sigir26-134 · Full research paper · Rihui Jin, Yuchen Lu, Ting Zhang, Jun Wang, Kuicai Dong, Zhaocheng Du, Dongping Liu, Gang Wang, Yong Liu, Guilin Qi
Multi-hop and Complex Reasoning QA · 50% Domain-Specific and Structured Retri · 25% Evaluation Resources and Tools · 25%
Abstract

Table Retrieval (TR) has traditionally been formulated as an ad-hoc retrieval problem, where relevance is primarily determined by topical semantic similarity. With the growing adoption of LLM-based agentic systems, access to structured data is increasingly instruction-driven, where relevance is conditional on explicit content and schema constraints rather than topical similarity alone. We therefore formalize Instruction-Following Table Retrieval (IFTR), a new task that requires models to jointly satisfy topical relevance and fine-grained instruction constraints. We identify two core challenges in IFTR: (i) sensitivity to content scope, such as inclusion and exclusion constraints, and (ii) awareness of schema-grounded requirements, including column semantics and representation granularity--capabilities largely absent in existing retrievers. To support systematic evaluation, we introduce FollowTable, the first large-scale benchmark for IFTR, constructed via a taxonomy-driven annotation pipeline. We further propose a new metric, termed the Instruction Responsiveness Score, to evaluate whether retrieval rankings consistently adapt to user instructions relative to a topic-only baseline. Our results indicate that existing retrieval models struggle to follow fine-grained instructions over tabular data. In particular, they exhibit systematic biases toward surface-level semantic cues and remain limited in handling schema-grounded constraints, highlighting substantial room for future improvements.

Assigned reviewers

#ReviewerMatchWhy (closest evidence publications in the taxonomy)
1 Wenyu Huang
55
67 Masking in Multi-hop QA: An Analysis of How Language Models Perform with Context Permutation (2025) Multi-hop and Complex Reasoning QA
51 Prompting large language models with knowledge graphs for question answering involving long-tail facts (2025) KG-Augmented LLM Reasoning
32 Retrieval Augmented Generation with Rich Answer Encoding (2023) Retrieval-Augmented Generation › Others
2 Andrew Parry
42
48 Constructing and Evaluating Declarative RAG Pipelines in PyTerrier (2025) Retrieval Toolkits and Reproducibility
47 Corpus Subsampling: Estimating the Effectiveness of Neural Retrieval Models on Large Corpora (2025) Evaluation Resources and Tools
40 Variations in Relevance Judgments and the Shelf Life of Test Collections (2025) Test Collection Construction and Pooling

Area chair

Area chairMatch
Zhijing Wu 0001 41
Area-chair candidates (4)
Zhijing Wu 0001 assigned 419 papers
Hongjin Qian COI: co-author 4012 papers
Xiaoxi Li 0005 3915 papers
Chenyan Xiong 3815 papers

Candidates from the panel ranked by match score

#ReviewerMatchLoadWhy
1 Adam Jatowt COI: co-author 56 3
68 ArchivalQA: A Large-scale Benchmark Dataset for Open-Domain Question Answering over Historical News Collections (2022) Multi-hop and Complex Reasoning QA
67 Inferential Question Answering (2026) Multi-hop and Complex Reasoning QA
2 Wenyu Huang assigned 55 3
67 Masking in Multi-hop QA: An Analysis of How Language Models Perform with Context Permutation (2025) Multi-hop and Complex Reasoning QA
51 Prompting large language models with knowledge graphs for question answering involving long-tail facts (2025) KG-Augmented LLM Reasoning
3 Chen Zhao 0013 51 3
67 Multi-Step Reasoning Over Unstructured Text with Beam Dense Retrieval (2021) Multi-hop and Complex Reasoning QA
67 TaPERA: Enhancing Faithfulness and Interpretability in Long-Form Table QA by Content Planning and Execution-based Reasoning (2024) Multi-hop and Complex Reasoning QA
4 Farzad Shami 48 3
50 Doctron: A Web-based Collaborative Annotation Tool for Ground Truth Creation in IR (2025) Retrieval Toolkits and Reproducibility
45 From Single to Multi-agent Reasoning: Advancing GeneGPT for Genomics QA (2026) Retrieval-Augmented and Open-Domain QA
5 Yu Gu 0016 47 3
65 ArcaneQA: Dynamic Program Induction and Contextualized Encoding for Knowledge Base Question Answering (2022) Knowledge Graph Question Answering
65 Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases (2021) Knowledge Graph Question Answering
6 Mark D. Smucker 47 3
47 Evaluating sentence-level relevance feedback for high-recall information retrieval (2020) Evaluation Resources and Tools
47 A Preference Judgment Tool for Authoritative Assessment (2023) Evaluation Resources and Tools
7 Yuzhong Qu COI: co-author 46 3
65 Enhancing Complex Question Answering over Knowledge Graphs through Evidence Pattern Retrieval (2024) Knowledge Graph Question Answering
65 Skeleton parsing for complex question answering over knowledge bases (2022) Knowledge Graph Question Answering
8 Qisheng Hu 46 3
67 Just What You Desire: Constrained Timeline Summarization with Self-Reflection for Enhanced Relevance (2025) Multi-hop and Complex Reasoning QA
34 Decomposition Dilemmas: Does Claim Decomposition Boost or Burden Fact-Checking Performance? (2025) Automated Fact Checking and Claim Verification
9 Ori Yoran 46 3
69 Answering Questions by Meta-Reasoning over Multiple Chains of Thought (2023) Reasoning-Intensive Retrieval and Ranking
63 Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills (2022) Multi-hop and Complex Reasoning QA
10 Idan Szpektor 45 3
63 MDCure: A Scalable Pipeline for Multi-Document Instruction-Following (2025) Multi-hop and Complex Reasoning QA
46 That's Not My Question: Learning to Weight Unmatched Terms in CQA Vertical Search (2016) Domain-Specific and Structured Retrieval
11 Qinggang Zhang 45 3
67 Knapsack Optimization-Based Schema Linking for LLM-Based Text-to-SQL Generation (2026) Multi-hop and Complex Reasoning QA
65 Hierarchy-Aware Multi-Hop Question Answering over Knowledge Graphs (2023) Knowledge Graph Question Answering
12 Leif Azzopardi COI: co-author 43 3
52 Broad expertise retrieval in sparse data environments (2007) Domain-Specific and Structured Retrieval
52 A language modeling framework for expert finding (2009) Domain-Specific and Structured Retrieval
13 Venktesh V 43 3
58 SUNAR: Semantic Uncertainty based Neighborhood Aware Retrieval for Complex QA (2025) Multi-Hop and Iterative Retrieval Reasoning
52 Understanding the User: An Intent-Based Ranking Dataset (2024) Query Reformulation and Expansion
14 Xinying Qian 43 3
67 Beyond Timestamps: Bridging Forward and Backward Reasoning in Temporal Numerical and Relational Understanding (2026) Multi-hop and Complex Reasoning QA
41 TimeR⁴ : Time-aware Retrieval-Augmented Large Language Models for Temporal Knowledge Graph Question Answering (2024) Multi-Hop and Iterative Retrieval Reasoning
15 Keping Bi 42 3
48 Learning a Fine-Grained Review-based Transformer Model for Personalized Product Search (2021) Recommendation with Search and Ranking
48 A Transformer-based Embedding Model for Personalized Product Search (2020) Recommendation with Search and Ranking
16 Andrew Parry assigned 42 3
48 Constructing and Evaluating Declarative RAG Pipelines in PyTerrier (2025) Retrieval Toolkits and Reproducibility
47 Corpus Subsampling: Estimating the Effectiveness of Neural Retrieval Models on Large Corpora (2025) Evaluation Resources and Tools
17 Weiwei Sun 0001 42 3
47 MAIR: A Massive Benchmark for Evaluating Instructed Retrieval (2024) Evaluation Resources and Tools
41 Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents (2023) Reasoning-Intensive Retrieval and Ranking
18 Daiting Shi 42 3
67 A Question-Oriented Propagation Network for News Reading Comprehension (2022) Multi-hop and Complex Reasoning QA
44 Utility-Focused LLM Annotation for Retrieval and Retrieval-Augmented Generation (2025) Evidence Selection and Context Refinement
19 Xueguang Ma 41 3
52 Toward Best Practices for Training Multilingual Dense Retrieval Models (2024) Evaluation Resources and Tools
50 Resources for Brewing BEIR: Reproducible Reference Models and Statistical Analyses (2024) Retrieval Toolkits and Reproducibility
20 Tianzhe Zhao 41 3
64 Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond (2025) Reasoning Evaluation and Benchmarks
63 PathReasoner: Modeling Reasoning Path with Equivalent Extension for Logical Question Answering (2024) Search Agents and LLM Reasoning › Others
21 Falk Scholer 41 3
68 WikiHowQA: A Comprehensive Benchmark for Multi-Document Non-Factoid Question Answering (2023) Multi-hop and Complex Reasoning QA
46 Ranking Documents by Answer-Passage Quality (2018) Retrieval-Augmented and Open-Domain QA
23 Paul Thomas 0001 41 3
50 cwl_eval: An Evaluation Tool for Information Retrieval (2019) Retrieval Toolkits and Reproducibility
42 Can Generative LLMs Create Query Variants for Test Collections? An Exploratory Study (2023) Query Reformulation and Expansion
24 Tingyu Song 40 3
43 IFIR: A Comprehensive Benchmark for Evaluating Instruction-Following in Expert-Domain Information Retrieval (2025) User Behavior and Search Interaction
41 LimRank: Less is More for Reasoning-Intensive Information Reranking (2025) Evidence Selection and Context Refinement
25 Xi Ye 0003 40 3
67 Connecting Attributions and QA Model Behavior on Realistic Counterfactuals (2021) Multi-hop and Complex Reasoning QA
65 RNG-KBQA: Generation Augmented Iterative Ranking for Knowledge Base Question Answering (2022) Knowledge Graph Question Answering
26 Jing Liu 0022 40 3
50 A Hierarchical Entity-Based Approach to Structuralize User Generated Content in Social Media: A Case of Yahoo! Answers (2013) Domain-Specific and Structured Retrieval
45 A Regularized Competition Model for Question Difficulty Estimation in Community Question Answering Services (2014) Retrieval-Augmented and Open-Domain QA
27 Tetsuya Sakai 40 3
40 ChatRetriever: Adapting Large Language Models for Generalized and Robust Conversational Dense Retrieval (2024) Conversational Search and Retrieval
40 On the Ordering of Pooled Web Pages, Gold Assessments, and Bronze Assessments (2024) Test Collection Construction and Pooling
28 Xiao Wang 0043 40 3
40 Improving zero-shot retrieval using dense external expansion (2022) Query Reformulation and Expansion
39 Retrieve, Rerank, Answer, Experiment: Hands-On IR Research with PyTerrier (2026) Retrieval Toolkits and Reproducibility
29 Xinyu Ma 0001 COI: co-author 40 3
51 FULTR: A Large-Scale Fusion Learning to Rank Dataset and Its Application for Satisfaction-Oriented Ranking (2025) Web-Scale Learning to Rank
45 Pre-training with Aspect-Content Text Mutual Prediction for Multi-Aspect Dense Retrieval (2023) Sparse and Dense Retrieval Encoders
31 Negar Arabzadeh 40 3
44 Can QPP Choose the Right Query variant? Evaluating Query Variant Selection for RAG Pipelines (2026) Evidence Selection and Context Refinement
43 Matches Made in Heaven: Toolkit and Large-Scale Datasets for Supervised Query Reformulation (2021) Query Reformulation and Expansion
32 Jurek Leonhardt 39 3
52 Understanding the User: An Intent-Based Ranking Dataset (2024) Query Reformulation and Expansion
35 Data Augmentation for Sample Efficient and Robust Document Ranking (2024) Ranking Distillation and Adaptation