FollowTable: A Benchmark for Instruction-Following Table Retrieval
Abstract
Table Retrieval (TR) has traditionally been formulated as an ad-hoc retrieval problem, where relevance is primarily determined by topical semantic similarity. With the growing adoption of LLM-based agentic systems, access to structured data is increasingly instruction-driven, where relevance is conditional on explicit content and schema constraints rather than topical similarity alone. We therefore formalize Instruction-Following Table Retrieval (IFTR), a new task that requires models to jointly satisfy topical relevance and fine-grained instruction constraints. We identify two core challenges in IFTR: (i) sensitivity to content scope, such as inclusion and exclusion constraints, and (ii) awareness of schema-grounded requirements, including column semantics and representation granularity--capabilities largely absent in existing retrievers. To support systematic evaluation, we introduce FollowTable, the first large-scale benchmark for IFTR, constructed via a taxonomy-driven annotation pipeline. We further propose a new metric, termed the Instruction Responsiveness Score, to evaluate whether retrieval rankings consistently adapt to user instructions relative to a topic-only baseline. Our results indicate that existing retrieval models struggle to follow fine-grained instructions over tabular data. In particular, they exhibit systematic biases toward surface-level semantic cues and remain limited in handling schema-grounded constraints, highlighting substantial room for future improvements.
Assigned reviewers
| # | Reviewer | Match | Why (closest evidence publications in the taxonomy) | |
|---|---|---|---|---|
| 1 | Wenyu Huang | 55 | 67 Masking in Multi-hop QA: An Analysis of How Language Models Perform with Context Permutation (2025) Multi-hop and Complex Reasoning QA 51 Prompting large language models with knowledge graphs for question answering involving long-tail facts (2025) KG-Augmented LLM Reasoning 32 Retrieval Augmented Generation with Rich Answer Encoding (2023) Retrieval-Augmented Generation › Others |
|
| 2 | Andrew Parry | 42 | 48 Constructing and Evaluating Declarative RAG Pipelines in PyTerrier (2025) Retrieval Toolkits and Reproducibility 47 Corpus Subsampling: Estimating the Effectiveness of Neural Retrieval Models on Large Corpora (2025) Evaluation Resources and Tools 40 Variations in Relevance Judgments and the Shelf Life of Test Collections (2025) Test Collection Construction and Pooling |
Area chair
| Area chair | Match | |
|---|---|---|
| Zhijing Wu 0001 | 41 |
Area-chair candidates (4)
| Zhijing Wu 0001 assigned | 41 | 9 papers | |
| Hongjin Qian COI: co-author | 40 | 12 papers | |
| Xiaoxi Li 0005 | 39 | 15 papers | |
| Chenyan Xiong | 38 | 15 papers |
Candidates from the panel ranked by match score
| # | Reviewer | Match | Load | Why | |
|---|---|---|---|---|---|
| 1 | Adam Jatowt COI: co-author | 56 | 3 | 68 ArchivalQA: A Large-scale Benchmark Dataset for Open-Domain Question Answering over Historical News Collections (2022) Multi-hop and Complex Reasoning QA 67 Inferential Question Answering (2026) Multi-hop and Complex Reasoning QA |
|
| 2 | Wenyu Huang assigned | 55 | 3 | 67 Masking in Multi-hop QA: An Analysis of How Language Models Perform with Context Permutation (2025) Multi-hop and Complex Reasoning QA 51 Prompting large language models with knowledge graphs for question answering involving long-tail facts (2025) KG-Augmented LLM Reasoning |
|
| 3 | Chen Zhao 0013 | 51 | 3 | 67 Multi-Step Reasoning Over Unstructured Text with Beam Dense Retrieval (2021) Multi-hop and Complex Reasoning QA 67 TaPERA: Enhancing Faithfulness and Interpretability in Long-Form Table QA by Content Planning and Execution-based Reasoning (2024) Multi-hop and Complex Reasoning QA |
|
| 4 | Farzad Shami | 48 | 3 | 50 Doctron: A Web-based Collaborative Annotation Tool for Ground Truth Creation in IR (2025) Retrieval Toolkits and Reproducibility 45 From Single to Multi-agent Reasoning: Advancing GeneGPT for Genomics QA (2026) Retrieval-Augmented and Open-Domain QA |
|
| 5 | Yu Gu 0016 | 47 | 3 | 65 ArcaneQA: Dynamic Program Induction and Contextualized Encoding for Knowledge Base Question Answering (2022) Knowledge Graph Question Answering 65 Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases (2021) Knowledge Graph Question Answering |
|
| 6 | Mark D. Smucker | 47 | 3 | 47 Evaluating sentence-level relevance feedback for high-recall information retrieval (2020) Evaluation Resources and Tools 47 A Preference Judgment Tool for Authoritative Assessment (2023) Evaluation Resources and Tools |
|
| 7 | Yuzhong Qu COI: co-author | 46 | 3 | 65 Enhancing Complex Question Answering over Knowledge Graphs through Evidence Pattern Retrieval (2024) Knowledge Graph Question Answering 65 Skeleton parsing for complex question answering over knowledge bases (2022) Knowledge Graph Question Answering |
|
| 8 | Qisheng Hu | 46 | 3 | 67 Just What You Desire: Constrained Timeline Summarization with Self-Reflection for Enhanced Relevance (2025) Multi-hop and Complex Reasoning QA 34 Decomposition Dilemmas: Does Claim Decomposition Boost or Burden Fact-Checking Performance? (2025) Automated Fact Checking and Claim Verification |
|
| 9 | Ori Yoran | 46 | 3 | 69 Answering Questions by Meta-Reasoning over Multiple Chains of Thought (2023) Reasoning-Intensive Retrieval and Ranking 63 Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills (2022) Multi-hop and Complex Reasoning QA |
|
| 10 | Idan Szpektor | 45 | 3 | 63 MDCure: A Scalable Pipeline for Multi-Document Instruction-Following (2025) Multi-hop and Complex Reasoning QA 46 That's Not My Question: Learning to Weight Unmatched Terms in CQA Vertical Search (2016) Domain-Specific and Structured Retrieval |
|
| 11 | Qinggang Zhang | 45 | 3 | 67 Knapsack Optimization-Based Schema Linking for LLM-Based Text-to-SQL Generation (2026) Multi-hop and Complex Reasoning QA 65 Hierarchy-Aware Multi-Hop Question Answering over Knowledge Graphs (2023) Knowledge Graph Question Answering |
|
| 12 | Leif Azzopardi COI: co-author | 43 | 3 | 52 Broad expertise retrieval in sparse data environments (2007) Domain-Specific and Structured Retrieval 52 A language modeling framework for expert finding (2009) Domain-Specific and Structured Retrieval |
|
| 13 | Venktesh V | 43 | 3 | 58 SUNAR: Semantic Uncertainty based Neighborhood Aware Retrieval for Complex QA (2025) Multi-Hop and Iterative Retrieval Reasoning 52 Understanding the User: An Intent-Based Ranking Dataset (2024) Query Reformulation and Expansion |
|
| 14 | Xinying Qian | 43 | 3 | 67 Beyond Timestamps: Bridging Forward and Backward Reasoning in Temporal Numerical and Relational Understanding (2026) Multi-hop and Complex Reasoning QA 41 TimeR⁴ : Time-aware Retrieval-Augmented Large Language Models for Temporal Knowledge Graph Question Answering (2024) Multi-Hop and Iterative Retrieval Reasoning |
|
| 15 | Keping Bi | 42 | 3 | 48 Learning a Fine-Grained Review-based Transformer Model for Personalized Product Search (2021) Recommendation with Search and Ranking 48 A Transformer-based Embedding Model for Personalized Product Search (2020) Recommendation with Search and Ranking |
|
| 16 | Andrew Parry assigned | 42 | 3 | 48 Constructing and Evaluating Declarative RAG Pipelines in PyTerrier (2025) Retrieval Toolkits and Reproducibility 47 Corpus Subsampling: Estimating the Effectiveness of Neural Retrieval Models on Large Corpora (2025) Evaluation Resources and Tools |
|
| 17 | Weiwei Sun 0001 | 42 | 3 | 47 MAIR: A Massive Benchmark for Evaluating Instructed Retrieval (2024) Evaluation Resources and Tools 41 Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents (2023) Reasoning-Intensive Retrieval and Ranking |
|
| 18 | Daiting Shi | 42 | 3 | 67 A Question-Oriented Propagation Network for News Reading Comprehension (2022) Multi-hop and Complex Reasoning QA 44 Utility-Focused LLM Annotation for Retrieval and Retrieval-Augmented Generation (2025) Evidence Selection and Context Refinement |
|
| 19 | Xueguang Ma | 41 | 3 | 52 Toward Best Practices for Training Multilingual Dense Retrieval Models (2024) Evaluation Resources and Tools 50 Resources for Brewing BEIR: Reproducible Reference Models and Statistical Analyses (2024) Retrieval Toolkits and Reproducibility |
|
| 20 | Tianzhe Zhao | 41 | 3 | 64 Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond (2025) Reasoning Evaluation and Benchmarks 63 PathReasoner: Modeling Reasoning Path with Equivalent Extension for Logical Question Answering (2024) Search Agents and LLM Reasoning › Others |
|
| 21 | Falk Scholer | 41 | 3 | 68 WikiHowQA: A Comprehensive Benchmark for Multi-Document Non-Factoid Question Answering (2023) Multi-hop and Complex Reasoning QA 46 Ranking Documents by Answer-Passage Quality (2018) Retrieval-Augmented and Open-Domain QA |
|
| 23 | Paul Thomas 0001 | 41 | 3 | 50 cwl_eval: An Evaluation Tool for Information Retrieval (2019) Retrieval Toolkits and Reproducibility 42 Can Generative LLMs Create Query Variants for Test Collections? An Exploratory Study (2023) Query Reformulation and Expansion |
|
| 24 | Tingyu Song | 40 | 3 | 43 IFIR: A Comprehensive Benchmark for Evaluating Instruction-Following in Expert-Domain Information Retrieval (2025) User Behavior and Search Interaction 41 LimRank: Less is More for Reasoning-Intensive Information Reranking (2025) Evidence Selection and Context Refinement |
|
| 25 | Xi Ye 0003 | 40 | 3 | 67 Connecting Attributions and QA Model Behavior on Realistic Counterfactuals (2021) Multi-hop and Complex Reasoning QA 65 RNG-KBQA: Generation Augmented Iterative Ranking for Knowledge Base Question Answering (2022) Knowledge Graph Question Answering |
|
| 26 | Jing Liu 0022 | 40 | 3 | 50 A Hierarchical Entity-Based Approach to Structuralize User Generated Content in Social Media: A Case of Yahoo! Answers (2013) Domain-Specific and Structured Retrieval 45 A Regularized Competition Model for Question Difficulty Estimation in Community Question Answering Services (2014) Retrieval-Augmented and Open-Domain QA |
|
| 27 | Tetsuya Sakai | 40 | 3 | 40 ChatRetriever: Adapting Large Language Models for Generalized and Robust Conversational Dense Retrieval (2024) Conversational Search and Retrieval 40 On the Ordering of Pooled Web Pages, Gold Assessments, and Bronze Assessments (2024) Test Collection Construction and Pooling |
|
| 28 | Xiao Wang 0043 | 40 | 3 | 40 Improving zero-shot retrieval using dense external expansion (2022) Query Reformulation and Expansion 39 Retrieve, Rerank, Answer, Experiment: Hands-On IR Research with PyTerrier (2026) Retrieval Toolkits and Reproducibility |
|
| 29 | Xinyu Ma 0001 COI: co-author | 40 | 3 | 51 FULTR: A Large-Scale Fusion Learning to Rank Dataset and Its Application for Satisfaction-Oriented Ranking (2025) Web-Scale Learning to Rank 45 Pre-training with Aspect-Content Text Mutual Prediction for Multi-Aspect Dense Retrieval (2023) Sparse and Dense Retrieval Encoders |
|
| 31 | Negar Arabzadeh | 40 | 3 | 44 Can QPP Choose the Right Query variant? Evaluating Query Variant Selection for RAG Pipelines (2026) Evidence Selection and Context Refinement 43 Matches Made in Heaven: Toolkit and Large-Scale Datasets for Supervised Query Reformulation (2021) Query Reformulation and Expansion |
|
| 32 | Jurek Leonhardt | 39 | 3 | 52 Understanding the User: An Intent-Based Ranking Dataset (2024) Query Reformulation and Expansion 35 Data Augmentation for Sample Efficient and Robust Document Ranking (2024) Ranking Distillation and Adaptation |