Decision-Theoretic Stopping Rules for Document Screening
sigir26-024 · Full research paper · Aaron H. A. Fletcher, Mark Stevenson
Evaluation Metrics and User Models · 100%
Abstract
Deciding when to stop reviewing the results of a search is a common problem with multiple applications. Existing stopping rules developed within Technology-Assisted Review (TAR) aim to achieve a pre-specified recall target and do not take into account the reason for examining the results, potentially leading to sub-optimal recommendations. This paper applies decision theory to the problem and uses it to derive three practical stopping policies based on the Expected Value of Perfect Information. The approach is applied to two professional search tasks: patent examining and systematic reviewing. Experiments on CLEF-IP and medical systematic review datasets show that the proposed approach generally produces more appropriate stopping decisions than existing methods, as demonstrated by higher net utility under the evaluated cost and payoff settings.
Assigned reviewers
| # | Reviewer | Match | Why (closest evidence publications in the taxonomy) | |
|---|---|---|---|---|
| 1 | Tetsuya Sakai | 74 | 100 Summaries, ranked retrieval and sessions: a unified framework for information access evaluation (2013) Evaluation Metrics and User Models 100 Give me just one highly relevant document: P-measure (2006) Evaluation Metrics and User Models 100 Diversification as Risk Minimization (2026) Evaluation Metrics and User Models |
|
| 2 | Mark D. Smucker | 55 | 100 Offline Evaluation by Maximum Similarity to an Ideal Ranking (2020) Evaluation Metrics and User Models 100 Assessing Top- Preferences (2021) Evaluation Metrics and User Models 68 Time-based calibration of effectiveness measures (2012) User Behavior and Search Interaction |
|
| 3 | Leif Azzopardi | 47 | 70 Efficiently Estimating Retrievability Bias (2014) Evaluation Metrics and User Models 70 Measuring Bias in a Ranked List Using Term-Based Representations (2024) Evaluation Metrics and User Models 68 User Models, Metrics and Measures of Search: A Tutorial on the C/W/L Evaluation Framework (2021) User Behavior and Search Interaction |
Area chair
| Area chair | Match | |
|---|---|---|
| Fernando Diaz 0001 | 45 |
Area-chair candidates (6)
| Fernando Diaz 0001 assigned | 45 | 12 papers | |
| Jun Xu 0001 | 37 | 2 papers | |
| Ziyi Ye | 36 | 14 papers | |
| Chenyan Xiong | 35 | 15 papers | |
| Xiaoxue Zang | 35 | 4 papers | |
| Zhijing Wu 0001 | 35 | 9 papers |
Candidates from the panel ranked by match score
| # | Reviewer | Match | Load | Why | |
|---|---|---|---|---|---|
| 1 | Tetsuya Sakai assigned | 74 | 3 | 100 Summaries, ranked retrieval and sessions: a unified framework for information access evaluation (2013) Evaluation Metrics and User Models 100 Give me just one highly relevant document: P-measure (2006) Evaluation Metrics and User Models |
|
| 2 | Mark D. Smucker assigned | 55 | 3 | 100 Offline Evaluation by Maximum Similarity to an Ideal Ranking (2020) Evaluation Metrics and User Models 100 Assessing Top- Preferences (2021) Evaluation Metrics and User Models |
|
| 3 | Leif Azzopardi assigned | 47 | 3 | 70 Efficiently Estimating Retrievability Bias (2014) Evaluation Metrics and User Models 70 Measuring Bias in a Ranked List Using Term-Based Representations (2024) Evaluation Metrics and User Models |
|
| 4 | Paul Thomas 0001 | 47 | 3 | 68 User Models, Metrics and Measures of Search: A Tutorial on the C/W/L Evaluation Framework (2021) User Behavior and Search Interaction 49 cwl_eval: An Evaluation Tool for Information Retrieval (2019) Retrieval Toolkits and Reproducibility |
|
| 6 | Negar Arabzadeh | 44 | 3 | 49 Learning Context-aware Term Importance for Query Performance Prediction (2026) Query Performance Prediction 49 Estimating Query Performance Through Rich Contextualized Query Representations (2024) Query Performance Prediction |
|
| 7 | Jaime Arguello | 43 | 3 | 49 Using Query Performance Predictors to Improve Spoken Queries (2016) Query Performance Prediction 49 Using Query Performance Predictors to Reduce Spoken Queries (2017) Query Performance Prediction |
|
| 8 | Falk Scholer | 42 | 3 | 49 sMARE: a new paradigm to evaluate and understand query performance prediction methods (2022) Query Performance Prediction 49 Is Query Performance Prediction With Multiple Query Variations Harder Than Topic Performance Prediction? (2021) Query Performance Prediction |
|
| 9 | Xiao Wang 0043 | 41 | 3 | 49 Retrieve, Rerank, Answer, Experiment: Hands-On IR Research with PyTerrier (2026) Retrieval Toolkits and Reproducibility 45 Improving zero-shot retrieval using dense external expansion (2022) Query Reformulation and Expansion |
|
| 10 | Fan Zhang 0053 | 41 | 3 | 68 Evaluating Mobile Search with Height-Biased Gain (2017) User Behavior and Search Interaction 55 Incorporating Query Reformulating Behavior into Web Search Evaluation (2021) User Behavior and Search Interaction |
|
| 11 | Theresia Veronika Rampisela | 40 | 3 | 40 Evaluation Measures of Individual Item Fairness for Recommender Systems: A Critical Study (2025) Fairness Metrics and Evaluation 40 Joint Evaluation of Fairness and Relevance in Recommender Systems with Pareto Frontier (2025) Fairness Metrics and Evaluation |
|
| 12 | Andrew Parry | 39 | 3 | 49 Corpus Subsampling: Estimating the Effectiveness of Neural Retrieval Models on Large Corpora (2025) Evaluation Resources and Tools 49 [inline-graphic not available: see fulltext] SuiteEval: Simplifying Retrieval Benchmarks (2026) Retrieval Toolkits and Reproducibility |
|
| 13 | Yuchen Li 0006 | 38 | 3 | 46 FULTR: A Large-Scale Fusion Learning to Rank Dataset and Its Application for Satisfaction-Oriented Ranking (2025) Web-Scale Learning to Rank 43 GS2P: A Generative Pre-trained Learning to Rank Model with Over-parameterization for Web-Scale Search (Extended Abstract) (2024) Web-Scale Learning to Rank |
|
| 14 | Xuanhui Wang | 38 | 3 | 46 Situational Context for Ranking in Personal Search (2017) Personalized and Context-Aware Retrieval 45 Learning to rank with multi-aspect relevance for vertical search (2012) Personalized and Context-Aware Retrieval |
|
| 15 | Zechun Niu | 38 | 3 | 43 Investigating the Robustness of Counterfactual Learning to Rank Models: A Reproducibility Study (2025) Unbiased Learning to Rank 37 Addressing Personalized Bias for Unbiased Learning to Rank (2025) Unbiased Learning to Rank |
|
| 16 | Jiqun Liu | 38 | 3 | 47 TRUE: A Reproducible Framework for LLM-Driven Relevance Judgment in Information Retrieval (2026) LLM-Based Relevance Assessment 47 LLM-Driven Usefulness Labeling for IR Evaluation (2025) LLM-Based Relevance Assessment |
|
| 17 | Yimeng Bai | 38 | 3 | 45 Leveraging Watch-time Feedback for Short-Video Recommendations: A Causal Labeling Framework (2023) Recommendation Systems and Evaluation 45 LabelCraft: Empowering Short Video Recommendations with Automated Label Crafting (2024) Recommendation Systems and Evaluation |
|
| 18 | Sara Allawati | 38 | 3 | 42 A Comparative Analysis of Linguistic and Retrieval Diversity in LLM-Generated Search Queries (2025) User Behavior and Search Interaction 38 VulGen: Workshop on Vulnerabilities in Generative Systems for Information Retrieval (2026) Bias Mitigation Methods |
|
| 19 | Xueguang Ma | 38 | 3 | 45 Improving Query Representations for Dense Retrieval with Pseudo Relevance Feedback: A Reproducibility Study (2022) Query Reformulation and Expansion 45 Resources for Brewing BEIR: Reproducible Reference Models and Statistical Analyses (2024) Retrieval Toolkits and Reproducibility |
|
| 20 | Antonio Mallia | 38 | 3 | 49 Feature Extraction for Large-Scale Text Collections (2020) Retrieval Toolkits and Reproducibility 45 Unveiling DIME: Reproducibility, Generalizability, and Formal Analysis of Dimension Importance Estimation for Dense Retrieval (2025) Query Reformulation and Expansion |
|
| 21 | Vuong Thanh Tung | 37 | 3 | 43 Interactive Visual Facets to Support Fluid Exploratory Search (2021) Interactive Information Seeking and Evaluation 41 Does More Context Help? Effects of Context Window and Application Source on Retrieval Performance (2022) User Behavior and Search Interaction |
|
| 22 | Gustavo Penha | 37 | 3 | 45 Improving Content Retrievability in Search with Controllable Query Generation (2023) Query Reformulation and Expansion 45 Contextualizing Spotify's Audiobook List Recommendations with Descriptive Shelves (2025) Recommendation Systems and Evaluation |
|
| 23 | Zhen Qin 0001 | 37 | 3 | 45 Bootstrapping Recommendations at Chrome Web Store (2021) Recommendation Systems and Evaluation 45 Improving Recommendation Quality in Google Drive (2020) Recommendation Systems and Evaluation |
|
| 24 | Rolf Jagerman | 37 | 3 | 47 Reliable Confidence Intervals for Information Retrieval Evaluation Using Generative A.I (2024) LLM-Based Relevance Assessment 45 Bootstrapping Recommendations at Chrome Web Store (2021) Recommendation Systems and Evaluation |
|
| 25 | Daiting Shi | 37 | 3 | 100 MA4DIV: Multi-Agent Reinforcement Learning for Search Result Diversification (2025) Evaluation Metrics and User Models 48 Population-Guided Intent-Aware Query Rewriting for Web Search (2026) Query Reformulation and Expansion |
|
| 26 | Shengyao Zhuang | 37 | 3 | 44 Leveraging LLMs for Unsupervised Dense Retriever Ranking (2024) Query Performance Prediction 43 Improving Query Representations for Dense Retrieval with Pseudo Relevance Feedback: A Reproducibility Study (2022) Query Reformulation and Expansion |
|
| 27 | Keping Bi | 37 | 3 | 61 Beyond Relevance: Utility-Centric Retrieval in the LLM Era (2026) Evidence Selection and Context Refinement 43 Extracting search-focused key n-grams for relevance ranking in web search (2012) Web-Scale Learning to Rank |
|
| 29 | Shuaiqiang Wang | 36 | 3 | 43 GS2P: A Generative Pre-trained Learning to Rank Model with Over-parameterization for Web-Scale Search (Extended Abstract) (2024) Web-Scale Learning to Rank 43 COLTR: Semi-Supervised Learning to Rank With Co-Training and Over-Parameterization for Web Search (2023) Web-Scale Learning to Rank |
|
| 30 | Ruqing Zhang 0001 | 36 | 3 | 43 Recent Advances in Generative Information Retrieval (2024) Generative Retrieval 43 Listwise Generative Retrieval Models via a Sequential Learning Process (2024) Generative Retrieval |
|
| 31 | Saber Zerhoudi | 36 | 3 | 44 The SimIIR 2.0 Framework: User Types, Markov Model-Based Interaction Simulation, and Advanced Query Generation (2022) User Simulation for Interactive Systems 44 Simulation of Interactive Information Retrieval: A Guided Tour (2026) User Simulation for Interactive Systems |
|
| 33 | Sebastian Bruch 0001 | 36 | 3 | 45 Effectiveness/efficiency tradeoffs for candidate generation in multi-stage retrieval architectures (2013) Efficient Retrieval and Indexing 44 TF-Ranking: Scalable TensorFlow Library for Learning-to-Rank (2019) Retrieval Toolkits and Reproducibility |