Learning to Rank with Multi-Criteria LLM-Judge Annotations
sigir26-056 · Full research paper · Naghmeh Farzi, Laura Dietz
Ranking Losses and Objectives · 50% LLM-Based Relevance Assessment · 50%
Abstract
Large Language Models (LLMs) are increasingly used as automated judges (LLM judges) to evaluate Information Retrieval (IR) systems, offering a cost-effective complement to human assessments. However, most prior work treats evaluation mainly as a tool for comparison rather than as a signal for improving the retrieval system. We study whether criterion grades from Multi-Criteria LLM-Judge relevance labeling, which decomposes relevance into Exactness, Coverage, Topicality, and Contextual Fit, can serve as effective ranking features for learning-to-rank (L2R). We evaluate this approach using manual relevance labels from TREC TREC DL 2019, DL 2020, and DL 2023. We then analyze how the Multi-Criteria LLM-Judge feature importance varies relative to retrieval scores across IR system performance levels.
Assigned reviewers
| # | Reviewer | Match | Why (closest evidence publications in the taxonomy) | |
|---|---|---|---|---|
| 1 | Paul Thomas 0001 | 59 | 68 On the Use of LLMs for Relevance Labelling (2026) LLM-Based Relevance Assessment 65 System Comparison Using Automated Generation of Relevance Judgements in Multiple Languages (2025) LLM-Based Relevance Assessment 65 LLM4Eval: Large Language Model for Evaluation in IR (2025) LLM-Based Relevance Assessment |
|
| 2 | Rolf Jagerman | 56 | 73 Rax: Composable Learning-to-Rank Using JAX (2022) Retrieval Toolkits and Reproducibility 70 Regression Compatible Listwise Objectives for Calibrated Ranking with Binary Relevance (2023) Ranking Losses and Objectives 69 Consolidating Ranking and Relevance Predictions of Large Language Models through Post-Processing (2024) LLM-Based Reranking |
|
| 3 | Zhen Qin 0001 | 54 | 73 Rax: Composable Learning-to-Rank Using JAX (2022) Retrieval Toolkits and Reproducibility 69 Consolidating Ranking and Relevance Predictions of Large Language Models through Post-Processing (2024) LLM-Based Reranking 65 Regression Compatible Listwise Objectives for Calibrated Ranking with Binary Relevance (2023) Ranking Losses and Objectives |
Area chair
| Area chair | Match | |
|---|---|---|
| Fernando Diaz 0001 | 41 |
Area-chair candidates (6)
| Fernando Diaz 0001 assigned | 41 | 12 papers | |
| Jun Xu 0001 | 38 | 2 papers | |
| Chenyan Xiong COI: co-author | 36 | 15 papers | |
| Xiaoxue Zang | 35 | 4 papers | |
| Kam-Fai Wong | 35 | 6 papers | |
| Zhijing Wu 0001 | 34 | 9 papers |
Candidates from the panel ranked by match score
| # | Reviewer | Match | Load | Why | |
|---|---|---|---|---|---|
| 1 | Paul Thomas 0001 assigned | 59 | 3 | 68 On the Use of LLMs for Relevance Labelling (2026) LLM-Based Relevance Assessment 65 System Comparison Using Automated Generation of Relevance Judgements in Multiple Languages (2025) LLM-Based Relevance Assessment |
|
| 2 | Rolf Jagerman assigned | 56 | 3 | 73 Rax: Composable Learning-to-Rank Using JAX (2022) Retrieval Toolkits and Reproducibility 70 Regression Compatible Listwise Objectives for Calibrated Ranking with Binary Relevance (2023) Ranking Losses and Objectives |
|
| 3 | Zhen Qin 0001 assigned | 54 | 3 | 73 Rax: Composable Learning-to-Rank Using JAX (2022) Retrieval Toolkits and Reproducibility 69 Consolidating Ranking and Relevance Predictions of Large Language Models through Post-Processing (2024) LLM-Based Reranking |
|
| 4 | Sebastian Bruch 0001 | 49 | 3 | 65 An Alternative Cross Entropy Loss for Learning-to-Rank (2021) Ranking Losses and Objectives 65 Revisiting Approximate Metric Optimization in the Age of Deep Neural Networks (2019) Ranking Losses and Objectives |
|
| 5 | Andrew Parry | 48 | 3 | 68 Large Language Model Relevance Assessors Agree With One Another More Than With Human Assessors (2025) LLM-Based Relevance Assessment 65 Towards a Relevance Posterior in Neural Information Access (2026) Ranking Losses and Objectives |
|
| 6 | Xuanhui Wang | 46 | 3 | 73 Rax: Composable Learning-to-Rank Using JAX (2022) Retrieval Toolkits and Reproducibility 73 TF-Ranking: Scalable TensorFlow Library for Learning-to-Rank (2019) Retrieval Toolkits and Reproducibility |
|
| 7 | Yuchen Li 0006 | 44 | 3 | 52 FULTR: A Large-Scale Fusion Learning to Rank Dataset and Its Application for Satisfaction-Oriented Ranking (2025) Web-Scale Learning to Rank 45 S2phere: Semi-Supervised Pre-training for Web Search over Heterogeneous Learning to Rank Data (2023) Web-Scale Learning to Rank |
|
| 8 | Negar Arabzadeh | 42 | 3 | 58 LLM-as-a-Judge in Entity Retrieval: Assessing Explicit and Implicit Relevance (2025) Entity Linking and Alignment 42 BERT-QPP: Contextualized Pre-trained transformers for Query Performance Prediction (2021) Query Performance Prediction |
|
| 10 | Shuaiqiang Wang | 41 | 3 | 45 GS2P: A Generative Pre-trained Learning to Rank Model with Over-parameterization for Web-Scale Search (Extended Abstract) (2024) Web-Scale Learning to Rank 45 GS2P: a generative pre-trained learning to rank model with over-parameterization for web-scale search (2024) Web-Scale Learning to Rank |
|
| 11 | Corby Rosset | 41 | 3 | 65 An Axiomatic Approach to Regularizing Neural Ranking Models (2019) Ranking Losses and Objectives 36 Contextual Re-Ranking with Behavior Aware Transformers (2020) Conversational Search and Retrieval |
|
| 12 | Keping Bi | 41 | 3 | 65 Learning a Deep Listwise Context Model for Ranking Refinement (2018) Ranking Losses and Objectives 62 Utility-Focused LLM Annotation for Retrieval and Retrieval-Augmented Generation (2025) Evidence Selection and Context Refinement |
|
| 13 | Falk Scholer | 40 | 3 | 58 On the Use of LLMs for Relevance Labelling (2026) User Behavior and Search Interaction 58 Enhancing Human Annotation: Leveraging Large Language Models and Efficient Batch Processing (2024) User Behavior and Search Interaction |
|
| 14 | Adam Jatowt | 40 | 3 | 65 WassRank: Listwise Document Ranking Using Optimal Transport Theory (2019) Ranking Losses and Objectives 45 BracketRank: Large Language Model Document Ranking via Reasoning-based Competitive Elimination (2026) Reasoning-Intensive Retrieval and Ranking |
|
| 15 | Yimeng Bai | 40 | 3 | 67 Unconstrained Monotonic Calibration of Predictions in Deep Ranking Systems (2025) CTR and Conversion Rate Prediction 34 GradCraft: Elevating Multi-task Recommendations through Holistic Gradient Crafting (2024) Multi-Task and Multi-Objective Optimization |
|
| 16 | Zechun Niu | 40 | 3 | 46 Investigating the Robustness of Counterfactual Learning to Rank Models: A Reproducibility Study (2025) Unbiased Learning to Rank 38 Addressing Personalized Bias for Unbiased Learning to Rank (2025) Unbiased Learning to Rank |
|
| 17 | Tetsuya Sakai | 40 | 3 | 41 On the Ordering of Pooled Web Pages, Gold Assessments, and Bronze Assessments (2024) Test Collection Construction and Pooling 41 WWW3E8: 259, 000 Relevance Labels for Studying the Effect of Document Presentation Order for Relevance Assessors (2021) Test Collection Construction and Pooling |
|
| 18 | Franco Maria Nardini | 39 | 3 | 66 Learning to Rank in Theory and Practice: From Gradient Boosting to Neural Networks and Unbiased Learning (2019) Unbiased Learning to Rank 65 Selective Gradient Boosting for Effective Learning to Rank (2018) Ranking Losses and Objectives |
|
| 19 | Daiting Shi | 39 | 3 | 62 Utility-Focused LLM Annotation for Retrieval and Retrieval-Augmented Generation (2025) Evidence Selection and Context Refinement 43 TourRank: Utilizing Large Language Models for Documents Ranking with a Tournament-Inspired Strategy (2025) LLM-Based Reranking |
|
| 20 | Gustavo Penha | 39 | 3 | 58 As It Was: Aligning LLM Search Evaluation with Historical User Preferences (2026) User Behavior and Search Interaction 52 On the Calibration and Uncertainty of Neural Learning to Rank Models for Conversational Search (2021) Conversational Search and Retrieval |
|
| 21 | Shengyao Zhuang | 38 | 3 | 45 R2LLMs: Retrieval and Ranking with LLMs (2025) Reasoning-Intensive Retrieval and Ranking 43 Where Relevance Emerges: A Layer-Wise Study of Internal Attention for Zero-Shot Re-Ranking (2026) LLM-Based Reranking |
|
| 22 | Xiao Wang 0043 | 38 | 3 | 39 Retrieve, Rerank, Answer, Experiment: Hands-On IR Research with PyTerrier (2026) Retrieval Toolkits and Reproducibility 39 Neural Pseudo-Relevance Feedback Models for Sparse and Dense Retrieval (2022) Query Reformulation and Expansion |
|
| 23 | Xueguang Ma | 38 | 3 | 45 Tevatron: An Efficient and Flexible Toolkit for Neural Retrieval (2023) Learned Sparse Retrieval Efficiency 43 R2LLMs: Retrieval and Ranking with LLMs (2025) Reasoning-Intensive Retrieval and Ranking |
|
| 25 | Mark D. Smucker | 37 | 3 | 41 Dynamic Sampling Meets Pooling (2019) Test Collection Construction and Pooling 41 Optimizing Nugget Annotations with Active Learning (2016) Test Collection Construction and Pooling |
|
| 26 | Weiwei Sun 0001 | 37 | 3 | 45 TourRank: Utilizing Large Language Models for Documents Ranking with a Tournament-Inspired Strategy (2025) Reasoning-Intensive Retrieval and Ranking 43 Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents (2023) Reasoning-Intensive Retrieval and Ranking |
|
| 27 | Idan Szpektor | 37 | 3 | 66 That's Not My Question: Learning to Weight Unmatched Terms in CQA Vertical Search (2016) Domain-Specific and Structured Retrieval 39 From query to question in one click: suggesting synthetic questions to searchers (2013) Query Reformulation and Expansion |
|
| 28 | Leif Azzopardi | 36 | 3 | 42 Measuring Bias in a Ranked List Using Term-Based Representations (2024) Evaluation Metrics and User Models 42 Efficiently Estimating Retrievability Bias (2014) Evaluation Metrics and User Models |
|
| 30 | Yang Zhang 0072 | 36 | 3 | 41 Don't Start Over: A Cost-Effective Framework for Migrating Personalized Prompts Between LLMs (2026) Efficiency and Deployment 36 Reason4Rec: Deliberative User Preference Alignment of Large Language Models for Recommendation (2026) Preference Alignment and Optimization |
|
| 31 | Jurek Leonhardt | 36 | 3 | 44 Understanding the User: An Intent-Based Ranking Dataset (2024) Query Reformulation and Expansion 41 Data Augmentation for Sample Efficient and Robust Document Ranking (2024) Ranking Distillation and Adaptation |
|
| 32 | Theresia Veronika Rampisela | 36 | 3 | 36 Can We Trust Recommender System Fairness Evaluation? The Role of Fairness and Relevance (2024) Fairness Metrics and Evaluation 36 Joint Evaluation of Fairness and Relevance in Recommender Systems with Pareto Frontier (2025) Fairness Metrics and Evaluation |
|
| 34 | Xinyu Ma 0001 | 35 | 3 | 52 FULTR: A Large-Scale Fusion Learning to Rank Dataset and Its Application for Satisfaction-Oriented Ranking (2025) Web-Scale Learning to Rank 45 M2oERank: Multi-Objective Mixture-of-Experts Enhanced Ranking for Satisfaction-Oriented Web Search (2025) Web-Scale Learning to Rank |