Demonstration venue · read-only. Every page can be browsed; the buttons that would change it are switched off. Create an account to run TaxoReview on your own data. A simulation from public publication records: the people listed were not invited and did not agree to review.
? Tutorial for this page

Think When Needed: Model-Aware Reasoning Routing for LLM-based Ranking

sigir26-185 · Full research paper · Huizhong Guo, Tianjun Wei, Dongxia Wang, Yingpeng Du, Ziyan Wang, Jie Zhang, Zhu Sun
LLM-Based Reranking · 50% Reward Modeling and Process Supervis · 25% Reasoning-Intensive Retrieval and Ra · 25%
Abstract

Large language models (LLMs) are increasingly applied to ranking tasks in retrieval and recommendation. Although reasoning prompting can enhance ranking utility, our preliminary exploration reveals that its benefits are inconsistent and come at a substantial computational cost, suggesting that when to reason is as crucial as how to reason. To address this issue, we propose a reasoning routing framework that employs a lightweight, plug-and-play router head to decide whether to use direct inference (Non-Think) or reasoning (Think) for each instance before generation. The router head relies solely on pre-generation signals: i) compact ranking-aware features (e.g., candidate dispersion) and ii) model-aware difficulty signals derived from a diagnostic checklist reflecting the model's estimated need for reasoning. By leveraging these features before generation, the router outputs a controllable token that determines whether to apply the Think mode. Furthermore, the router can adaptively select its operating policy along the validation Pareto frontier at deployment time, enabling dynamic allocation of computational resources toward instances most likely to benefit from Think under varying system constraints. Experiments on three public ranking datasets with different scales of open-source LLMs show consistent improvements in ranking utility with reduced token consumption (e.g., +6.3% NDCG@10 with –49.5% tokens on MovieLens with Qwen3-4B), demonstrating reasoning routing as a practical solution to the accuracy-efficiency trade-off.

Assigned reviewers

#ReviewerMatchWhy (closest evidence publications in the taxonomy)
1 Daiting Shi
54
68 TourRank: Utilizing Large Language Models for Documents Ranking with a Tournament-Inspired Strategy (2025) LLM-Based Reranking
48 ReflectRM: Boosting Generative Reward Models via Self-Reflection within a Unified Judgment Framework (2026) Reward Modeling and Process Supervision
48 ConsistRM: Improving Generative Reward Models via Consistency-Aware Self-Training (2026) Reward Modeling and Process Supervision
2 Weiwei Sun 0001
51
89 ReasonRank: Empowering Passage Ranking with Strong Reasoning Ability (2026) LLM-Based Reranking
85 TourRank: Utilizing Large Language Models for Documents Ranking with a Tournament-Inspired Strategy (2025) Reasoning-Intensive Retrieval and Ranking
74 Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents (2023) Reasoning-Intensive Retrieval and Ranking
3 Nancy F. Chen
45
48 AdaMCoT: Rethinking Cross-Lingual Factual Reasoning Through Adaptive Multilingual Chain-of-Thought (2026) Reward Modeling and Process Supervision
48 Learning Planning-based Reasoning by Trajectories Collection and Process Reward Synthesizing (2024) Reward Modeling and Process Supervision
46 SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization (2022) Cross-Encoder and Neural Reranking

Area chair

Area chairMatch
Xiaoxue Zang 42
Area-chair candidates (8)
Xiaoxi Li 0005 4315 papers
Dacheng Tao COI: co-author 425 papers
Xiaoxue Zang assigned 424 papers
Jun Xu 0001 422 papers
Zhijing Wu 0001 419 papers
Pengyue Jia 4110 papers
Chenyan Xiong 4115 papers
Kam-Fai Wong 406 papers

Candidates from the panel ranked by match score

#ReviewerMatchLoadWhy
1 Daiting Shi assigned 54 3
68 TourRank: Utilizing Large Language Models for Documents Ranking with a Tournament-Inspired Strategy (2025) LLM-Based Reranking
48 ReflectRM: Boosting Generative Reward Models via Self-Reflection within a Unified Judgment Framework (2026) Reward Modeling and Process Supervision
2 Archiki Prasad 54 3
56 PRInTS: Reward Modeling for Long-Horizon Information Seeking (2026) Reward Modeling and Process Supervision
49 MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning (2025) Reward Modeling and Process Supervision
3 Shengyao Zhuang 53 3
85 R2LLMs: Retrieval and Ranking with LLMs (2025) Reasoning-Intensive Retrieval and Ranking
77 Rank-R1: Enhancing Reasoning in LLM-based Document Rerankers via Reinforcement Learning (2026) Reasoning-Intensive Retrieval and Ranking
4 Haritz Puerto 53 3
48 Fine-Tuning on Diverse Reasoning Chains Drives Within-Inference CoT Refinement in LLMs (2025) Reward Modeling and Process Supervision
48 Code Prompting Elicits Conditional Reasoning Abilities in Text+Code LLMs (2024) Reward Modeling and Process Supervision
5 Weiwei Sun 0001 assigned 51 3
89 ReasonRank: Empowering Passage Ranking with Strong Reasoning Ability (2026) LLM-Based Reranking
85 TourRank: Utilizing Large Language Models for Documents Ranking with a Tournament-Inspired Strategy (2025) Reasoning-Intensive Retrieval and Ranking
6 Yongqi Li 0001 51 3
48 Parallel Test-Time Scaling for Latent Reasoning Models (2026) Reward Modeling and Process Supervision
48 TokenSkip: Controllable Chain-of-Thought Compression in LLMs (2025) Reward Modeling and Process Supervision
7 Jialong Wu 0007 50 3
48 SEED: Accelerating Reasoning Tree Construction via Scheduled Speculative Decoding (2025) Reward Modeling and Process Supervision
45 WebDancer: Towards Autonomous Information Seeking Agency (2025) Deep Research and Report Generation
8 Yuchen Li 0006 49 3
68 ReasonRank: Empowering Passage Ranking with Strong Reasoning Ability (2026) LLM-Based Reranking
55 Retain to Refine: Adaptive Online Question Answering via Query Routing and Long-Short Memory (2026) Agentic and Deep Search RAG
9 Qingyi Si 48 3
48 Test-time Prompt Intervention (2026) Reward Modeling and Process Supervision
48 Outcome-Grounded Advantage Reshaping for Fine-Grained Credit Assignment in Mathematical Reasoning (2026) Reward Modeling and Process Supervision
10 Xinyu Ma 0001 46 3
89 ReasonRank: Empowering Passage Ranking with Strong Reasoning Ability (2026) LLM-Based Reranking
74 Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents (2023) Reasoning-Intensive Retrieval and Ranking
11 Shuaiqiang Wang 46 3
55 Thinking Forward and Backward: Multi-Objective Reinforcement Learning for Retrieval-Augmented Reasoning (2026) Multi-Hop and Iterative Retrieval Reasoning
50 Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents (2023) Text Embedding and Representation Learning
12 Nancy F. Chen assigned 45 3
48 AdaMCoT: Rethinking Cross-Lingual Factual Reasoning Through Adaptive Multilingual Chain-of-Thought (2026) Reward Modeling and Process Supervision
48 Learning Planning-based Reasoning by Trajectories Collection and Process Reward Synthesizing (2024) Reward Modeling and Process Supervision
13 Chen Zhao 0013 45 3
68 LimRank: Less is More for Reasoning-Intensive Information Reranking (2025) LLM-Based Reranking
56 Rethinking Reasoning-Intensive Retrieval: Evaluating and Advancing Retrievers in Agentic Search Systems (2026) Reasoning-Intensive Retrieval and Ranking
14 Keqin Bao 44 3
48 Chain of Execution Supervision Promotes General Reasoning in Large Language Models (2025) Reward Modeling and Process Supervision
38 TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendation (2023) Preference Alignment and Optimization
15 Xi Ye 0003 43 3
55 Query-Focused Retrieval Heads Improve Long-Context Reasoning and Re-ranking (2025) Multi-hop and Complex Reasoning QA
43 Beyond Single-shot Writing: Deep Research Agents are Unreliable at Multi-turn Report Revision (2026) Deep Research and Report Generation
16 Wonduk Seo 43 3
41 A New Query Expansion Approach for Enhancing Information Retrieval via Agent-Mediated Dialogic Inquiry (2026) Query Reformulation and Expansion
37 Automated Visualization Code Synthesis via Multi-path Reasoning and Feedback-Driven Optimizations (2026) Reasoning Evaluation and Benchmarks
17 Kaixin Ma 43 3
42 WebEvolver: Enhancing Web Agent Self-Improvement with Co-evolving World Model (2025) RL Training of Search Agents
42 OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimization (2025) RL Training of Search Agents
18 Farima Fatahi Bayat 43 3
50 From Proof to Program: Characterizing Tool-Induced Reasoning Hallucinations in Large Language Models (2026) Reward Modeling and Process Supervision
34 CompactIE: Compact Facts in Open Information Extraction (2022) Language Model Pre-training and Adaptation
19 Adam Jatowt 43 3
85 BracketRank: Large Language Model Document Ranking via Reasoning-based Competitive Elimination (2026) Reasoning-Intensive Retrieval and Ranking
41 Search Intent Discovery by Structurization of Community QA Contents (2012) Query Reformulation and Expansion
21 Jing Liu 0022 42 3
68 Self-Calibrated Listwise Reranking with Large Language Models (2025) LLM-Based Reranking
42 RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking (2021) Dense Bi-Encoder Training and Representation Learning
25 Ori Yoran 42 3
54 Answering Questions by Meta-Reasoning over Multiple Chains of Thought (2023) Reasoning-Intensive Retrieval and Ranking
41 Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills (2022) Multi-hop and Complex Reasoning QA
26 Yu-An Liu 0028 42 3
56 Thinking Forward and Backward: Multi-Objective Reinforcement Learning for Retrieval-Augmented Reasoning (2026) Agentic and Deep Search RAG
50 Is a Busy Search Agent a Good One? Overthinking and Overretrieval at Scale (2026) Reward Modeling and Process Supervision
27 Tingyu Song 42 3
59 LimRank: Less is More for Reasoning-Intensive Information Reranking (2025) Evidence Selection and Context Refinement
56 Rethinking Reasoning-Intensive Retrieval: Evaluating and Advancing Retrievers in Agentic Search Systems (2026) Reasoning-Intensive Retrieval and Ranking
28 Weipeng Chen 41 3
70 RichRAG: Crafting Rich Responses for Multi-faceted Queries in Retrieval-Augmented Generation (2025) Evidence Selection and Context Refinement
45 ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning (2025) RL Training of Search Agents
29 Yingyi Zhang 0001 41 3
54 Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning (2025) Agentic and Deep Search RAG
45 Personalized Deep Research: A User-Centric Framework, Dataset, and Hybrid Evaluation for Knowledge Discovery (2026) Deep Research and Report Generation
30 Jurek Leonhardt 41 3
47 Data Augmentation for Sample Efficient and Robust Document Ranking (2024) Ranking Distillation and Adaptation
46 Extractive Explanations for Interpretable Text Ranking (2023) Cross-Encoder and Neural Reranking
33 Zhen Qin 0001 41 3
66 Consolidating Ranking and Relevance Predictions of Large Language Models through Post-Processing (2024) LLM-Based Reranking
48 Can Query Expansion Improve Generalization of Strong Cross-Encoder Rankers? (2024) Query Reformulation and Expansion
35 Wenlin Zhang 0001 40 3
54 Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning (2025) Agentic and Deep Search RAG
44 To Search or Not to Search: Aligning the Decision Boundary of Deep Search Agents via Causal Intervention (2026) Agentic and Deep Search RAG
36 Rolf Jagerman 40 3
66 Consolidating Ranking and Relevance Predictions of Large Language Models through Post-Processing (2024) LLM-Based Reranking
48 Can Query Expansion Improve Generalization of Strong Cross-Encoder Rankers? (2024) Query Reformulation and Expansion
37 Xuanhui Wang 40 3
72 Consolidating Ranking and Relevance Predictions of Large Language Models through Post-Processing (2024) Reasoning-Intensive Retrieval and Ranking
48 Can Query Expansion Improve Generalization of Strong Cross-Encoder Rankers? (2024) Query Reformulation and Expansion