SmartSearch: Process Reward-Guided Query Refinement for Search Agents
sigir26-176 · Full research paper · Tongyu Wen, Guanting Dong, Zhicheng Dou
Query Reformulation and Expansion · 50% Reward Modeling and Process Supervis · 25% Process and Outcome Reward Design · 12% RL Training of Search Agents · 12%
Abstract
Large Language Model (LLM)-based search agents have proven promising for addressing knowledge-intensive problems by incorporating information retrieval capabilities. Existing works largely focus on optimizing the reasoning paradigms of search agents, yet the quality of intermediate search queries during reasoning remains overlooked. As a result, the generated queries often remain inaccurate, leading to unexpected retrieval results and ultimately limiting search agents' overall effectiveness. To mitigate this issue, we introduce SmartSearch, a framework built upon two key mechanisms: (1) Process Rewards, which provide fine-grained supervision for the quality of each intermediate search query through Dual-Level Credit Assessment. (2) Query Refinement, which promotes the optimization of query generation by selectively refining low-quality search queries and regenerating subsequent search rounds based on these refinements. To enable the search agent to progressively internalize the ability to improve query quality under the guidance of process rewards, we design a three-stage curriculum learning framework. This framework guides the agent through a progression from imitation, to alignment, and ultimately to generalization. Experimental results show that SmartSearch consistently surpasses existing baselines, and additional quantitative analyses further confirm its significant gains in both search efficiency and query quality. The code is available at https://github.com/RUC-NLPIR/SmartSearch.
Assigned reviewers
| # | Reviewer | Match | Why (closest evidence publications in the taxonomy) | |
|---|---|---|---|---|
| 1 | Jialong Wu 0007 | 60 | 65 AdaRewriter: Unleashing the Power of Prompting-based Conversational Query Reformulation via Test-Time Adaptation (2025) Query Reformulation and Expansion 65 AdaCQR: Enhancing Query Reformulation for Conversational Search via Sparse and Dense Retrieval Alignment (2025) Query Reformulation and Expansion 51 WebDancer: Towards Autonomous Information Seeking Agency (2025) Deep Research and Report Generation |
|
| 2 | Haritz Puerto | 56 | 50 Fine-Tuning on Diverse Reasoning Chains Drives Within-Inference CoT Refinement in LLMs (2025) Reward Modeling and Process Supervision 50 Code Prompting Elicits Conditional Reasoning Abilities in Text+Code LLMs (2024) Reward Modeling and Process Supervision 45 MetaQA: Combining Expert Agents for Multi-Skill Question Answering (2023) Retrieval-Augmented and Open-Domain QA |
|
| 3 | Yongqi Li 0001 | 53 | 50 Parallel Test-Time Scaling for Latent Reasoning Models (2026) Reward Modeling and Process Supervision 50 TokenSkip: Controllable Chain-of-Thought Compression in LLMs (2025) Reward Modeling and Process Supervision 50 SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration (2025) Reward Modeling and Process Supervision |
Area chair
| Area chair | Match | |
|---|---|---|
| Dacheng Tao | 50 |
Area-chair candidates (7)
| Dacheng Tao assigned | 50 | 5 papers | |
| Xiaoxi Li 0005 COI: co-author | 50 | 15 papers | |
| Xiaoxue Zang | 48 | 4 papers | |
| Fernando Diaz 0001 | 46 | 12 papers | |
| Jun Xu 0001 COI: co-author | 45 | 2 papers | |
| Chenyan Xiong COI: co-author | 41 | 15 papers | |
| Hongjin Qian COI: co-author | 40 | 12 papers |
Candidates from the panel ranked by match score
| # | Reviewer | Match | Load | Why | |
|---|---|---|---|---|---|
| 1 | Jialong Wu 0007 assigned | 60 | 3 | 65 AdaRewriter: Unleashing the Power of Prompting-based Conversational Query Reformulation via Test-Time Adaptation (2025) Query Reformulation and Expansion 65 AdaCQR: Enhancing Query Reformulation for Conversational Search via Sparse and Dense Retrieval Alignment (2025) Query Reformulation and Expansion |
|
| 2 | Daiting Shi COI: co-author | 57 | 3 | 64 Accurate and Efficient Personalized Query Rewriting in Baidu Search (2026) Query Reformulation and Expansion 64 Population-Guided Intent-Aware Query Rewriting for Web Search (2026) Query Reformulation and Expansion |
|
| 3 | Haritz Puerto assigned | 56 | 3 | 50 Fine-Tuning on Diverse Reasoning Chains Drives Within-Inference CoT Refinement in LLMs (2025) Reward Modeling and Process Supervision 50 Code Prompting Elicits Conditional Reasoning Abilities in Text+Code LLMs (2024) Reward Modeling and Process Supervision |
|
| 4 | Yongqi Li 0001 assigned | 53 | 3 | 50 Parallel Test-Time Scaling for Latent Reasoning Models (2026) Reward Modeling and Process Supervision 50 TokenSkip: Controllable Chain-of-Thought Compression in LLMs (2025) Reward Modeling and Process Supervision |
|
| 5 | Archiki Prasad | 52 | 3 | 59 PRInTS: Reward Modeling for Long-Horizon Information Seeking (2026) Reward Modeling and Process Supervision 57 MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning (2025) Reward Modeling and Process Supervision |
|
| 6 | Negar Arabzadeh | 50 | 3 | 64 Enhanced Retrieval Effectiveness through Selective Query Generation (2024) Query Reformulation and Expansion 64 ReFormeR: Learning and Applying Explicit Query Reformulation Patterns (2026) Query Reformulation and Expansion |
|
| 7 | Xiao Wang 0043 | 50 | 3 | 62 Improving zero-shot retrieval using dense external expansion (2022) Query Reformulation and Expansion 61 Neural Pseudo-Relevance Feedback Models for Sparse and Dense Retrieval (2022) Query Reformulation and Expansion |
|
| 9 | Wonduk Seo | 50 | 3 | 65 A New Query Expansion Approach for Enhancing Information Retrieval via Agent-Mediated Dialogic Inquiry (2026) Query Reformulation and Expansion 38 SPIO: Ensemble and Selective Strategies via LLM-Based Multi-Agent Planning in Automated Data Science (2026) Multi-Agent and Tool-Use Orchestration |
|
| 11 | Falk Scholer | 48 | 3 | 64 Can Generative LLMs Create Query Variants for Test Collections? An Exploratory Study (2023) Query Reformulation and Expansion 64 Query association for effective retrieval (2002) Query Reformulation and Expansion |
|
| 13 | Kaixin Ma | 48 | 3 | 49 WebEvolver: Enhancing Web Agent Self-Improvement with Co-evolving World Model (2025) RL Training of Search Agents 49 OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimization (2025) RL Training of Search Agents |
|
| 14 | Qingyi Si | 46 | 3 | 57 S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models (2025) Reward Modeling and Process Supervision 50 Test-time Prompt Intervention (2026) Reward Modeling and Process Supervision |
|
| 16 | Shengyao Zhuang | 45 | 3 | 62 To Interpolate or not to Interpolate: PRF, Dense and Sparse Retrievers (2022) Query Reformulation and Expansion 61 Revisiting Document Expansion and Filtering for Effective First-Stage Retrieval (2024) Query Reformulation and Expansion |
|
| 17 | Idan Szpektor | 45 | 3 | 64 From query to question in one click: suggesting synthetic questions to searchers (2013) Query Reformulation and Expansion 64 Improving recommendation for long-tail queries via templates (2011) Query Reformulation and Expansion |
|
| 19 | Weiwei Sun 0001 COI: co-author | 42 | 3 | 56 ReasonRank: Empowering Passage Ranking with Strong Reasoning Ability (2026) LLM-Based Reranking 45 Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learning (2025) Agentic and Deep Search RAG |
|
| 20 | Shuaiqiang Wang COI: co-author | 42 | 3 | 73 Thinking Forward and Backward: Multi-Objective Reinforcement Learning for Retrieval-Augmented Reasoning (2026) Multi-Hop and Iterative Retrieval Reasoning 54 Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learning (2025) Multi-Hop and Iterative Retrieval Reasoning |
|
| 21 | Tetsuya Sakai COI: co-author | 42 | 3 | 64 Flexible Pseudo-Relevance Feedback Using Optimization Tables (2001) Query Reformulation and Expansion 38 ChatRetriever: Adapting Large Language Models for Generalized and Robust Conversational Dense Retrieval (2024) Conversational Search and Retrieval |
|
| 22 | Wenlin Zhang 0001 | 42 | 3 | 64 Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning (2025) Agentic and Deep Search RAG 53 To Search or Not to Search: Aligning the Decision Boundary of Deep Search Agents via Causal Intervention (2026) Agentic and Deep Search RAG |
|
| 23 | Yingyi Zhang 0001 | 42 | 3 | 64 Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning (2025) Agentic and Deep Search RAG 53 To Search or Not to Search: Aligning the Decision Boundary of Deep Search Agents via Causal Intervention (2026) Agentic and Deep Search RAG |
|
| 24 | Gustavo Penha | 42 | 3 | 61 Improving Content Retrievability in Search with Controllable Query Generation (2023) Query Reformulation and Expansion 61 Evaluating the Robustness of Retrieval Pipelines with Query Variation Generators (2022) Query Reformulation and Expansion |
|
| 26 | Nancy F. Chen | 41 | 3 | 57 Learning Planning-based Reasoning by Trajectories Collection and Process Reward Synthesizing (2024) Reward Modeling and Process Supervision 50 AdaMCoT: Rethinking Cross-Lingual Factual Reasoning Through Adaptive Multilingual Chain-of-Thought (2026) Reward Modeling and Process Supervision |
|
| 27 | Yuanzhao Zhai | 41 | 3 | 64 Empowering Large Language Model Agent through Step-Level Self-Critique and Self-Training (2025) Reward Modeling and Process Supervision 64 Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models (2025) Reward Modeling and Process Supervision |
|
| 28 | Xinyu Ma 0001 COI: co-author | 41 | 3 | 56 ReasonRank: Empowering Passage Ranking with Strong Reasoning Ability (2026) LLM-Based Reranking 45 Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learning (2025) Agentic and Deep Search RAG |
|
| 29 | Jaime Arguello | 40 | 3 | 61 Document Representation and Query Expansion Models for Blog Recommendation (2008) Query Reformulation and Expansion 40 Using Query Performance Predictors to Improve Spoken Queries (2016) Query Performance Prediction |
|
| 30 | Keqin Bao | 40 | 3 | 50 Chain of Execution Supervision Promotes General Reasoning in Large Language Models (2025) Reward Modeling and Process Supervision 38 Item-side Fairness of Large Language Model-based Recommendation System (2024) Preference Alignment and Optimization |
|
| 31 | Xueguang Ma | 40 | 3 | 62 To Interpolate or not to Interpolate: PRF, Dense and Sparse Retrievers (2022) Query Reformulation and Expansion 61 Improving Query Representations for Dense Retrieval with Pseudo Relevance Feedback: A Reproducibility Study (2022) Query Reformulation and Expansion |
|
| 32 | Xuanhui Wang | 40 | 3 | 63 Can Query Expansion Improve Generalization of Strong Cross-Encoder Rankers? (2024) Query Reformulation and Expansion 59 Accurate language model estimation with document expansion (2005) Query Reformulation and Expansion |
|
| 33 | Farzad Shami | 40 | 3 | 37 GROKE: Vision-Free Navigation Instruction Evaluation via Graph Reasoning on OpenStreetMap (2026) Reasoning Evaluation and Benchmarks 36 From Single to Multi-agent Reasoning: Advancing GeneGPT for Genomics QA (2026) Retrieval-Augmented and Open-Domain QA |
|
| 34 | Yu-An Liu 0028 | 40 | 3 | 60 Thinking Forward and Backward: Multi-Objective Reinforcement Learning for Retrieval-Augmented Reasoning (2026) Agentic and Deep Search RAG 51 Is a Busy Search Agent a Good One? Overthinking and Overretrieval at Scale (2026) Reward Modeling and Process Supervision |
|
| 35 | Mark D. Smucker | 40 | 3 | 60 Find-similar: similarity browsing as a search tool (2006) Query Reformulation and Expansion 43 Simulating the Lateral Reader for News Trustworthiness Reports with an Iterative Multi-Agent RAG System (2026) Agentic and Deep Search RAG |
|
| 37 | Yuchen Li 0006 COI: co-author | 40 | 3 | 47 Retain to Refine: Adaptive Online Question Answering via Query Routing and Long-Short Memory (2026) Agentic and Deep Search RAG 41 S2phere: Semi-Supervised Pre-training for Web Search over Heterogeneous Learning to Rank Data (2023) Web-Scale Learning to Rank |