How Variability Influences Podcast Search: Queries, Transcriptions, and Judges
sigir26-144 · Full research paper · Watheq Ahmad Mansour, J. Shane Culpepper, Andrew Yates, Joel Mackenzie
User Behavior and Search Interaction · 50% Test Collection Construction and Poo · 50%
Abstract
Podcasts have continued to grow in popularity over the last two decades, with more than 4.52 million podcasts and 584 million listeners across the globe in 2025. Developing effective search systems for web-scale podcast corpora is of vital importance. Previous research has approached the search task primarily through text representation using a single transcription of the audio content via automatic speech recognition (ASR) models. However, there is currently limited understanding about how variation in podcast representations influences ranking, retrieval, and relevance assessment.
Assigned reviewers
| # | Reviewer | Match | Why (closest evidence publications in the taxonomy) | |
|---|---|---|---|---|
| 1 | Mark D. Smucker | 58 | 75 A System for Efficient High-Recall Retrieval (2018) User Behavior and Search Interaction 75 Effective User Interaction for High-Recall Retrieval: Less is More (2018) User Behavior and Search Interaction 74 Evaluating Streams of Evolving News Events (2015) User Behavior and Search Interaction |
|
| 2 | Jaime Arguello | 56 | 71 Tip of the Tongue Query Elicitation for Simulated Evaluation (2025) User Simulation for Interactive Systems 71 Multilingual and Domain-Agnostic Tip-of-the-Tongue Query Generation for Simulated Evaluation (2026) User Simulation for Interactive Systems 68 The Effects of Vertical Rank and Border on Aggregated Search Coherence and Search Behavior (2014) User Behavior and Search Interaction |
|
| 3 | Tetsuya Sakai | 54 | 62 On the Ordering of Pooled Web Pages, Gold Assessments, and Bronze Assessments (2024) Test Collection Construction and Pooling 62 WWW3E8: 259, 000 Relevance Labels for Studying the Effect of Document Presentation Order for Relevance Assessors (2021) Test Collection Construction and Pooling 62 Lessons from BMIR-J2: A Test Collection for Japanese IR Systems (1998) Test Collection Construction and Pooling |
Area chair
| Area chair | Match | |
|---|---|---|
| Ziyi Ye | 51 |
Area-chair candidates (7)
| Ziyi Ye assigned | 51 | 14 papers | |
| Zhijing Wu 0001 | 46 | 9 papers | |
| Fernando Diaz 0001 | 36 | 12 papers | |
| Xiaoxi Li 0005 | 33 | 15 papers | |
| Jun Xu 0001 | 32 | 2 papers | |
| Chenyan Xiong | 32 | 15 papers | |
| Xiaoxue Zang | 30 | 4 papers |
Candidates from the panel ranked by match score
| # | Reviewer | Match | Load | Why | |
|---|---|---|---|---|---|
| 1 | Sara Allawati COI: co-author | 67 | 3 | 70 A Comparative Analysis of Linguistic and Retrieval Diversity in LLM-Generated Search Queries (2025) User Behavior and Search Interaction 62 What do Users Really Ask Large Language Models? An Initial Log Analysis of Google Bard Interactions in the Wild (2024) User Behavior and Search Interaction |
|
| 2 | Jiqun Liu | 66 | 3 | 62 Matching Search Result Diversity with User Diversity Acceptance in Web Search Sessions (2022) User Behavior and Search Interaction 62 Constructing and meta-evaluating state-aware evaluation metrics for interactive search systems (2023) User Behavior and Search Interaction |
|
| 3 | Vuong Thanh Tung | 65 | 3 | 69 Does More Context Help? Effects of Context Window and Application Source on Retrieval Performance (2022) User Behavior and Search Interaction 68 Predicting Representations of Information Needs from Digital Activity Context (2024) User Behavior and Search Interaction |
|
| 4 | Fan Zhang 0053 | 64 | 3 | 74 Evaluating Mobile Search with Height-Biased Gain (2017) User Behavior and Search Interaction 72 Incorporating Query Reformulating Behavior into Web Search Evaluation (2021) User Behavior and Search Interaction |
|
| 5 | Falk Scholer COI: co-author | 58 | 3 | 73 On the Use of LLMs for Relevance Labelling (2026) User Behavior and Search Interaction 73 Enhancing Human Annotation: Leveraging Large Language Models and Efficient Batch Processing (2024) User Behavior and Search Interaction |
|
| 6 | Mark D. Smucker assigned | 58 | 3 | 75 A System for Efficient High-Recall Retrieval (2018) User Behavior and Search Interaction 75 Effective User Interaction for High-Recall Retrieval: Less is More (2018) User Behavior and Search Interaction |
|
| 7 | Jaime Arguello assigned | 56 | 3 | 71 Tip of the Tongue Query Elicitation for Simulated Evaluation (2025) User Simulation for Interactive Systems 71 Multilingual and Domain-Agnostic Tip-of-the-Tongue Query Generation for Simulated Evaluation (2026) User Simulation for Interactive Systems |
|
| 8 | Paul Thomas 0001 COI: co-author | 55 | 3 | 74 User Models, Metrics and Measures of Search: A Tutorial on the C/W/L Evaluation Framework (2021) User Behavior and Search Interaction 73 Large Language Models can Accurately Predict Searcher Preferences (2024) User Behavior and Search Interaction |
|
| 9 | Tetsuya Sakai assigned | 54 | 3 | 62 On the Ordering of Pooled Web Pages, Gold Assessments, and Bronze Assessments (2024) Test Collection Construction and Pooling 62 WWW3E8: 259, 000 Relevance Labels for Studying the Effect of Document Presentation Order for Relevance Assessors (2021) Test Collection Construction and Pooling |
|
| 11 | Romain Deffayet | 48 | 3 | 62 Evaluating the Robustness of Click Models to Policy Distributional Shift (2023) User Behavior and Search Interaction 62 An Offline Metric for the Debiasedness of Click Models (2023) User Behavior and Search Interaction |
|
| 13 | Xuanhui Wang | 44 | 3 | 66 Learning with Sparse and Biased Feedback for Personal Search (2018) User Behavior and Search Interaction 66 Revisiting Two-tower Models for Unbiased Learning to Rank (2022) User Behavior and Search Interaction |
|
| 14 | Leif Azzopardi COI: co-author | 43 | 3 | 74 User Models, Metrics and Measures of Search: A Tutorial on the C/W/L Evaluation Framework (2021) User Behavior and Search Interaction 71 Query quality: user ratings and system predictions (2010) User Behavior and Search Interaction |
|
| 15 | Saber Zerhoudi | 41 | 3 | 67 In-Browser Agents for Search Assistance (2026) User Behavior and Search Interaction 66 From SERPs to Agents: A Platform for Comparative Studies of Information Interaction (2026) User Behavior and Search Interaction |
|
| 16 | Zhen Qin 0001 | 38 | 3 | 66 Cross-Positional Attention for Debiasing Clicks (2021) User Behavior and Search Interaction 66 Revisiting Two-tower Models for Unbiased Learning to Rank (2022) User Behavior and Search Interaction |
|
| 17 | Yujia Zhou 0002 | 37 | 3 | 67 Intent-Oriented Dynamic Interest Modeling for Personalized Web Search (2024) User Behavior and Search Interaction 67 Incorporating Explicit Subtopics in Personalized Search (2023) User Behavior and Search Interaction |
|
| 18 | Negar Arabzadeh | 37 | 3 | 62 MS MARCO Chameleons: Challenging the MS MARCO Leaderboard with Extremely Obstinate Queries (2021) Test Collection Construction and Pooling 38 Query Performance Prediction: From Ad-hoc to Conversational Search (2023) Query Performance Prediction |
|
| 19 | Shuaiqiang Wang COI: co-author | 37 | 3 | 71 FULTR: A Large-Scale Fusion Learning to Rank Dataset and Its Application for Satisfaction-Oriented Ranking (2025) User Behavior and Search Interaction 68 M2oERank: Multi-Objective Mixture-of-Experts Enhanced Ranking for Satisfaction-Oriented Web Search (2025) User Behavior and Search Interaction |
|
| 20 | Idan Szpektor | 37 | 3 | 62 Predicting web searcher satisfaction with existing community-based answers (2011) User Behavior and Search Interaction 34 Identifying Web Queries with Question Intent (2016) Personalized and Context-Aware Retrieval |
|
| 21 | Andrew Parry | 36 | 3 | 62 Variations in Relevance Judgments and the Shelf Life of Test Collections (2025) Test Collection Construction and Pooling 36 [inline-graphic not available: see fulltext] SuiteEval: Simplifying Retrieval Benchmarks (2026) Retrieval Toolkits and Reproducibility |
|
| 23 | Gustavo Penha | 33 | 3 | 73 As It Was: Aligning LLM Search Evaluation with Historical User Preferences (2026) User Behavior and Search Interaction 33 Semantic IDs for Joint Generative Search and Recommendation (2025) Recommendation with Search and Ranking |
|
| 25 | Tingyu Song | 33 | 3 | 69 IFIR: A Comprehensive Benchmark for Evaluating Instruction-Following in Expert-Domain Information Retrieval (2025) User Behavior and Search Interaction 29 Rethinking Reasoning-Intensive Retrieval: Evaluating and Advancing Retrievers in Agentic Search Systems (2026) Reasoning-Intensive Retrieval and Ranking |
|
| 27 | Weiwei Sun 0001 | 32 | 3 | 38 Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems (2021) Conversational Search and Retrieval 37 Learning to Tokenize for Generative Retrieval (2023) Generative Retrieval |
|
| 29 | Rolf Jagerman | 32 | 3 | 67 Improving Cloud Storage Search with User Activity (2021) User Behavior and Search Interaction 35 Reliable Confidence Intervals for Information Retrieval Evaluation Using Generative A.I (2024) LLM-Based Relevance Assessment |
|
| 30 | Keping Bi | 32 | 3 | 65 A Study of Context Dependencies in Multi-page Product Search (2019) User Behavior and Search Interaction 38 Beyond Relevance: Utility-Centric Retrieval in the LLM Era (2026) Evidence Selection and Context Refinement |
|
| 31 | Yong Li 0004 | 31 | 3 | 62 Towards an Inclusive Mobile Web: A Dataset and Framework for Focusability in UI Accessibility (2025) User Behavior and Search Interaction 30 PG-Agent: An Agent Powered by Page Graph (2025) Multi-Agent and Tool-Use Orchestration |
|
| 32 | Jingtao Zhan | 31 | 3 | 70 Query Augmentation with Brain Signals (2024) User Behavior and Search Interaction 62 Evaluating Interpolation and Extrapolation Performance of Neural Retrieval Models (2022) Test Collection Construction and Pooling |
|
| 33 | Xiao Zhang 0034 | 31 | 3 | 33 LargePiG for Hallucination-Free Query Generation: Your Large Language Model is Secretly a Pointer Generator (2025) Knowledge Conflict and Faithfulness 33 ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretability (2025) Knowledge Conflict and Faithfulness |
|
| 34 | Jialong Wu 0007 | 31 | 3 | 34 OmniThink: Expanding Knowledge Boundaries in Machine Writing through Thinking (2025) Deep Research and Report Generation 33 Nested Browser-Use Learning for Agentic Information Seeking (2026) Deep Research and Report Generation |
|
| 35 | Farzad Shami | 30 | 3 | 39 Doctron: A Web-based Collaborative Annotation Tool for Ground Truth Creation in IR (2025) Retrieval Toolkits and Reproducibility 29 From Single to Multi-agent Reasoning: Advancing GeneGPT for Genomics QA (2026) Retrieval-Augmented and Open-Domain QA |
|
| 36 | Corby Rosset | 30 | 3 | 64 LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts (2024) User Behavior and Search Interaction 33 Zero-shot Clarifying Question Generation for Conversational Search (2023) Conversational and Session Search |