Demonstration venue · read-only. Every page can be browsed; the buttons that would change it are switched off. Create an account to run TaxoReview on your own data. A simulation from public publication records: the people listed were not invited and did not agree to review.
? Tutorial for this page

Diagnosing Identifiability in Two-Tower Models for Unbiased Learning to Rank

sigir26-116 · Full research paper · Stan Fris, Philipp Hager
User Behavior and Search Interaction · 50% Unbiased Learning to Rank · 50%
Abstract

Two-tower models are a popular unbiased learning-to-rank (ULTR) approach for mitigating position bias in clicks. A major challenge in two-tower models is identifiability: whether relevance and position bias can be uniquely determined from clicks. Recent work shows that two-tower models can be identified when the same documents (or documents with similar features) are observed across positions. However, these conditions are defined in infinite data. In practice, we lack methods for diagnosing identifiability in real-world datasets with small sample sizes and limited positional variability. In this work, we present a practical identifiability diagnostic for two-tower models. We quantify whether individual bias parameters are uniquely determined by shifting them away from their optimal values, retraining parts of the model, and measuring whether click-prediction performance degrades significantly. Identified parameters exhibit measurable performance loss when shifted, while unidentified parameters can be freely adjusted without affecting model fit. Our method can distinguish between cases in which identifiability fails due to limitations in model assumptions or data collection and cases caused by a limited sample size. We demonstrate, through simulation, how non-deterministic logging policies, feature overlap, and increased sample size affect identifiability. Applying our method to the real-world Baidu-ULTR dataset, we find that despite large amounts of click data, some parameters of two-tower models remain unidentified, highlighting the practical need for diagnostic methods. All code, data, and results are available at: https://github.com/stanfris/practical-identifiability-ultr

Assigned reviewers

#ReviewerMatchWhy (closest evidence publications in the taxonomy)
1 Zechun Niu
62
64 Investigating the Robustness of Counterfactual Learning to Rank Models: A Reproducibility Study (2025) Unbiased Learning to Rank
62 Addressing Personalized Bias for Unbiased Learning to Rank (2025) Unbiased Learning to Rank
61 Distributionally Robust Optimization for Unbiased Learning to Rank (2025) Unbiased Learning to Rank
2 Xuanhui Wang
56
100 Revisiting Two-tower Models for Unbiased Learning to Rank (2022) User Behavior and Search Interaction
100 Learning with Sparse and Biased Feedback for Personal Search (2018) User Behavior and Search Interaction
78 Towards Disentangling Relevance and Bias in Unbiased Learning to Rank (2023) User Behavior and Search Interaction

Area chair

Area chairMatch
Ziyi Ye 51
Area-chair candidates (6)
Ziyi Ye assigned 5114 papers
Zhijing Wu 0001 429 papers
Fernando Diaz 0001 3512 papers
Chenyan Xiong 3315 papers
Jun Xu 0001 322 papers
Xiaoxue Zang 324 papers

Candidates from the panel ranked by match score

#ReviewerMatchLoadWhy
1 Sara Allawati 68 3
68 A Comparative Analysis of Linguistic and Retrieval Diversity in LLM-Generated Search Queries (2025) User Behavior and Search Interaction
62 What do Users Really Ask Large Language Models? An Initial Log Analysis of Google Bard Interactions in the Wild (2024) User Behavior and Search Interaction
2 Vuong Thanh Tung 66 3
70 Incorporating Cognitive Abilities into Web Search Re-ranking (2025) User Behavior and Search Interaction
70 Predicting Representations of Information Needs from Digital Activity Context (2024) User Behavior and Search Interaction
3 Jiqun Liu 64 3
62 The Paradox of Personalization: Does Task Prediction Require Individualized Models? (2018) User Behavior and Search Interaction
62 A Reference-Dependent Model for Web Search Evaluation: Understanding and Measuring the Experience of Boundedly Rational Users (2023) User Behavior and Search Interaction
4 Zechun Niu assigned 62 3
64 Investigating the Robustness of Counterfactual Learning to Rank Models: A Reproducibility Study (2025) Unbiased Learning to Rank
62 Addressing Personalized Bias for Unbiased Learning to Rank (2025) Unbiased Learning to Rank
5 Fan Zhang 0053 61 3
69 Evaluating Mobile Search with Height-Biased Gain (2017) User Behavior and Search Interaction
68 Incorporating Query Reformulating Behavior into Web Search Evaluation (2021) User Behavior and Search Interaction
6 Romain Deffayet COI: co-author 58 3
62 An Offline Metric for the Debiasedness of Click Models (2023) User Behavior and Search Interaction
62 Unbiased Learning to Rank Meets Reality: Lessons from Baidu's Large-Scale Search Dataset (2024) Unbiased Learning to Rank
7 Xuanhui Wang assigned 56 3
100 Revisiting Two-tower Models for Unbiased Learning to Rank (2022) User Behavior and Search Interaction
100 Learning with Sparse and Biased Feedback for Personal Search (2018) User Behavior and Search Interaction
8 Jaime Arguello 53 3
70 The Effects of Vertical Rank and Border on Aggregated Search Coherence and Search Behavior (2014) User Behavior and Search Interaction
70 The effect of aggregated search coherence on search behavior (2012) User Behavior and Search Interaction
10 Zhen Qin 0001 47 3
100 Revisiting Two-tower Models for Unbiased Learning to Rank (2022) User Behavior and Search Interaction
100 Towards Disentangling Relevance and Bias in Unbiased Learning to Rank (2023) User Behavior and Search Interaction
11 Rolf Jagerman 47 3
67 Improving Cloud Storage Search with User Activity (2021) User Behavior and Search Interaction
62 To Model or to Intervene: A Comparison of Counterfactual and Online Learning to Rank from User Interactions (2019) Unbiased Learning to Rank
12 Falk Scholer 45 3
71 Can Users Predict Relative Query Effectiveness? (2022) User Behavior and Search Interaction
68 Classifying Term Variants in Query Formulation (2025) User Behavior and Search Interaction
13 Mark D. Smucker 44 3
69 Time-based calibration of effectiveness measures (2012) User Behavior and Search Interaction
69 IR Evaluation: Modeling User Behavior for Measuring Effectiveness (2015) User Behavior and Search Interaction
14 Shuaiqiang Wang 42 3
87 RankExpert: A Mixture of Textual-and-Behavioral Experts for Multi-Objective Learning-to-Rank in Web Search (2025) User Behavior and Search Interaction
74 M2oERank: Multi-Objective Mixture-of-Experts Enhanced Ranking for Satisfaction-Oriented Web Search (2025) User Behavior and Search Interaction
16 Paul Thomas 0001 41 3
69 User Models, Metrics and Measures of Search: A Tutorial on the C/W/L Evaluation Framework (2021) User Behavior and Search Interaction
66 Large Language Models can Accurately Predict Searcher Preferences (2024) User Behavior and Search Interaction
17 Yifan Wang 0017 40 3
65 User Perception of Recommendation Explanation: Are Your Explanations What Users Need? (2023) User Behavior and Search Interaction
35 Improving Long-tail User CTR Prediction via Hierarchical Distribution Alignment (2025) Bias Mitigation Methods
18 Leif Azzopardi 39 3
71 Query quality: user ratings and system predictions (2010) User Behavior and Search Interaction
69 User Models, Metrics and Measures of Search: A Tutorial on the C/W/L Evaluation Framework (2021) User Behavior and Search Interaction
19 Saber Zerhoudi 39 3
66 Beyond the Click: A Framework for Inferring Cognitive Traces in Search (2026) User Behavior and Search Interaction
65 From SERPs to Agents: A Platform for Comparative Studies of Information Interaction (2026) User Behavior and Search Interaction
20 Sunhao Dai 36 3
65 NExT-Search: Rebuilding User Feedback Ecosystem for Generative AI Search (2025) User Behavior and Search Interaction
34 Counteracting User Attention Bias in Music Streaming Recommendation via Reward Modification (2022) Popularity Bias and Debiasing in Recommendation
21 Yuta Saito 36 3
37 Off-Policy Evaluation of Ranking Policies under Diverse User Behavior (2023) Recommendation with Search and Ranking
35 Evaluating the Robustness of Off-Policy Evaluation (2021) Recommendation Systems and Evaluation
22 Yuchen Li 0006 35 3
61 RankExpert: A Mixture of Textual-and-Behavioral Experts for Multi-Objective Learning-to-Rank in Web Search (2025) Unbiased Learning to Rank
35 GS2P: a generative pre-trained learning to rank model with over-parameterization for web-scale search (2024) Web-Scale Learning to Rank
24 Chongming Gao 34 3
35 Treatment Effect Estimation for User Interest Exploration on Recommender Systems (2024) Explainable and Causal Recommendation
35 KuaiRec: A Fully-observed Dataset and Insights for Evaluating Recommender Systems (2022) Recommendation Systems and Evaluation
25 Tetsuya Sakai 34 3
37 Diversification as Risk Minimization (2026) Evaluation Metrics and User Models
37 Give me just one highly relevant document: P-measure (2006) Evaluation Metrics and User Models
26 Bo Peng 0009 34 3
67 SWaT: Statistical Modeling of Video Watch Time through User Behavior Analysis (2025) User Behavior and Search Interaction
33 eCeLLM: Generalizing Large Language Models for E-commerce from Large-scale, High-quality Instruction Data (2024) Efficiency and Deployment
27 Xiao Zhang 0034 34 3
36 UniSAR: Modeling User Transition Behaviors between Search and Recommendation (2024) Recommendation with Search and Ranking
35 Similarity = Value? Consultation Value-Assessment and Alignment for Personalized Search (2025) Personalized and Context-Aware Retrieval
28 Yang Song 0008 34 3
72 Query-Less: Predicting Task Repetition for NextGen Proactive Search and Recommendation Engines (2016) User Behavior and Search Interaction
35 LabelCraft: Empowering Short Video Recommendations with Automated Label Crafting (2024) Recommendation Systems and Evaluation
29 Yimeng Bai 34 3
38 Unconstrained Monotonic Calibration of Predictions in Deep Ranking Systems (2025) CTR and Conversion Rate Prediction
35 LabelCraft: Empowering Short Video Recommendations with Automated Label Crafting (2024) Recommendation Systems and Evaluation
30 Dan Luo 0004 34 3
61 Unbiased Learning-to-Rank Needs Unconfounded Propensity Estimation (2024) Unbiased Learning to Rank
61 Model-based Unbiased Learning to Rank (2023) Unbiased Learning to Rank
31 Gustavo Penha 33 3
66 As It Was: Aligning LLM Search Evaluation with Historical User Preferences (2026) User Behavior and Search Interaction
35 Contextualizing Spotify's Audiobook List Recommendations with Descriptive Shelves (2025) Recommendation Systems and Evaluation
32 Yujia Zhou 0002 33 3
63 Intent-Oriented Dynamic Interest Modeling for Personalized Web Search (2024) User Behavior and Search Interaction
63 Incorporating Explicit Subtopics in Personalized Search (2023) User Behavior and Search Interaction
35 Shengyao Zhuang 32 3
83 Implicit Feedback for Dense Passage Retrieval: A Counterfactual Approach (2022) User Behavior and Search Interaction
32 Rank-R1: Enhancing Reasoning in LLM-based Document Rerankers via Reinforcement Learning (2026) Reasoning-Intensive Retrieval and Ranking