Diagnosing Identifiability in Two-Tower Models for Unbiased Learning to Rank
Abstract
Two-tower models are a popular unbiased learning-to-rank (ULTR) approach for mitigating position bias in clicks. A major challenge in two-tower models is identifiability: whether relevance and position bias can be uniquely determined from clicks. Recent work shows that two-tower models can be identified when the same documents (or documents with similar features) are observed across positions. However, these conditions are defined in infinite data. In practice, we lack methods for diagnosing identifiability in real-world datasets with small sample sizes and limited positional variability. In this work, we present a practical identifiability diagnostic for two-tower models. We quantify whether individual bias parameters are uniquely determined by shifting them away from their optimal values, retraining parts of the model, and measuring whether click-prediction performance degrades significantly. Identified parameters exhibit measurable performance loss when shifted, while unidentified parameters can be freely adjusted without affecting model fit. Our method can distinguish between cases in which identifiability fails due to limitations in model assumptions or data collection and cases caused by a limited sample size. We demonstrate, through simulation, how non-deterministic logging policies, feature overlap, and increased sample size affect identifiability. Applying our method to the real-world Baidu-ULTR dataset, we find that despite large amounts of click data, some parameters of two-tower models remain unidentified, highlighting the practical need for diagnostic methods. All code, data, and results are available at: https://github.com/stanfris/practical-identifiability-ultr
Assigned reviewers
| # | Reviewer | Match | Why (closest evidence publications in the taxonomy) | |
|---|---|---|---|---|
| 1 | Zechun Niu | 62 | 64 Investigating the Robustness of Counterfactual Learning to Rank Models: A Reproducibility Study (2025) Unbiased Learning to Rank 62 Addressing Personalized Bias for Unbiased Learning to Rank (2025) Unbiased Learning to Rank 61 Distributionally Robust Optimization for Unbiased Learning to Rank (2025) Unbiased Learning to Rank |
|
| 2 | Xuanhui Wang | 56 | 100 Revisiting Two-tower Models for Unbiased Learning to Rank (2022) User Behavior and Search Interaction 100 Learning with Sparse and Biased Feedback for Personal Search (2018) User Behavior and Search Interaction 78 Towards Disentangling Relevance and Bias in Unbiased Learning to Rank (2023) User Behavior and Search Interaction |
Area chair
| Area chair | Match | |
|---|---|---|
| Ziyi Ye | 51 |
Area-chair candidates (6)
| Ziyi Ye assigned | 51 | 14 papers | |
| Zhijing Wu 0001 | 42 | 9 papers | |
| Fernando Diaz 0001 | 35 | 12 papers | |
| Chenyan Xiong | 33 | 15 papers | |
| Jun Xu 0001 | 32 | 2 papers | |
| Xiaoxue Zang | 32 | 4 papers |
Candidates from the panel ranked by match score
| # | Reviewer | Match | Load | Why | |
|---|---|---|---|---|---|
| 1 | Sara Allawati | 68 | 3 | 68 A Comparative Analysis of Linguistic and Retrieval Diversity in LLM-Generated Search Queries (2025) User Behavior and Search Interaction 62 What do Users Really Ask Large Language Models? An Initial Log Analysis of Google Bard Interactions in the Wild (2024) User Behavior and Search Interaction |
|
| 2 | Vuong Thanh Tung | 66 | 3 | 70 Incorporating Cognitive Abilities into Web Search Re-ranking (2025) User Behavior and Search Interaction 70 Predicting Representations of Information Needs from Digital Activity Context (2024) User Behavior and Search Interaction |
|
| 3 | Jiqun Liu | 64 | 3 | 62 The Paradox of Personalization: Does Task Prediction Require Individualized Models? (2018) User Behavior and Search Interaction 62 A Reference-Dependent Model for Web Search Evaluation: Understanding and Measuring the Experience of Boundedly Rational Users (2023) User Behavior and Search Interaction |
|
| 4 | Zechun Niu assigned | 62 | 3 | 64 Investigating the Robustness of Counterfactual Learning to Rank Models: A Reproducibility Study (2025) Unbiased Learning to Rank 62 Addressing Personalized Bias for Unbiased Learning to Rank (2025) Unbiased Learning to Rank |
|
| 5 | Fan Zhang 0053 | 61 | 3 | 69 Evaluating Mobile Search with Height-Biased Gain (2017) User Behavior and Search Interaction 68 Incorporating Query Reformulating Behavior into Web Search Evaluation (2021) User Behavior and Search Interaction |
|
| 6 | Romain Deffayet COI: co-author | 58 | 3 | 62 An Offline Metric for the Debiasedness of Click Models (2023) User Behavior and Search Interaction 62 Unbiased Learning to Rank Meets Reality: Lessons from Baidu's Large-Scale Search Dataset (2024) Unbiased Learning to Rank |
|
| 7 | Xuanhui Wang assigned | 56 | 3 | 100 Revisiting Two-tower Models for Unbiased Learning to Rank (2022) User Behavior and Search Interaction 100 Learning with Sparse and Biased Feedback for Personal Search (2018) User Behavior and Search Interaction |
|
| 8 | Jaime Arguello | 53 | 3 | 70 The Effects of Vertical Rank and Border on Aggregated Search Coherence and Search Behavior (2014) User Behavior and Search Interaction 70 The effect of aggregated search coherence on search behavior (2012) User Behavior and Search Interaction |
|
| 10 | Zhen Qin 0001 | 47 | 3 | 100 Revisiting Two-tower Models for Unbiased Learning to Rank (2022) User Behavior and Search Interaction 100 Towards Disentangling Relevance and Bias in Unbiased Learning to Rank (2023) User Behavior and Search Interaction |
|
| 11 | Rolf Jagerman | 47 | 3 | 67 Improving Cloud Storage Search with User Activity (2021) User Behavior and Search Interaction 62 To Model or to Intervene: A Comparison of Counterfactual and Online Learning to Rank from User Interactions (2019) Unbiased Learning to Rank |
|
| 12 | Falk Scholer | 45 | 3 | 71 Can Users Predict Relative Query Effectiveness? (2022) User Behavior and Search Interaction 68 Classifying Term Variants in Query Formulation (2025) User Behavior and Search Interaction |
|
| 13 | Mark D. Smucker | 44 | 3 | 69 Time-based calibration of effectiveness measures (2012) User Behavior and Search Interaction 69 IR Evaluation: Modeling User Behavior for Measuring Effectiveness (2015) User Behavior and Search Interaction |
|
| 14 | Shuaiqiang Wang | 42 | 3 | 87 RankExpert: A Mixture of Textual-and-Behavioral Experts for Multi-Objective Learning-to-Rank in Web Search (2025) User Behavior and Search Interaction 74 M2oERank: Multi-Objective Mixture-of-Experts Enhanced Ranking for Satisfaction-Oriented Web Search (2025) User Behavior and Search Interaction |
|
| 16 | Paul Thomas 0001 | 41 | 3 | 69 User Models, Metrics and Measures of Search: A Tutorial on the C/W/L Evaluation Framework (2021) User Behavior and Search Interaction 66 Large Language Models can Accurately Predict Searcher Preferences (2024) User Behavior and Search Interaction |
|
| 17 | Yifan Wang 0017 | 40 | 3 | 65 User Perception of Recommendation Explanation: Are Your Explanations What Users Need? (2023) User Behavior and Search Interaction 35 Improving Long-tail User CTR Prediction via Hierarchical Distribution Alignment (2025) Bias Mitigation Methods |
|
| 18 | Leif Azzopardi | 39 | 3 | 71 Query quality: user ratings and system predictions (2010) User Behavior and Search Interaction 69 User Models, Metrics and Measures of Search: A Tutorial on the C/W/L Evaluation Framework (2021) User Behavior and Search Interaction |
|
| 19 | Saber Zerhoudi | 39 | 3 | 66 Beyond the Click: A Framework for Inferring Cognitive Traces in Search (2026) User Behavior and Search Interaction 65 From SERPs to Agents: A Platform for Comparative Studies of Information Interaction (2026) User Behavior and Search Interaction |
|
| 20 | Sunhao Dai | 36 | 3 | 65 NExT-Search: Rebuilding User Feedback Ecosystem for Generative AI Search (2025) User Behavior and Search Interaction 34 Counteracting User Attention Bias in Music Streaming Recommendation via Reward Modification (2022) Popularity Bias and Debiasing in Recommendation |
|
| 21 | Yuta Saito | 36 | 3 | 37 Off-Policy Evaluation of Ranking Policies under Diverse User Behavior (2023) Recommendation with Search and Ranking 35 Evaluating the Robustness of Off-Policy Evaluation (2021) Recommendation Systems and Evaluation |
|
| 22 | Yuchen Li 0006 | 35 | 3 | 61 RankExpert: A Mixture of Textual-and-Behavioral Experts for Multi-Objective Learning-to-Rank in Web Search (2025) Unbiased Learning to Rank 35 GS2P: a generative pre-trained learning to rank model with over-parameterization for web-scale search (2024) Web-Scale Learning to Rank |
|
| 24 | Chongming Gao | 34 | 3 | 35 Treatment Effect Estimation for User Interest Exploration on Recommender Systems (2024) Explainable and Causal Recommendation 35 KuaiRec: A Fully-observed Dataset and Insights for Evaluating Recommender Systems (2022) Recommendation Systems and Evaluation |
|
| 25 | Tetsuya Sakai | 34 | 3 | 37 Diversification as Risk Minimization (2026) Evaluation Metrics and User Models 37 Give me just one highly relevant document: P-measure (2006) Evaluation Metrics and User Models |
|
| 26 | Bo Peng 0009 | 34 | 3 | 67 SWaT: Statistical Modeling of Video Watch Time through User Behavior Analysis (2025) User Behavior and Search Interaction 33 eCeLLM: Generalizing Large Language Models for E-commerce from Large-scale, High-quality Instruction Data (2024) Efficiency and Deployment |
|
| 27 | Xiao Zhang 0034 | 34 | 3 | 36 UniSAR: Modeling User Transition Behaviors between Search and Recommendation (2024) Recommendation with Search and Ranking 35 Similarity = Value? Consultation Value-Assessment and Alignment for Personalized Search (2025) Personalized and Context-Aware Retrieval |
|
| 28 | Yang Song 0008 | 34 | 3 | 72 Query-Less: Predicting Task Repetition for NextGen Proactive Search and Recommendation Engines (2016) User Behavior and Search Interaction 35 LabelCraft: Empowering Short Video Recommendations with Automated Label Crafting (2024) Recommendation Systems and Evaluation |
|
| 29 | Yimeng Bai | 34 | 3 | 38 Unconstrained Monotonic Calibration of Predictions in Deep Ranking Systems (2025) CTR and Conversion Rate Prediction 35 LabelCraft: Empowering Short Video Recommendations with Automated Label Crafting (2024) Recommendation Systems and Evaluation |
|
| 30 | Dan Luo 0004 | 34 | 3 | 61 Unbiased Learning-to-Rank Needs Unconfounded Propensity Estimation (2024) Unbiased Learning to Rank 61 Model-based Unbiased Learning to Rank (2023) Unbiased Learning to Rank |
|
| 31 | Gustavo Penha | 33 | 3 | 66 As It Was: Aligning LLM Search Evaluation with Historical User Preferences (2026) User Behavior and Search Interaction 35 Contextualizing Spotify's Audiobook List Recommendations with Descriptive Shelves (2025) Recommendation Systems and Evaluation |
|
| 32 | Yujia Zhou 0002 | 33 | 3 | 63 Intent-Oriented Dynamic Interest Modeling for Personalized Web Search (2024) User Behavior and Search Interaction 63 Incorporating Explicit Subtopics in Personalized Search (2023) User Behavior and Search Interaction |
|
| 35 | Shengyao Zhuang | 32 | 3 | 83 Implicit Feedback for Dense Passage Retrieval: A Counterfactual Approach (2022) User Behavior and Search Interaction 32 Rank-R1: Enhancing Reasoning in LLM-based Document Rerankers via Reinforcement Learning (2026) Reasoning-Intensive Retrieval and Ranking |