Unbiased learning to rank (ULTR), which aims to learn unbiased ranking models from biased user behavior logs, plays an important role in Web search. Previous research on ULTR has studied a variety of biases in users' clicks, such as position bias, presentation bias, and outlier bias. However, existing work often assumes that the behavior logs are collected from an ''average'' user, neglecting the differences between different users in their search and browsing behaviors. In this paper, we introduce personalized factors into the ULTR framework, which we term the user-aware ULTR problem. Through a formal causal analysis of this problem, we demonstrate that existing user-oblivious methods are biased when different users have different preferences over queries and personalized propensities of examining documents. To address such a personalized bias, we propose a novel user-aware inverse-propensity-score estimator for learning-to-rank objectives. Specifically, our approach models the distribution of user browsing behaviors for each query and aggregates user-weighted examination probabilities to determine propensities. We theoretically prove that the user-aware estimator is unbiased under some mild assumptions and shows lower variance compared to the straightforward way of calculating a user-dependent propensity for each impression. Finally, we empirically verify the effectiveness of our user-aware estimator by conducting extensive experiments on two semi-synthetic datasets and a real-world dataset.

Unbiased learning to rank (ULTR), which utilizes historical click logs to train ranking models, has attracted much attention in the IR community. Previous studies on ULTR have focused on mitigating a variety of biases in click logs, such as position bias, trust bias, and presentation bias, to recover the true relevance of the query-document pairs. However, they overlooked the intrinsic distribution shifts between the training data and test data. In this paper, we first validate and analyze the distribution shift problem with a real-world ULTR dataset. To solve this problem, we propose distributionally robust unbiased learning to rank (DRO-ULTR) methods. Specifically, we design two kinds of group distributionally robust optimization (group-DRO) frameworks for the existing ULTR methods, one using the pointwise click prediction loss and the other using the listwise counterfactual ranking loss. Finally, we empirically verify the effectiveness of our DRO-ULTR methods by conducting extensive experiments on the real-world dataset.

Counterfactual learning to rank (CLTR) has attracted extensive attention in the IR community for its ability to leverage massive logged user interaction data to train ranking models. While the CLTR models can be theoretically unbiased when the user behavior assumption is correct and the propensity estimation is accurate, their effectiveness is usually empirically evaluated via simulation-based experiments due to a lack of widely available, large-scale, real click logs. However, many previous simulation-based experiments are somewhat limited because they may have one or more of the following deficiencies: 1) using a weak production ranker to generate initial ranked lists, 2) relying on a simplified user simulation model to simulate user clicks, and 3) generating a fixed number of synthetic click logs. As a result, the robustness of CLTR models in complex and diverse situations is largely unknown and needs further investigation.