Neural rankers often reflect societal biases (e.g., gender) present in training data. Existing methods typically require protected-attribute labels or model modifications. We introduce a simple, model-agnostic approach that uses the model's loss as a bias proxy within a self-paced learning curriculum. By prioritizing lower-loss examples and gradually incorporating harder ones, our method reduces reliance on biased data without demographic annotations. We theoretically show a monotonic decrease in the gender loss gap and empirically demonstrate reduced bias with comparable or improved ranking performance on MS MARCO, outperforming strong baselines.

The varying performance of information retrieval (IR) methods, including state-of-the-art transformer-based neural retrievers, across diverse queries poses a significant challenge for achieving robust and reliable retrieval effectiveness. Query Performance Prediction (QPP) seeks to estimate the effectiveness of a retrieval method for individual queries, enabling adaptive strategies to improve retrieval outcomes, particularly for challenging queries. However, existing QPP approaches face fundamental challenges: pre-retrieval methods often rely on surface-level query features that fail to capture the nuanced relationship between queries and retrieval effectiveness, while post-retrieval methods depend heavily on the quality of retrieved documents, which can be unreliable for difficult queries. To this end, we propose the Query Space Distance-Based QPP ( QSD-QPP ) framework, which leverages the deterministic and consistent behavior of retrieval methods to estimate query performance by referencing historical queries with known effectiveness. The approach is motivated by the observation that semantically or syntactically similar queries often exhibit consistent retrieval performance, a property that can be exploited to make reliable predictions for unseen queries. QSD-QPP operates in two modes: (1) a lightweight pre-retrieval instantiation that dynamically constructs a query subspace based on embedding distances to interpolate the performance of proximate historical queries, and (2) an enriched post-retrieval instantiation that incorporates contextualized embeddings, document interactions, and historical query associations to enhance prediction accuracy. By utilizing large-scale contextualized embeddings derived from pre-trained language models, QSD-QPP efficiently identifies semantically similar queries and leverages their performance for robust predictions. By addressing the inherent limitations of prior approaches, QSD-QPP achieves a balanced trade-off between computational efficiency, prediction accuracy, and scalability. We evaluate QSD-QPP on four benchmark datasets, including MS MARCO Dev and TREC Deep Learning tracks (2019, 2020, and DL-Hard), demonstrating its superior accuracy and robustness compared to state-of-the-art baselines in both pre-retrieval and post-retrieval QPP tasks. To ensure reproducibility and encourage further research, we publicly release the implementation of our work.

Neural ranking models are widely used to retrieve and rank relevant documents. However, these models may inherit and amplify biases present in the training data, posing challenges for fairness and relevance in ranking outputs. In this paper, we propose a novel curriculum-based training approach that manages bias exposure throughout the training process. We design a bias-aware curriculum that stages the exposure of the model to biased samples during the training stages, allowing the model to establish a fair relevance baseline. We conduct extensive experiments across different LLMs and datasets to evaluate the effectiveness of our approach. Our results demonstrate that our proposed strategy outperforms other bias reduction methods in terms of both fairness and relevance, without sacrificing retrieval effectiveness.

Neural rankers have achieved strong retrieval effectiveness but require large amounts of labeled data, limiting their applicability in few-shot settings. In this paper, we address the sample inefficiency of neural ranking methods by introducing a Reinforcement Learning (RL)-based re-ranking model that achieves high effectiveness with minimal training data. Built on a Deep Q-learning Network (DQN) framework, our approach is designed for few-shot settings, maximizing sample efficiency to ensure robust generalization from limited interactions. Extensive experiments show that our model significantly outperforms data-intensive methods and existing few-shot baselines, demonstrating RL's potential to enhance IR capabilities in few-shot scenarios.

Researchers have shown that the retrieval effectiveness of queries may depend on other factors in addition to the semantics of the query. In other words, several queries expressed with the same intent, and even using overlapping keywords, may exhibit completely different degrees of retrieval effectiveness. As such, the objective of our work in this paper is to propose a neural disentanglement method that is able to disentangle query semantics from query difficulty. The disentangled query semantics representation provides the means to determine semantic association between queries whereas the disentangled query difficulty representation would allow for the estimation of query effectiveness. We show through our experiments on the query performance prediction; and, query similarity calculation tasks that our proposed disentanglement method is able to show better performance compared to the state of the art.

Recent studies have shown that it is possible for stereotypical gender biases to find their way into representational and algorithmic aspects of retrieval methods; hence, exhibit themselves in retrieval outcomes. In this tutorial, we inform the audience of various studies that have systematically reported the presence of stereotypical gender biases in Information Retrieval (IR) systems. We further classify existing work on gender biases in IR systems as being related to (1) relevance judgement datasets, (2) structure of retrieval methods, and (3) representations learnt for queries and documents. We present how each of these components can be impacted by or cause intensified biases during retrieval. Based on these identified issues, we then present a collection of approaches from the literature that have discussed how such biases can be measured, controlled, or mitigated. Additionally, we introduce publicly available datasets that are often used for investigating gender biases in IR systems as well as evaluation methodology adopted for determining the utility of gender bias mitigation strategies.

While neural rankers continue to show notable performance improvements over a wide variety of information retrieval tasks, there have been recent studies that show such rankers may intensify certain stereotypical biases. In this paper, we investigate whether neural rankers introduce retrieval effectiveness (performance) disparities over queries related to different genders. We specifically study whether there are significant performance differences between male and female queries when retrieved by neural rankers. Through our empirical study over the MS MARCO collection, we find that such performance disparities are notable and that the performance disparities may be due to the difference between how queries and their relevant judgements are collected and distributed for different gendered queries. More specifically, we observe that male queries are more closely associated with their relevant documents compared to female queries and hence neural rankers are able to more easily learn associations between male queries and their relevant documents. We show that it is possible to systematically balance relevance judgment collections in order to reduce performance disparity between different gendered queries without negatively compromising overall model performance.

Researchers have already shown that it is possible to improve retrieval effectiveness through the systematic reformulation of users' queries. Traditionally, most query reformulation techniques relied on unsupervised approaches such as query expansion through pseudo-relevance feedback. More recently and with the increasing effectiveness of neural sequence-to-sequence architectures, the problem of query reformulation has been studied as a supervised query translation problem, which learns to rewrite a query into a more effective alternative. While quite effective in practice, such supervised query reformulation methods require a large number of training instances. In this paper, we present three large-scale query reformulation datasets, namely Diamond, Platinum and Gold datasets, based on the queries in the MS MARCO dataset. The Diamond dataset consists of over 188,000 query pairs where the original source query is matched with an alternative query that has a perfect retrieval effectiveness (an average precision of 1). To the best of our knowledge, this is the first set of datasets for supervised query reformulation that offers perfect query reformulations for a large number of queries. The implementation of our fully automated tool, which is based on a transformer architecture, and our three datasets are made publicly available. We also establish a neural query reformulation baseline performance on our datasets by reporting the performance of strong neural query reformulation baselines. It is our belief that our datasets will significantly impact the development of supervised query reformulation methods in the future.

Various researchers have recently explored the impact of different types of biases on information retrieval tasks such as ad hoc retrieval and question answering. While the impact of bias needs to be controlled in order to avoid increased prejudices, the literature has often viewed the relationship between increased retrieval utility (effectiveness) and reduced bias as a tradeoff where one can suffer from the other. In this paper, we empirically study this tradeoff and explore whether it would be possible to reduce bias while maintaining similar retrieval utility. We show this would be possible by revising the input query through a bias-aware pseudo-relevance feedback framework. We report our findings based on four widely used TREC corpora namely Robust04, Gov2, ClueWeb09 and ClueWeb12 and using two classes of bias metrics. The findings of this paper are significant as they are among the first to show that decrease in bias does not necessarily need to come at the cost of reduced utility.