Recommender Systems (RS) should provide diverse recommendations, not just relevant ones. Diversity helps handle uncertainty and offers users meaningful choices. The literature proposes various methods to improve diversity, most notably by re-ranking and selecting from a larger set of candidate recommendations. Driven by promising insights from the literature on how to incorporate versatile Large Language Models (LLMs) into the RS pipeline, in this paper we show how LLMs can be used for diversity re-ranking. We prompt LLMs to generate a diverse ranking from a candidate ranking using various prompt templates with different re-ranking instructions in a zero-shot fashion. We conduct experiments testing state-of-the-art LLMs from the GPT and Llama families. We compare their re-ranking capabilities with random re-ranking and various traditional re-ranking methods from the literature. We open-source the code of our experiments for reproducibility. Our findings suggest that the trade-offs (in terms of performance and costs, among others) of LLM-based re-rankers are superior to those of random re-rankers but, as yet, inferior to the ones of traditional re-rankers. However, because LLMs exhibit improved performance on many natural language processing and recommendation tasks and lower inference costs, we can expect LLM-based re-ranking to become more competitive soon.

Cloud computing has seen widespread adoption because it increases the productivity and efficiency of industries and allows for effective scalability of their business [1]. Guaranteeing performance levels is at the core of cloud services and requires huge computational resources, especially with the latest advances in technologies such as Artificial Intelligence and the Internet of Things [2]. Typically, customers subscribe to agreements where cloud providers ensure specific levels of reliability, availability and responsiveness to systems and applications and describe penalties if the service levels are not met. At the same time, massive computational resources are a cost for providers and have a significant environmental impact, which will increase in the future. It is estimated that the energy consumption of data centres (which host cloud services) will grow from 292 TWh in 2016 to 353 TWh in 2030 [3], and greenhouse gas emissions will increase over 14% in 2040, compared to a 1-1.6% increase in the 2007–2016 [4].

Abstract Offline evaluation of recommender systems (RSs) mostly relies on historical data, which is often biased. The bias is a result of many confounders that affect the data collection process. In such biased data, user-item interactions are Missing Not At Random (MNAR). Measures of recommender system performance on MNAR test data are unlikely to be reliable indicators of real-world performance unless something is done to mitigate the bias. One widespread way that researchers try to obtain less biased offline evaluation is by designing new, supposedly unbiased performance metrics for use on MNAR test data. We investigate an alternative solution, a sampling approach . The general idea is to use a sampling strategy on MNAR data to generate an intervened test set with less bias — one in which interactions are Missing At Random (MAR) or, at least, one that is more MAR-like. An existing example of this approach is SKEW, a sampling strategy that aims to adjust for the confounding effect that an item’s popularity has on its likelihood of being observed. In this paper, after extensively surveying the literature on the bias problem in the offline evaluation of RSs, we propose and formulate a novel sampling approach, which we call WTD; we also propose a more practical variant, which we call WTD_H. We compare our methods to SKEW and to two baselines which perform a random intervention on MNAR data. We empirically validate for the first time the effectiveness of SKEW and we show our approach to be a better estimator of the performance that one would obtain on (unbiased) MAR test data. Our strategy benefits from high generality (e.g. it can also be employed for training a recommender) and low overheads (e.g. it does not require any learning).