This tutorial provides practical training in designing and conducting online user experiments with recommender systems, and in statistically analyzing the results of such experiments. It covers the development of a research question and hypotheses, the selection of study participants, the manipulation of system aspects and measurement of behaviors, perceptions and user experiences, and the evaluation of subjective measurement scales and study hypotheses. Interested parties can find the slides, example datset, and other resources at https://www.usabart.nl/QRMS/.

A users’ preference for a bundle – a set of items that can be purchased together – can be expressed by the utility of this bundle to the user. The multi-attribute utility theory motivate us to characterize the utility of a bundle using its attributes to improve the personalized bundle recommendation systems. This extended abstract for the Doctoral Symposium describes my PhD project for studying the utility of a bundle using its attributes. The steps taken and some preliminary results are presented, with an outline of the future plans.

A rapidly changing news ecosystem presents new challenges to research, media organizations, consumers, and societies. The 10th edition of the International Workshop on News Recommendation and Analytics (INRA) serves to exchange ideas and discuss recent trends, technological advancements, and open problems concerning news. We welcome contributions in scientific articles, demonstrations, and ideas. We strive to bring together researchers, practitioners, and decision-makers to address crucial challenges. The workshop provides an opportunity to learn about recent research and interactively discuss technical and interdisciplinary aspects related to news. Topics of interest include information access systems for news, advances in natural language processing, multi-modality, mis- and disinformation, trust and user experiences, and personalization.

In recommender systems (RS) it is important to ensure that the various (past) areas of interest of a user are reflected with their corresponding proportions in the recommendation lists. In other words, when a user has watched, say, 60 romance movies and 40 Comedy movies, then it is reasonable to expect the personalized list of recommended movies to contain about 60% romance and 40% comedy movies as well. This property is known as calibration, and it has recently received much attention in the RS community. Greedy heuristic approaches have been proposed to calibrate recommendations, and although they provide great improvements, they can result in inefficient solutions in that a better one can be missed because of the myopic nature of these algorithms. This paper addresses the calibration problem from a constrained optimization perspective and provides a model to combine both accuracy and calibration. Experimental results show that our approach outperforms the state-of-the-art heuristics for calibration in most cases on both accuracy of the recommendations and the level of calibrations the recommendation lists achieve. We give a small example to illustrate why the heuristic fails to find the optimal solution.

In research practice, recommender systems are typically evaluated on their ability to provide items that satisfy the needs and interests of the end user. However, in many recommendation domains, the user for whom recommendations are generated is not the only stakeholder in the recommendation outcome. For example, fairness and balance across stakeholders is important in some recommendation applications; achieving a goal such as promoting new sellers in a marketplace might be important in others. Such multistakeholder environments present unique challenges for recommender system design and evaluation, and these challenges were the focus of this workshop.

Recommender systems are typically designed to optimize the utility of the end user. In many settings, however, the end user is not the only stakeholder and this exclusive focus may produce unsatisfactory results for other stakeholders. One such setting is found in multisided platforms, which bring together buyers and sellers. In such platforms, it may be necessary to jointly optimize the value for both buyers and sellers. This paper proposes a constraint-based integer programming optimization model, in which different sets of constraints are used to reflect the goals of the different stakeholders. This model is applied as a post-processing step, so it can easily be added onto an existing recommendation system to make it multi-stakeholder aware. For computational tractability with larger data sets, we reformulate the integer problem using the Lagrangian dual and use subgradient optimization. In experiments with two data sets, we evaluate empirically the interaction between the utilities of buyers and sellers and show that our approximation can achieve good upper and lower bounds in practical situations.

We study the effect of the volume of consumer reviews on the purchase likelihood (conversion rate) of users browsing a product page. We propose using the exponential learning curve model to study how conversion rates change with the number of reviews. We call the difference in conversion rate between having no reviews and an infinite number the value of reviews. We find that, on average, the conversion rate of a product can increase by as much as 270% as it accumulates reviews, amongst the users that choose to display them. We also find diminishing marginal value as a product accumulates reviews, with the first five reviews driving the bulk of the aforementioned increase. To address the problem of simultaneity of increase of reviews and conversion rate, we use customer sessions in which reviews were not displayed as a control for trends that would have happened regardless of the increase in the review volume. Using our framework, we further find that high priced items have a higher value for reviews than lower priced items. High priced items can see their conversion rate increase by as much as 380% as they accumulate reviews compared to 190% for low priced items.We infer that the existence of reviews provides valuable signals to the customers, increasing their propensity to purchase. We also infer that users usually don't pay attention to the entire set of reviews, especially if there are a lot of them, but instead they focus on the first few available. Our approach can be extended and applied in a variety of settings to gain further insights.