Research in group recommender systems (GRS) is still largely dominated by offline evaluation or simplified user studies, often relying on simulated groups or tightly controlled synchronous interactions. However, studying realistic group behavior in real-world settings remains challenging due to the need to coordinate multiple users simultaneously. In this work, we present a general framework for studying GRS with real users through distributed interaction without requiring a shared interface. By allowing users to provide preferences independently over time, the framework reduces coordination barriers and enables more realistic group studies. To demonstrate its applicability, we instantiate the framework as Match-it, a mobile application in the movie and restaurant domains. The application supports preference elicitation, iterative group interaction, and multiple recommendation strategies, enabling the collection of rich behavioral data in realistic settings. We argue that this framework provides a practical foundation for investigating user-centric challenges such as fairness, satisfaction, and group dynamics in real-world GRS scenarios. Source codes are available from https://osf.io/5wpbj, while a demonstration video is available from https://youtu.be/MYrbK1zM0lk.

Recommender systems (RS) rely on interaction data between users and items to generate effective results. Originally, RS aimed solely at predicting items’ relevance, but additional (beyond-relevance) quality criteria gained increased attention over time. Objectives such as diversity, novelty, fairness, or serendipity are nowadays at the center of RS research and also among the core components in production systems. Naturally, to properly steer toward such objectives, the system has to gain an understanding of how the users perceive these objectives, to what extent they require them in the recommendations, and how they evaluate the sufficiency of the results w.r.t. these objectives. However, so far, there is no publicly available dataset that would capture all the necessary knowledge. This results in a half-blind algorithmic design and evaluation, where the importance of individual objectives or the metrics for their evaluation cannot be validated from the users’ perspective. To address this issue, we present SM-RS 2.0, an expansion of the original single- and multi-objective recommendations dataset. The dataset links the self-declared propensity toward individual objectives with impressions, item selections, and explicit evaluation of individual quality criteria. Together with the dataset, we also distribute an evaluation framework containing six rather unique tasks that are rarely available to conduct on existing RS datasets. These include impression-aware click prediction, predicting propensity toward individual objectives, construction of proportional recommendations, and predicting the user-perceived fulfillment of individual objectives as well as their overall satisfaction. The dataset is available at https://osf.io/wsakx .

Multi-objective recommender systems (MORS) aim to optimize multiple criteria while generating recommendations, such as relevance, novelty, diversity, or exploration.These algorithms are based on the assumption that an operationalization of these criteria (i.e., translating abstract goals into measurable metrics), will reflect how users perceive them.Nevertheless, such beliefs are rarely rigorously evaluated, which can lead to a mismatch between algorithmic goals and user satisfaction.Moreover, if users are allowed to control the RS via their propensities towards such objectives, the misconceptions may further impact users' trust and engagement.To characterize this problem, we conduct a large user study focusing on recommender systems in two domains: books and movies.Part of the study is focused on how users perceive different recommendation objectives, which we compared with well-established metrics aiming at the same objectives.We found that despite such metrics correlating to some extent with users' perceptions, the mapping is far from perfect.Moreover, we also report on conceptual-level differences in users' understanding of RS objectives and how this affects the results.Study data are available from https://osf.io/2n9mf/.

Group recommender systems (GRS) aim to deliver recommendations to groups of individuals, assisting them in planning activities such as going to the cinema with friends, organizing a family vacation, or dining out with colleagues.Unlike traditional recommender systems (RS), GRS must account for the preferences of multiple individuals, often balancing potentially conflicting goals.In this context, it is crucial to provide recommendations that are perceived as fair by all group members.While numerous aggregation strategies have been proposed, understanding users' perspectives on fairness remains an open challenge.In this paper, we present the results of a user study in which real participants acted as external judges, evaluating the fairness of group recommendations.The study investigates the impact of quantitative explanations, conditioned by specific GRS and group types, on fairness perception.Our findings suggest that, without additional information, the task may be too difficult for users, and their ability to distinguish between different group types is limited, further underscoring the importance of explanations.Study data are available from https://osf.io/9fpyr/.

Recommender systems (RS) rely on interaction data between users and items to generate effective results. Historically, RS aimed to deliver the most consistent (i.e., accurate) items to the trained user profiles. However, the attention towards additional (beyond-accuracy) quality criteria has increased tremendously in recent years. Both the research and applied models are being optimized for diversity, novelty, or fairness, to name a few. Naturally, the proper functioning of such optimization methods depends on the knowledge of users' propensities towards interacting with recommendations having certain quality criteria. However, so far, no dataset that captures such propensities exists. To bridge this research gap, we present SM-RS (single-objective + multi-objective recommendations dataset) that links users' self-declared propensity toward relevance, novelty, and diversity criteria with impressions and corresponding item selections. After presenting the dataset's collection procedure and basic statistics, we propose three tasks that are rarely available to conduct using existing RS datasets: impressions-aware click prediction, users' propensity scores prediction, and construction of recommendations proportional to the users' propensity scores. For each task, we also provide detailed evaluation procedures and competitive baselines. The dataset is available at https://osf.io/hkzje/.

In the context of recommender systems (RS), the concept of diversity is probably the most studied perspective beyond mere accuracy. Despite the extensive development of diversity measures and enhancement methods, the understanding of how users perceive diversity in recommendations remains limited. This gap hinders progress in multi-objective RS, as it challenges the alignment of algorithmic advancements with genuine user needs. Addressing this, our study delves into two key aspects of diversity perception in RS. We investigate user responses to recommendation lists generated using varied diversity metrics but identical diversification thresholds, and lists created with the same metrics but differing thresholds. Our findings reveal a user preference for metadata and content-based diversity metrics over collaborative ones. Interestingly, while users typically recognize more diversified lists as being more diverse in scenarios with significant diversification differences, this perception is not consistently linear and quickly diminishes when the diversification variance between lists is less pronounced. This study sheds light on the nuanced user perceptions of diversity in RS, providing valuable insights for the development of more user-centric recommendation algorithms. Study data and analysis scripts are available from https://osf.io/9y8gx/.

Improvements in the recommender systems (RS) domain are not possible without a thorough way to evaluate and compare newly proposed approaches. User studies represent a viable alternative to online and offline evaluation schemes, but despite their numerous benefits, they are only rarely used. One of the main reasons behind this fact is that preparing a user study from scratch involves a lot of extra work on top of a simple algorithm proposal. To simplify this task, we propose EasyStudy, a modular framework built on the credo “Make simple things fast and hard things possible”. It features ready-to-use datasets, preference elicitation methods, incrementally tuned baseline algorithms, study flow plugins, and evaluation metrics. As a result, a simple study comparing several RS can be deployed with just a few clicks, while more complex study designs can still benefit from a range of reusable components, such as preference elicitation. Overall, EasyStudy dramatically decreases the gap between the laboriousness of offline evaluation vs. user studies and, therefore, may contribute towards the more reliable and insightful user-centric evaluation of next-generation RS. The project repository is available from https://bit.ly/easy-study-repo.

Multi-objective recommender systems (MORS) provide suggestions to users according to multiple (and possibly conflicting) goals. When a system optimizes its results at the individual-user level, it tailors them on a user’s propensity towards the different objectives. Hence, the capability to understand users’ fine-grained needs towards each goal is crucial. In this paper, we present the results of a user study in which we monitored the way users interacted with recommended items, as well as their self-proclaimed propensities towards relevance, novelty, and diversity objectives. The study was divided into several sessions, where users evaluated recommendation lists originating from a relevance-only single-objective baseline as well as MORS. We show that, despite MORS-based recommendations attracting fewer selections, their presence in the early sessions are crucial for users’ satisfaction in the later stages. Surprisingly, the self-proclaimed willingness of users to interact with novel and diverse items is not always reflected in the recommendations they accept. Post-study questionnaires provide insights on how to deal with this matter, suggesting that MORS-based results should be accompanied by elements that allow users to understand the recommendations, so as to facilitate the choice of whether a recommendation should be accepted or not. Detailed study results are available at https://bit.ly/looks-can-be-deceiving-repo.

Going beyond accuracy in the evaluation of a recommender system is an aspect that is receiving more and more attention. Among the many perspectives that can be considered, the impact of presentation bias is of central importance. Under presentation bias, the attention of the users to the items in a recommendation list changes, thus affecting their possibility to be considered and the effectiveness of a model. Page-wise within-subject studies are widely employed in the recommender systems literature to compare algorithms by displaying their results in parallel. However, no study has ever been performed to assess the impact of presentation bias in this context. In this paper, we characterize how presentation bias affects different layout options, which present the results in column- or row-wise fashion. Concretely, we present a user study where six layout variants are proposed to the users in a page-wise within-subject setting, so as to evaluate their perception of the displayed recommendations. Results show that presentation bias impacts users clicking behavior (low-level feedback), but not so much the perceived performance of a recommender system (high-level feedback). Source codes and raw results are available at https://tinyurl.com/PresBiasSIGIR2023.

Group recommender systems (GRS) focus on recommending items to groups of users. GRS need to tackle the heterogeneity of group members’ preferences and produce recommendations of high overall utility while also considering some sense of fairness among group members. This work plans to aim for novel applications of GRS involving construction of large-scale groups of users and focusing on the long-term fairness of these groups which is in contrast with current research that concentrates on small groups of ephemeral nature. We believe that these directions could bring results of significant societal impact and scope of the effect expanding beyond currently considered GRS domains, e.g., helping to mitigate the filter bubble problem

The main focus of our work is the problem of multiple objectives optimization (MOO) while providing a final list of recommendations to the user. Currently, system designers can tune MOO by setting importance of individual objectives, usually in some kind of weighted average setting. However, this does not have to translate into the presence of such objectives in the final results. In contrast, in our work we would like to allow system designers or end-users to directly quantify the required relative ratios of individual objectives in the resulting recommendations, e.g., the final results should have 60% relevance, 30% diversity and 10% novelty. If individual objectives are transformed to represent quality on the same scale, these result conditioning expressions may greatly contribute towards recommendations tuneability and explainability as well as user's control over recommendations.