Recommender systems are one of the most widely used applications of artificial intelligence.Their use can have far-reaching consequences for stakeholders, users, and society at large.In this third edition of the NORMalize workshop, we once again seek to advance the research agenda of normative thinking, considering the norms and values that underpin recommender systems, as well as to introduce the concept to a broader audience.We aim to bring together a growing community of researchers and practitioners across disciplines who want to think about the norms and values that should be considered in the design and evaluation of recommender systems, and to further educate them on how to reflect on, prioritise, and operationalise such norms and values.NORMalize 2025 is a half-day workshop focusing on discussion and interdisciplinary collaboration, building upon its two successful runs at previous RecSys conferences in 2023 and 2024.

Many news recommender systems (NRS) adopt a one-recommenderfor-all approach, overlooking that users engage with news in fundamentally different ways.In this work, we identify user clusters based on various engagement metrics that go beyond clicks by employing cluster analysis on two real-world datasets: EB-NeRD and Adressa.Next to that, we evaluate the performance of common recommender strategies: popularity, collaborative filtering (EASE and ItemKNN), and a content-based model across these user clusters, which exhibit varying reading behaviors and information needs.Our findings show that different recommender strategies are effective to varying degrees depending on the user cluster.This study contributes to NRS research by providing a grounded clustering of users derived from real-world datasets and emphasizes the importance of user-centered evaluations for understanding how NRS strategies serve audiences with varying levels of news engagement.

Information access systems, such as Google News or YouTube, increasingly employ algorithms to rank diverse content such as music, recipes, and news articles. Acknowledging the influential role of these algorithms as gatekeepers to online content, the research community is increasingly exploring ‘beyond-accuracy’ metrics. However, deciding what norms and values are relevant and should be prioritized when designing and evaluating information access systems is a challenging task. This tutorial aims to cultivate normative thinking and decision-making in the design and evaluation of information access systems. The tutorial comprises two key components. The first part involves a lecture on the foundational principles of normative thinking, emphasizing the importance of reflecting on the desired state of a system rather than its current state. The second part is an interactive session where participants engage in group discussions, applying normative thinking to a specific use case. Participants analyze the system’s usage, stakeholders, and relevant norms and values and address potential conflicts between stakeholders and/or values. Through a point-allocation exercise, participants represent stakeholders and advocate for specific values, fostering a deeper understanding of normative decision-making in the context of information access systems.

When designing recommender-systems experiments, a key question that has been largely overlooked is the choice of datasets. In a brief survey of ACM RecSys papers, we found that authors typically justified their dataset choices by labelling them as public, benchmark, or ‘real-world’ without further explanation. We propose the Algorithm Performance Space (APS) as a novel method for informed dataset selection. The APS is an n-dimensional space where each dimension represents the performance of a different algorithm. Each dataset is depicted as an n-dimensional vector, with greater distances indicating higher diversity. In our experiment, we ran 29 algorithms on 95 datasets to construct an actual APS. Our findings show that many datasets, including most Amazon datasets, are clustered closely in the APS, i.e. they are not diverse. However, other datasets, such as MovieLens and Docear, are more dispersed. The APS also enables the grouping of datasets based on the solvability of the underlying problem. Datasets in the top right corner of the APS are considered ’solved problems’ because all algorithms perform well on them. Conversely, datasets in the bottom left corner lack well-performing algorithms, making them ideal candidates for new recommender-system research due to the challenges they present.

Previous research has used Large Language Models (LLMs) to develop personalized Conversational Recommender Systems (CRS) with text-based user interfaces (UIs). However, the potential of LLMs to generate interactive graphical elements that enhance user experience remains largely unexplored. To address this gap, we introduce "GenUI(ne) CRS," a novel framework designed to leverage LLMs for adaptive and interactive UIs. Our framework supports domain-specific graphical elements such as buttons and cards, in addition to text-based inputs. It also addresses the common LLM issue of outdated knowledge, known as the "knowledge cut-off," by implementing Retrieval-Augmented Generation (RAG). To illustrate its potential, we developed a prototype movie CRS. This work demonstrates the feasibility of LLM-powered interactive UIs and paves the way for future CRS research, including user experience validation, transparent explanations, and addressing LLM biases.

Recommender systems are among the most widely used applications of artificial intelligence. Their use can have far-reaching consequences for users, stakeholders, and society at large. In this second edition of the NORMalize workshop, we once again seek to advance the research agenda of normative thinking, considering the norms and values that underpin recommender systems, as well as to introduce the concept to a broader audience. We aim to bring together a growing community of researchers and practitioners across disciplines who want to think about the norms and values that should be considered in the design and evaluation of recommender systems, and to further educate them on how to reflect on, prioritise, and operationalise such norms and values. NORMalize 2024 is a half-day workshop consisting of a combination of paper presentations and an interactive session, building upon its successful full-day run last year at RecSys’23.

Evaluating recommender systems adequately and thoroughly is an important task. Significant efforts are dedicated to proposing metrics, methods, and protocols for doing so. However, there has been little discussion in the recommender systems’ literature on the topic of testing. In this work, we adopt and adapt concepts from the software testing domain, e.g., code coverage, metamorphic testing, or property-based testing, to help researchers to detect and correct faults in recommendation algorithms. We propose a test suite that can be used to validate the correctness of a recommendation algorithm, and thus identify and correct issues that can affect the performance and behavior of these algorithms. Our test suite contains both black box and white box tests at every level of abstraction, i.e., system, integration, and unit. To facilitate adoption, we release RecPack Tests , an open-source Python package containing template test implementations. We use it to test four popular Python packages for recommender systems: RecPack , PyLensKit , Surprise , and Cornac . Despite the high test coverage of each of these packages, we find that we are still able to uncover undocumented functional requirements and even some bugs. This validates our thesis that testing the correctness of recommendation algorithms can complement traditional methods for evaluating recommendation algorithms.

News media play an important role in democratic societies. Central to fulfilling this role is the premise that users should be exposed to diverse news. However, news recommender systems are gaining popularity on news websites, which has sparked concerns over filter bubbles. More specifically, editors, policy-makers and scholars are worried that these news recommender systems may expose users to less diverse content over time. To the best of our knowledge, this hypothesis has not been tested in a longitudinal observational study of real users that interact with a real news website. Such observational studies require the use of research methods that are robust and can account for the many covariates that may influence the diversity of recommendations at any given time. In this work, we propose an analysis model to study whether the variety of articles recommended to a user decreases over time in such an observational study design. Further, we present results from two case studies using aggregated and anonymized data that were collected by two western European news websites employing a collaborative filtering-based news recommender system to serve (personalized) recommendations to their users. Through these case studies we validate empirically that our modeling assumptions are sound and supported by the data, and that our model obtains more reliable and interpretable results than analysis methods used in prior empirical work on filter bubbles. Our case studies provide evidence of a small decrease in the topic variety of a user’s recommendations in the first weeks after they sign up, but no evidence of a decrease in political variety.

Recommender systems are among the most widely used applications of artificial intelligence. Since they are so widely used, it is important that we, as practitioners and researchers, think about the impact these systems may have on users, society, and other stakeholders. To that effect, the NORMalize workshop seeks to introduce normative thinking, to consider the norms and values that underpin recommender systems in the recommender systems community. The objective of NORMalize is to bring together a growing community of researchers and practitioners across disciplines who want to think about the norms and values that should be considered in the design and evaluation of recommender systems; and further educate them on how to reflect on, prioritise, and operationalise such norms and values. NORMalize offers a comprehensive program designed to cater to both the norm-curious and the norm-active. The morning session is on-site and features a lecture on normative thinking and an interactive workshop. The afternoon is a hybrid program focused on the dissemination of results. NORMalize publishes proceedings, as well as a technical report that summarises the outcomes of the interactive morning session.

RecPack is an easy-to-use, flexible and extensible toolkit for top-N recommendation with implicit feedback data. Its goal is to support researchers with the development of their recommendation algorithms, from similarity-based to deep learning algorithms, and allow for correct, reproducible and reusable experimentation. In this demo, we give an overview of the package and show how researchers can use it to their advantage when developing recommendation algorithms.