Music recommendation has been a core area of interest within the recommender systems community since its early days. With the rise of music streaming platforms, algorithmic recommendations have become central to the music industry. However, critical challenges remain concerning the design, evaluation, and societal impact of music recommender systems. This third edition of the Music Recommender Systems Workshop (MuRS) centers on the growing influence of generative content on music recommendation. The rapid influx of AI-generated music transforms the streaming ecosystem, raising critical questions about discoverability, authenticity, and the curational role of recommender systems. The challenges and opportunities associated with AI-generated content extend beyond music and recommender systems, demanding transparent, fair, and accountable recommendation frameworks that reflect the interests of the diverse set of stakeholders.

As recommender systems are prone to various biases, mitigation approaches are needed to ensure that recommendations are fair to various stakeholders. One particular concern in music recommendation is artist gender fairness. Recent work has shown that the gender imbalance in the sector translates to the output of music recommender systems, creating a feedback loop that can reinforce gender biases over time.

Music recommendation has been relevant to the Recommender Systems (RecSys) community since the early days. With the growth of music streaming platforms, algorithmic recommendations have become critical in the music industry. However, many challenges are still wide open in the area of music recommender systems. Such challenges are currently being addressed in several research communities, including and beyond the RecSys and the Music Information Retrieval (MIR) communities. The RecSys conference has traditionally not focused very much on music content understanding. In contrast, while music content understanding is central to the MIR community, research on recommender systems is not prominent in MIR research. The Music Recommender Systems Workshop (MuRS) aims at bridging the existing gap between the diverse research communities focused on the specific challenges of music recommender systems. The workshop provides a space for researchers and practitioners from multiple disciplines to jointly discuss and exchange perspectives and solutions, and to promote discussion from both academia and industry upon future research directions in the area of music recommender systems.

With the RecSys conference now turning 18 years old, the recommender systems (RS) discipline ventures into adulthood. This workshop serves as a platform for introspection, examining the evolution of RS from its origins in CHI to its current state heavily influenced by and focusing on machine learning. The INTROSPECTIVES workshop aims to foster discussions on the past, present, and future of the RS discipline, inviting the community to reflect on key questions such as the maturation of RS, shifts in research focus, and the impact and success of RS in practice. Topics include the changing landscape of RS problems, the evolving role of RS in addressing choice overload to the current motivations driving RS adoption.

In the recommender systems field, it is increasingly recognized that focusing on accuracy measures is limiting and misguided. Unsurprisingly, in recent years, the field has witnessed more interest in the research of values “beyond accuracy.” This trend is particularly pronounced in the news domain where recommender systems perform parts of the editorial function, required to uphold journalistic values of news organizations. In the literature, various values and approaches have been proposed and evaluated. This article reviews the current state of the proposed news recommender systems (NRS). We perform a systematic literature review, analyzing 183 papers. The primary aim is to study the development, scope, and focus of value-aware NRS over time. In contrast to previous surveys, we are particularly interested in identifying the range of values discussed and evaluated in the context of NRS and embrace an interdisciplinary view. We identified a total of 40 values, categorized into five value groups. Most research on value-aware NRS has taken an algorithmic approach, whereas conceptual discussions are comparably scarce. Often, algorithms are evaluated by accuracy-based metrics, but the values are not evaluated with respective measures. Overall, our work identifies research gaps concerning values that have not received much attention. Values need to be targeted on a more fine-grained and specific level.

Evaluation plays a vital role in recommender systems—in research and practice—whether for confirming algorithmic concepts or assessing the operational validity of designs and applications. It may span the evaluation of early ideas and approaches up to elaborate implementations of systems integrated into everyday product settings; it may target a wide spectrum of different factors being evaluated. In this special issue, we explore recommender systems evaluation—theory and practice—while considering a diverse set of perspectives. These include recommender systems purposes, stakeholders, methodological approaches, and consequences. The collection of articles in this special issue offers insightful analyses of current recommender system evaluation practices, acknowledging their limitations, and setting out future research directions. As recommender systems evolve, the need for adequate evaluation methods and approaches increases. This special issue sheds light on areas undergoing development or requiring added attention from the research and practitioner communities in recommender systems. The compilation serves as a call to the recommender systems research community, motivating continued research and exploration of evaluation metrics, methods, and strategies.

Recommender systems research and practice are fast-developing topics with growing adoption in a wide variety of information access scenarios. In this article, we present an overview of research specifically focused on the evaluation of recommender systems. We perform a systematic literature review, in which we analyze 57 papers spanning six years (2017–2022). Focusing on the processes surrounding evaluation, we dial in on the methods applied, the datasets utilized, and the metrics used. Our study shows that the predominant experiment type in research on the evaluation of recommender systems is offline experimentation and that online evaluations are primarily used in combination with other experimentation methods, e.g., an offline experiment. Furthermore, we find that only a few datasets (MovieLens, Amazon review dataset) are widely used, while many datasets are used in only a few papers each. We observe a similar scenario when analyzing the employed performance metrics—a few metrics are widely used (precision, normalized Discounted Cumulative Gain, and Recall), while many others are used in only a few papers. Overall, our review indicates that beyond-accuracy qualities are rarely assessed. Our analysis shows that the research community working on evaluation has focused on the development of evaluation in a rather narrow scope, with the majority of experiments focusing on a few metrics, datasets, and methods.

FairRecKit is a web-based analysis software that supports researchers in performing, analyzing, and understanding recommendation computations. The idea behind FairRecKit is to facilitate the in-depth analysis of recommendation outcomes considering fairness aspects. With (nested) filters on user or item attributes, metrics can easily be compared across user and item subgroups. Further, (nested) filters can be used on the dataset level; this way, recommendation outcomes can be compared across several sub-datasets to analyze for differences considering fairness aspects. The software currently features five datasets, 11 metrics, and 21 recommendation algorithms to be used in computational experimentation. It is open source and developed in a modular manner to facilitate extension. The analysis software consists of two components: A software package (FairRecKitLib) for running recommendation algorithms on the available datasets and a web-based user interface (FairRecKitApp) to start experiments, retrieve results of previous experiments, and analyze details. The application also comes with extensive documentation and options for result customization, which makes for a flexible tool that supports in-depth analysis.

Evaluation is important when developing and deploying recommender systems. The PERSPECTIVES workshop sheds light on the different, potentially diverging or contradictory perspectives on the evaluation of recommender systems. Building on the discussions and outcomes of the PERSPECTIVES workshops held at RecSys 2021 and 2022, the third edition of the PERSPECTIVES workshop held at RecSys 2023 brought together researchers and practitioners from academia and industry to reflect on the evaluation of recommender systems critically. The workshop featured a keynote and focused on the interactive part with discussions in small groups and the plenum. We discussed problems and lessons learned, encouraged the exchange of the many perspectives on evaluation, and aimed to move the discourse forward within the community.

Evaluation of recommender systems is a central activity when developing recommender systems, both in industry and academia. The second edition of the PERSPECTIVES workshop held at RecSys 2022 brought together academia and industry to critically reflect on the evaluation of recommender systems. In the 2022 edition of PERSPECTIVES, we discussed problems and lessons learned, encouraged the exchange of the various perspectives on evaluation, and aimed to move the discourse forward within the community. We deliberately solicited papers reporting a reflection on problems regarding recommender systems evaluation and lessons learned. The workshop featured interactive parts with discussions in small groups as well as in the plenum, both on-site and online, and an industry keynote.

As recommender systems play an important role in everyday life, there is an increasing pressure that such systems are fair. Besides serving diverse groups of users, recommenders need to represent and serve item providers fairly as well. In interviews with music artists, we identified that gender fairness is one of the artists' main concerns. They emphasized that female artists should be given more exposure in music recommendations. We analyze a widely-used collaborative filtering approach with two public datasets-enriched with gender information-to understand how this approach performs with respect to the artists' gender. To achieve gender balance, we propose a progressive re-ranking method that is based on the insights from the interviews. For the evaluation, we rely on a simulation of feedback loops and provide an in-depth analysis using state-of-the-art performance measures and metrics concerning gen-der fairness.

Evaluation is a cornerstone in the process of developing and deploying recommender systems. The PERSPECTIVES workshop brought together academia and industry to critically reflect on the evaluation of recommender systems. Particularly, the workshop aimed to shed light on the different, and maybe even diverging or contradictory perspectives on the evaluation of recommender systems. Papers reporting a reflection on problems regarding recommender systems evaluation and lessons learned were solicited. The workshop combined flash presentations of accepted papers, a keynote from industry, and an interactive part with discussions in break-out rooms as well as in the plenum. The workshop complemented the program of the main conference as it emphasized problems and lessons learned, fostered exchange integrating various perspectives on evaluation, and sought to move the recommender systems community forward as an outcome of the workshop.

Research in the field of information retrieval and recommendation mostly focuses on one single evaluation method and one single quality objective. On the one hand, many research endeavors focus on system-centric evaluation from an algorithmic perspective and consider the context of use only to a minor extent. On the other hand, there are research endeavors focusing on user-centric approaches to the design and evaluation of systems. However, algorithmic quality and perceived quality of user experience do not necessarily match. Thus, it is essential for system evaluation to substantially integrate multiple evaluation methods that cover a variety of relevant aspects and perspectives. Only such an integrated combination of methods may lead to a deep understanding of users, their behavior, and experience in their interaction with a system. This half-day tutorial follows the objective to raise awareness in the CHIIR community concerning the significance of using multiple methods in the evaluation of information retrieval and recommender systems. The tutorial illustrates the "blind spots'' when using single methods. It introduces the concept of "multi-method evaluation'' and discusses its benefits and challenges. While multi-method evaluations may be designed very flexibly, the tutorial presents broadly-defined basic options of how multiple methods may be integrated in an evaluation design. In group work, participants are encouraged to select and fine-tune a specific design that best matches their research endeavor's purpose.

At the heart of ubiquitous and pervasive computing is the integration of semantically rich contextual information into systems that intelligently adapt their behavior to the context. This paper presents an analysis of the contextual elements considered in the scientific discourse on pervasive computing. To support researchers with positioning their work, this paper explores how well the facets of context are represented and which context elements are particularly important in specific application domains, such as healthcare or traffic. Results suggest that context elements are considered diversely among domains. Context spreads across a long tail of heterogeneous, rather specific context elements. Potential factors explaining this high diversity relate to sensor technology, structure of context information as well as purposes and design of context-aware systems.

Designing layouts that are perceived as pleasant by the viewer is no easy task: it requires a wide variety of skills, including a sense for aesthetics. When numerous documents with different content need to be created, one of the bottlenecks is to manually create appealing layouts for each document. Thus, automation for aesthetic layout creation is becoming increasingly important. Prerequisite for this automation are algorithms to measure aesthetics. While the literature proposes basic theoretical fundamentals and mathematical formulas as aesthetic measures, researchers have not operationalized these measures yet. This paper presents the challenges associated with and the lessons learned from operationalizing 36 aesthetics measures derived from the literature for the context of marketing brochures. We measured the aesthetics of 744 brochure pages from 10 major retailers and found very strong and highly significant correlations between at least 11 of the aesthetic measures, which represent five latent aesthetic concepts. Still, most of the measures were found to be independent in our sample, and they cover a wide range of different aesthetic concepts. Nevertheless, our results suggest that retailers optimize some of these measures more than others. In terms of the aesthetic measures, retailers seem to design brochure pages in the same way regardless of which category products on this page belong to or if it is the first, last, an odd, or an even page. We propose to consider the quality values of aesthetic measures derived from our analysis of the measured brochures as target values for automated document layout creation for aesthetic marketing brochures.

The introduction of crowdsourcing offers numerous business opportunities. In recent years, manifold forms of crowdsourcing have emerged on the market -- also in logistics. Thereby, the ubiquitous availability and sensor-supported assistance functions of mobile devices support crowdsourcing applications, which promotes contextual interactions between users at the right place at the right time. This paper presents the results of an in-depth-analysis on crowdsourcing in logistics in the course of ongoing research in the field of location-based crowdsourcing (LBCS). This paper analyzes LBCS for both, 'classic' logistics as well as 'information' logistics. Real-world examples of crowdsourcing applications are used to underpin the two evaluated types of logistics using crowdsourcing. Potential advantages and challenges of logistics with the crowd ('crowd-logistics') are discussed. Accordingly, this paper aims to provide the necessary basis for a novel interdisciplinary research field.