Hyperparameter optimization is critical for improving the performance of recommender systems, yet its implementation is often treated as a neutral or secondary concern. In this work, we shift focus from model benchmarking to auditing the behavior of RecBole, a widely used recommendation framework. We show that RecBole's internal defaults, particularly an undocumented early-stopping policy, can prematurely terminate Random Search and Bayesian Optimization. This limits search coverage in ways that are not visible to users. Using six models and two datasets, we compare search strategies and quantify both performance variance and search path instability. Our findings reveal that hidden framework logic can introduce variability comparable to the differences between search strategies. These results highlight the importance of treating frameworks as active components of experimental design and call for more transparent, reproducibility-aware tooling in recommender systems research. We provide actionable recommendations for researchers and developers to mitigate hidden configuration behaviors and improve the transparency of hyperparameter tuning workflows.

Offline evaluation plays a central role in benchmarking recommender systems when online testing is impractical or risky.However, it is susceptible to two key sources of bias: exposure bias, where users only interact with items they are shown, and sampling bias, introduced when evaluation is performed on a subset of logged items rather than the full catalog.While prior work has proposed methods to mitigate sampling bias, these are typically assessed on fixed logged datasets rather than for their ability to support reliable model comparisons under varying exposure conditions or relative to true user preferences.In this paper, we investigate how different combinations of logging and sampling choices affect the reliability of offline evaluation.Using a fully observed dataset as ground truth, we systematically simulate diverse exposure biases and assess the reliability of common sampling strategies along four dimensions: sampling resolution (recommender model separability), fidelity (agreement with full evaluation), robustness (stability under exposure bias), and predictive power (alignment with ground truth).Our findings highlight when and how sampling distorts evaluation outcomes and offer practical guidance for selecting strategies that yield faithful and robust offline comparisons.

We explore the influence of recommender systems on trust among consumers in the fashion e-commerce domain. Anchoring on the Trust Building Model (TBM) [13], we investigate its adaptability and applicability in the context of interactive communication in recommender systems. Primarily leaning on qualitative data collection methods, namely semi-structured interviews, our work evaluates the classic TBM components – structure assurance, perceived reputation, perceived site quality, perceived web risk, trusting belief, and behavioral intention – affirming their relevance to recommender systems. Furthermore, new components, i.e., perceived service and recommendation quality, previous experience, perceived enjoyment, perceived recommendation authenticity, and intention to share interaction data, were examined in the context of recommender systems. Significantly, our study unveils that trusting beliefs can notably influence TBM’s preliminary behavioral intentions, with the competence belief having the most substantial impact, challenging the conventional TBM findings. The outcomes highlight that consumers place heightened value on the tangible provisions from the company over ethics-based factors like integrity. The proposed refined TBM offers potential in enhancing recommender systems in fashion e-commerce, facilitating a better understanding of consumer behavior and trust dynamics.

The AltRecsys workshop, held in conjunction with the 18th edition of the ACM Conference on Recommender Systems (RecSys) in Bari, Italy, provides a platform for highlighting “alternative” work in recommender systems. Modeled after alt.chi and the CRAFT sessions at the FAccT conference, AltRecSys offers a space to discuss interesting, preliminary, offbeat, unexpected, and critical ideas in recommender systems that do not (yet) fit well into the kinds of publications and formats for the main conference or traditional workshops. This workshop is not a venue to showcase research advances. Instead, it is envisioned as a forum where researchers, (industry) practitioners, and other associated stakeholders can exchange ideas and together identify areas of study and new questions to expand the discussions and research agendas of the RecSys community in future years. The call for contributions and the workshop sessions are centered around the question “what are the vital questions, needs, or opportunities that the RecSys community is currently overlooking?”

Fairness in AI-driven decision-making systems has become a critical concern, especially when these systems directly affect human lives. This paper explores the public’s comprehension of fairness in healthcare recommendations. We conducted a survey where participants selected from four fairness metrics – Demographic Parity, Equal Accuracy, Equalized Odds, and Positive Predictive Value – across different healthcare scenarios to assess their understanding of these concepts. Our findings reveal that fairness is a complex and often misunderstood concept, with a generally low level of public understanding regarding fairness metrics in recommender systems. This study highlights the need for enhanced information and education on algorithmic fairness to support informed decision-making in using these systems. Furthermore, the results suggest that a one-size-fits-all approach to fairness may be insufficient, pointing to the importance of context-sensitive designs in developing equitable AI systems.

With the RecSys conference now turning 18 years old, the recommender systems (RS) discipline ventures into adulthood. This workshop serves as a platform for introspection, examining the evolution of RS from its origins in CHI to its current state heavily influenced by and focusing on machine learning. The INTROSPECTIVES workshop aims to foster discussions on the past, present, and future of the RS discipline, inviting the community to reflect on key questions such as the maturation of RS, shifts in research focus, and the impact and success of RS in practice. Topics include the changing landscape of RS problems, the evolving role of RS in addressing choice overload to the current motivations driving RS adoption.

As global warming soars, the need to assess the environmental impact of research is becoming increasingly urgent. Despite this, few recommender systems research papers address their environmental impact. In this study, we estimate the environmental impact of recommender systems research by reproducing typical experimental pipelines. Our analysis spans 79 full papers from the 2013 and 2023 ACM RecSys conferences, comparing traditional “good old-fashioned AI’’ algorithms with modern deep learning algorithms. We designed and reproduced representative experimental pipelines for both years, measuring energy consumption with a hardware energy meter and converting it to CO2 equivalents. Our results show that papers using deep learning algorithms emit approximately 42 times more CO2 equivalents than papers using traditional methods. On average, a single deep learning-based paper generates 3,297 kilograms of CO2 equivalents—more than the carbon emissions of one person flying from New York City to Melbourne or the amount of CO2 one tree sequesters over 300 years.

Evaluation plays a vital role in recommender systems—in research and practice—whether for confirming algorithmic concepts or assessing the operational validity of designs and applications. It may span the evaluation of early ideas and approaches up to elaborate implementations of systems integrated into everyday product settings; it may target a wide spectrum of different factors being evaluated. In this special issue, we explore recommender systems evaluation—theory and practice—while considering a diverse set of perspectives. These include recommender systems purposes, stakeholders, methodological approaches, and consequences. The collection of articles in this special issue offers insightful analyses of current recommender system evaluation practices, acknowledging their limitations, and setting out future research directions. As recommender systems evolve, the need for adequate evaluation methods and approaches increases. This special issue sheds light on areas undergoing development or requiring added attention from the research and practitioner communities in recommender systems. The compilation serves as a call to the recommender systems research community, motivating continued research and exploration of evaluation metrics, methods, and strategies.

Recommender systems research and practice are fast-developing topics with growing adoption in a wide variety of information access scenarios. In this article, we present an overview of research specifically focused on the evaluation of recommender systems. We perform a systematic literature review, in which we analyze 57 papers spanning six years (2017–2022). Focusing on the processes surrounding evaluation, we dial in on the methods applied, the datasets utilized, and the metrics used. Our study shows that the predominant experiment type in research on the evaluation of recommender systems is offline experimentation and that online evaluations are primarily used in combination with other experimentation methods, e.g., an offline experiment. Furthermore, we find that only a few datasets (MovieLens, Amazon review dataset) are widely used, while many datasets are used in only a few papers each. We observe a similar scenario when analyzing the employed performance metrics—a few metrics are widely used (precision, normalized Discounted Cumulative Gain, and Recall), while many others are used in only a few papers. Overall, our review indicates that beyond-accuracy qualities are rarely assessed. Our analysis shows that the research community working on evaluation has focused on the development of evaluation in a rather narrow scope, with the majority of experiments focusing on a few metrics, datasets, and methods.

Evaluation is important when developing and deploying recommender systems. The PERSPECTIVES workshop sheds light on the different, potentially diverging or contradictory perspectives on the evaluation of recommender systems. Building on the discussions and outcomes of the PERSPECTIVES workshops held at RecSys 2021 and 2022, the third edition of the PERSPECTIVES workshop held at RecSys 2023 brought together researchers and practitioners from academia and industry to reflect on the evaluation of recommender systems critically. The workshop featured a keynote and focused on the interactive part with discussions in small groups and the plenum. We discussed problems and lessons learned, encouraged the exchange of the many perspectives on evaluation, and aimed to move the discourse forward within the community.

Evaluation of recommender systems is a central activity when developing recommender systems, both in industry and academia. The second edition of the PERSPECTIVES workshop held at RecSys 2022 brought together academia and industry to critically reflect on the evaluation of recommender systems. In the 2022 edition of PERSPECTIVES, we discussed problems and lessons learned, encouraged the exchange of the various perspectives on evaluation, and aimed to move the discourse forward within the community. We deliberately solicited papers reporting a reflection on problems regarding recommender systems evaluation and lessons learned. The workshop featured interactive parts with discussions in small groups as well as in the plenum, both on-site and online, and an industry keynote.

Evaluation is a cornerstone in the process of developing and deploying recommender systems. The PERSPECTIVES workshop brought together academia and industry to critically reflect on the evaluation of recommender systems. Particularly, the workshop aimed to shed light on the different, and maybe even diverging or contradictory perspectives on the evaluation of recommender systems. Papers reporting a reflection on problems regarding recommender systems evaluation and lessons learned were solicited. The workshop combined flash presentations of accepted papers, a keynote from industry, and an interactive part with discussions in break-out rooms as well as in the plenum. The workshop complemented the program of the main conference as it emphasized problems and lessons learned, fostered exchange integrating various perspectives on evaluation, and sought to move the recommender systems community forward as an outcome of the workshop.

HealthRecSys 2020 was the 5th International Workshop on Health Recommender Systems held in conjunction with the 14th ACM Conference on Recommender Systems. This workshop followed the previous workshop in 2019 [4] and focused on the application and potentials of recommender systems on health promotion, health care, and health-related topics. By engaging in the discussion and representation of health domains into recommender systems, this workshop facilitated the cross-domain collaborations and exchange of knowledge and infrastructure. This year, in particular, COVID-19-related contributions were discussed.

HealthRecSys 2019 was the 4th International Workshop on Health Recommender Systems held in conjunction with the 2019 ACM Conference on Recommender Systems in Copenhagen, Denmark. This workshop followed on from of the previous workshop in 2018 [4] and focused on the application and potentials of recommender systems on health promotion, health care and health-related topics. By engaging the discussion and representation of health domains into recommender systems, this workshop facilitated the cross-domain collaborations and exchange of knowledge and infrastructure.

The first workshop on Interactive Data Mining is held in Melbourne, Australia, on February 15, 2019 and is co-located with 12th ACM International Conference on Web Search and Data Mining (WSDM 2019). The goal of this workshop is to share and discuss research and projects that focus on interaction with and interactivity of data mining systems. The program includes invited speaker, presentation of research papers, and a discussion session.

Over the past decade, recommendation algorithms for ratings prediction and item ranking have steadily matured. However, these state-of-the-art algorithms are typically applied in relatively straightforward scenarios. In reality, recommendation is often a more complex problem: it is usually just a single step in the user's more complex background need. These background needs can often place a variety of constraints on which recommendations are interesting to the user and when they are appropriate. However, relatively little research has been done on these complex recommendation scenarios. The ComplexRec 2018 workshop addresses this by providing an interactive venue for discussing approaches to recommendation in complex scenarios that have no simple one-size-fits-all solution.

The 3rd International Workshop on Health Recommender Systems was held in conjunction with the 2018 ACM Conference on Recommender Systems in Vancouver, Canada. Following the two prior workshops in 2016 [4] and 2017 [2], the focus of this workshop is to deepen the discussion on health promotion, health care as well as health related methods. This workshop also aims to strengthen the HealthRecSys community, to engage representatives of other health domains into cross-domain collaborations, and to exchange and share infrastructure.

Recommendation algorithms for ratings prediction and item ranking have steadily matured during the past decade. However, these state-of-the-art algorithms are typically applied in relatively straightforward scenarios. In reality, recommendation is often a more complex problem: it is usually just a single step in the user's more complex background need. These background needs can often place a variety of constraints on which recommendations are interesting to the user and when they are appropriate. However, relatively little research has been done on these complex recommendation scenarios. The ComplexRec 2017 workshop addressed this by providing an interactive venue for discussing approaches to recommendation in complex scenarios that have no simple one-size-fits-all-solution.

The 2017 Workshop on Health Recommender Systems was held in conjunction with the 2017 ACM Conference on Recommender Systems in Como, Italy. Following the fists workshop in 2016, the focus of this workshop was on enhancing the results of the first workshop by elaborating discussions on the topics, attracting scientist from other domains, finding cross-domain collaboration, and establishing shared infrastructures.

The first Workshop on Engendering Health with Recommender Systems was organized in conjunction with ACM RecSys 2016. The focus of the workshop was on bringing together researchers and practitioners from diverse areas of health, well-being, decision support, and behavioral change. Health-related issues in recommender systems have been a growing research topic in the recent years and this was a initial attempt at bringing together academics and practitioners to share their experiences on working on related issues.

other Share on Introduction to the Special Issue on Recommender System Benchmarking Authors: Paolo Cremonesi Politecnico di Milano Politecnico di MilanoView Profile , Alan Said Recorded Future Recorded FutureView Profile , Domonkos Tikk Gravity R&D, Hungary Gravity R&D, HungaryView Profile , Michelle X. Zhou Juji JujiView Profile Authors Info & Claims ACM Transactions on Intelligent Systems and TechnologyVolume 7Issue 3April 2016 Article No.: 38pp 1–4https://doi.org/10.1145/2870627Published:08 March 2016Publication History 2citation384DownloadsMetricsTotal Citations2Total Downloads384Last 12 Months12Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

WrapRec is an easy-to-use Recommender Systems toolkit, which allows users to easily implement or wrap recommendation algorithms from other frameworks. The main goals of WrapRec are to provide a flexible I/O, evaluation mechanism and code reusability. WrapRec provides a rich data model which makes it easy to implement algorithms for different recommender system problems, such as context-aware and cross-domain recommendation. The toolkit is written in C# and the source code is publicly available on Github under the GPL license.

The advantage of Factorization Machines over other factorization models is their ability to easily integrate and efficiently exploit auxiliary information to improve Collaborative Filtering. Until now, this auxiliary information has been drawn from external knowledge sources beyond the user-item matrix. In this paper, we demonstrate that Factorization Machines can exploit additional representations of information inherent in the user-item matrix to improve recommendation performance. We refer to our approach as 'Free Lunch' enhancement since it leverages clusters that are based on information that is present in the user-item matrix, but not otherwise directly exploited during matrix factorization. Borrowing clustering concepts from codebook sharing, our approach can also make use of 'Free Lunch' information inherent in a user-item matrix from a auxiliary domain that is different from the target domain of the recommender. Our approach improves performance both in the joint case, in which the auxiliary and target domains share users, and in the disjoint case, in which they do not. Although 'Free Lunch' enhancement does not apply equally well to any given domain or domain combination, our overall conclusion is that Factorization Machines present an opportunity to exploit information that is ubiquitously present, but commonly under-appreciated by Collaborative Filtering algorithms.

Recommender systems research is often based on comparisons of predictive accuracy: the better the evaluation scores, the better the recommender. However, it is difficult to compare results from different recommender systems due to the many options in design and implementation of an evaluation strategy. Additionally, algorithmic implementations can diverge from the standard formulation due to manual tuning and modifications that work better in some situations.

Currently, it is difficult to put in context and compare the results from a given evaluation of a recommender system, mainly because too many alternatives exist when designing and implementing an evaluation strategy. Furthermore, the actual implementation of a recommendation algorithm sometimes diverges considerably from the well-known ideal formulation due to manual tuning and modifications observed to work better in some situations. RiVal - a recommender system evaluation toolkit - allows for complete control of the different evaluation dimensions that take place in any experimental evaluation of a recommender system: data splitting, definition of evaluation strategies, and computation of evaluation metrics. In this demo we present some of the functionality of RiVal and show step-by-step how RiVal can be used to evaluate the results from any recommendation framework and make sure that the results are comparable and reproducible.

The 2014 ACM Recommender Systems Challenge invited researchers and practitioners to work towards a common goal, this goal being the prediction of users engagement in movie ratings expressed on Twitter. More than 200 participants sought to join the challenge and work on the new dataset released in its scope. The participants were asked to develop new algorithms to predict user engagement and evaluate them in a common setting, ensuring that the comparison was objective and unbiased, within the challenge.

Experiment replication and reproduction are key requirements for empirical research methodology, and an important open issue in the field of Recommender Systems. When an experiment is repeated by a different researcher and exactly the same result is obtained, we can say the experiment has been replicated. When the results are not exactly the same but the conclusions are compatible with the prior ones, we have a reproduction of the experiment. Reproducibility and replication involve recommendation algorithm implementations, experimental protocols, and evaluation metrics. While the problem of reproducibility and replication has been recognized in the Recommender Systems community, the need for a clear solution remains largely unmet, which motivates the present workshop.

Evaluating adaptive and personalized information retrieval tech-niques is known to be a difficult endeavor. The rapid evolution of novel technologies in this scope raises additional challenges that further stress the need for new evaluation approaches and method-ologies. The BARS 2013 workshop seeks to provide a specific venue for work on novel, personalization-centric benchmarking approaches to evaluate adaptive retrieval and recommender systems.

Context-aware information is widely available in various ways and is becoming more and more important for enhancing retrieval performance and recommendation results. The current main issue to cope with is not only recommending or retrieving the most relevant items and content, but defining them ad hoc. Other relevant issues include personalizing and adapting the information and the way it is displayed to the user's current situation and interests. Ubiquitous computing further provides new means for capturing user feedback on items and providing information.

The challenge and workshop on Context-Aware Movie Recommendation (CAMRa2010) were conducted jointly in 2010 with the Recommender Systems conference. The challenge focused on three context-aware recommendation scenarios: time-based, mood-based, and social recommendation. The participants were provided with anonymized datasets from two real-world online movie recommendation communities and competed against each other for obtaining the highest accuracy of recommendations. The datasets contained contextual features, such as tags, annotation, social relationsips, and comments, normally not available in public recommendation datasets. More than 40 teams from 21 countries participated in the challenge. Their participation was summarized by 10 papers published by the workshop, which have been extended and revised for this special section. In this preface we overview the challenge datasets, tasks, evaluation metrics, and the obtained outcomes.

The Recommender System Challenge 2012 invited participants to work on two tracks with real-world datasets and to submit their contributions that would be related to specific problem contexts. First of all, it asked participants to develop new algorithms and to compare them to other algorithms in given settings; in addition, it asked participants to explore with new recommendation methods, services, as well as added-value services related to recommendation.

Recommender System Challenges such as the Netflix Prize, KDD Cup, etc. have contributed vastly to the development and adoptability of recommender systems. Each year a number of challenges or contests are organized covering different aspects of recommendation. In this tutorial and panel, we present some of the factors involved in successfully organizing a challenge, whether for reasons purely related to research, industrial challenges, or to widen the scope of recommender systems applications.

Recommender systems are commonly evaluated by trying to predict known, withheld, ratings for a set of users. Measures such as the Root-Mean-Square Error are used to estimate the quality of the recommender algorithms. This process does however not acknowledge the inherent rating inconsistencies of users. In this paper we present the first results from a noise measurement user study for estimating the magic barrier of recommender systems conducted on a commercial movie recommendation community. The magic barrier is the expected squared error of the optimal recommendation algorithm, or, the lowest error we can expect from any recommendation algorithm. Our results show that the barrier can be estimated by collecting the opinions of users on already rated items.

This paper provides an overview of CAMRa2011, the second edition of the Challenge on Context-Aware Movie Recommendation. The challenge attracted a large number of participants to work on the challenge tracks, which this time focused on group related recommendation aspects.

CARS and CAMRa were organized under the Context-awareness in Recommendation Systems special event and gathered academic researchers as well as industrial practitioners in a workshop and challenge.

Context-aware recommender systems are becoming a popular topic, still, there are many untouched aspects. In this paper, research involving context identification and the concepts related to hybrid and context-aware systems is presented. A conceptual architecture for a context-aware recommender system for movies and TV shows is furthermore introduced. The system consists of a number of processes for context identification and recommendation. Key contextual features are identified and used for the creation of several sets of recommendations, based on the predicted context. The main focus of the research presented here is the identification of context, which in turn is used for recommendation. The results will be evaluated and incorporated into the recommendation engine of movie and TV recommendation website Moviepilot.