In the task of modeling user preferences for movie recommender systems, recent research has demonstrated the benefits of describing movies with their eudaimonic and hedonic scores (E and H scores), which reflect the depth of their message and the level of fun experience they provide, respectively. So far, the labeling of movies with their E and H scores has been done manually using a dedicated instrument (a questionnaire), which is time-consuming. To address this issue, we propose an automatic approach for predicting E and H scores. Specifically, we collected E and H scores of 709 movies from 370 users (with a total of 3699 records), augmented this dataset with metadata, audio, and low-level and high-level visual features, and trained machine learning models for predicting the E and H scores of movies. This study investigates the use of machine learning models in predicting the E and H scores of movies using various feature sets, including audio, low-level and high-level visual features, and metadata. We compared the performance of predictive models using different combinations of features with the majority classifier as the baseline approach. The results demonstrate that our proposed machine learning-based models significantly outperform the baseline in predicting E and H scores, particularly when leveraging metadata features. Specifically, the random forest classifier achieved a 20% increase in ROC AUC compared to the baseline when predicting both the E score and the H score. These improvements were found to be statistically significant. Overall, our findings suggest that automated tools for predicting E and H scores in movies are promising alternatives to traditional questionnaire-based approaches.

Despite their proven various benefits, Recommender Systems can cause or amplify certain undesired effects. In this paper, we focus on Popularity Bias, i.e., the tendency of a recommender system to utilize the effect of recommending popular items to the user. Prior research has studied the negative impact of this type of bias on individuals and society as a whole and proposed various approaches to mitigate this in various domains. However, almost all works adopted offline methodologies to evaluate the effectiveness of the proposed approaches. Unfortunately, such offline simulations can potentially be rather simplified and unable to capture the full picture. To contribute to this line of research and given a particular lack of knowledge about how debiasing approaches work not only offline, but online as well, we present in this paper the results of user study on a national broadcaster movie streaming platform in Norway, i.e., TV 2, following the A/B testing methodology. We deployed an effective mitigation approach for popularity bias, called Calibrated Popularity (CP), and monitored its performance in comparison to the platform’s existing collaborative filtering recommendation approach as a baseline over a period of almost four months. The results obtained from a large user base interacting in real-time with the recommendations indicate that the evaluated debiasing approach can be effective in addressing popularity bias while still maintaining the level of user interest and engagement.

Hybrid recommender systems utilize advanced algorithms capable of learning heterogeneous sources of data and generating personalized recommendations for users. The data can range from user preferences (e.g., ratings or reviews) to item content (e.g., description or category). Prior studies in the field of recommender systems have primarily relied on ”ratings” as the user feedback, when building user profiles or evaluating the quality of the recommendation. While ratings are informative, they may still fail to represent a comprehensive picture of actual user preferences. In contrast, there are other types of feedback data that differently or complementarily represent users and their preferences, including the reviews and the sentiments encapsulated within them. Such data can reveal important parts of a user’s profile that are not necessarily correlated with user ratings, and hence, they potentially reflect a different side of the user’s profile. In this paper, we propose a novel form of hybrid recommender system, capable of analyzing the reviews and extracting their sentiments that are incorporated into the recommendation process. We used advanced algorithms to generate recommendations for users capable of incorporating additional data, such as the review sentiment. We conducted analyses and showed that sentiments of user reviews are not always highly correlated with the ratings (e.g., in music domain). This might mean that sentiment can be indicative of a different aspect of user preferences and can be used as an alternative signal of user feedback. Hence, we have used both ratings and sentiments of reviews when evaluating our proposed hybrid recommender system. We selected two common datasets for the evaluation, Amazon Digital Music and Amazon Video Games, and showed the superior performance of the proposed hybrid recommender system compared to different baselines. The comparison were made in two evaluation scenarios, namely, when the ratings were considered the user feedback and when sentiments of the review were considered the user feedback.

Recommender Systems are becoming an inherent part of today’s Internet. They can be found anywhere from e-commerce platforms (eBay, Amazon) to music or movie streaming (Spotify, Netflix), social media (Facebook, Instagram, TikTok), travel platforms (Booking.com, Expedia), and much more. Whether a recommendation is successful or not can rely on multiple objectives such as user satisfaction, business value, and societal issues. In addition, the long-term happiness (along with short-term excitements and delight) of the users is critical for a recommender system to be considered successful. MORS workshop brings together researchers and practitioners to discuss the importance of these aspects of recommender systems and find ways to develop algorithms to build multi-objective recommenders and also evaluation metrics to assess their success.

Historically, the main criterion for a successful recommender system was the relevance of the recommended items to the user. In other words, the only objective for the recommendation algorithm was to learn user’s preferences for different items and generate recommendations accordingly. However, real-world recommender systems are well beyond a simple objective and often need to take into account multiple objectives simultaneously. These objectives can be either from the users’ perspective or they could come from other stakeholders such as item providers or any party that could be impacted by the recommendations. Such multi-objective and multi-stakeholder recommenders present unique challenges and these challenges were the focus of the MORS workshop.

The ACM Recommender Systems Challenge 20171 focused on the problem of job recommendations: given a new job advertisement, the goal was to identify those users who are both (a) interested in getting notified about the job advertisement, and (b) appropriate candidates for the given job. Participating teams had to balance between user interests and requirements for the given job as well as dealing with the cold-start situation. For the first time in the history of the conference, the RecSys challenge offered an online evaluation: teams first had to compete as part of a traditional offline evaluation and the top 25 teams were then invited to evaluate their algorithms in an online setting, where they could submit recommendations to real users. Overall, 262 teams registered for the challenge, 103 teams actively participated and submitted together more than 6100 solutions as part of the offline evaluation. Finally, 18 teams participated and rolled out recommendations to more than 900,000 users on XING2.

In the last years, there has been much attention given to the semantic gap problem in multimedia retrieval systems. Much effort has been devoted to bridge this gap by building tools for the extraction of high-level, semantics-based features from multimedia content, as low-level features are not considered useful because they deal primarily with representing the perceived content rather than the semantics of it.

We present a research tool for user preference elicitation that collects both explicit user feedback and unobtrusively acquired facial expressions. The concrete implementation is a web-based user interface where the user is presented with two music excerpts. After listening to both, the user provides a pairwise score (i.e. which of the two items is preferred) for each pair of music excerpts. The novelty of the demo is the integration of the unobtrusive acquisition of facial expressions through the webcam. During the listening of the music excerpts, the system extracts features related to the facial expressions of the user several times per second. The interaction runs as a web application, which allows for a large-scale remote acquisition of emotional data. Up to now, such acquisitions were usually done in controlled environments with few subjects, hence being of little use for the recommender systems community.

The accuracy of collaborative-filtering recommender systems largely depends on three factors: the quality of the rating prediction algorithm, and the quantity and quality of available ratings. While research in the field of recommender systems often concentrates on improving prediction algorithms, even the best algorithms will fail if they are fed poor-quality data during training, that is, garbage in, garbage out. Active learning aims to remedy this problem by focusing on obtaining better-quality data that more aptly reflects a user's preferences. However, traditional evaluation of active learning strategies has two major flaws, which have significant negative ramifications on accurately evaluating the system's performance (prediction error, precision, and quantity of elicited ratings). (1) Performance has been evaluated for each user independently (ignoring system-wide improvements). (2) Active learning strategies have been evaluated in isolation from unsolicited user ratings (natural acquisition). In this article we show that an elicited rating has effects across the system, so a typical user-centric evaluation which ignores any changes of rating prediction of other users also ignores these cumulative effects, which may be more influential on the performance of the system as a whole (system centric). We propose a new evaluation methodology and use it to evaluate some novel and state-of-the-art rating elicitation strategies. We found that the system-wide effectiveness of a rating elicitation strategy depends on the stage of the rating elicitation process, and on the evaluation measures (MAE, NDCG, and Precision). In particular, we show that using some common user-centric strategies may actually degrade the overall performance of a system. Finally, we show that the performance of many common active learning strategies changes significantly when evaluated concurrently with the natural acquisition of ratings in recommender systems.