Recommender Systems are becoming an inherent part of today’s Internet. They can be found anywhere from e-commerce platforms (eBay, Amazon) to music or movie streaming (Spotify, Netflix), social media (Facebook, Instagram, TikTok), travel platforms (Booking.com, Expedia), and much more. Whether a recommendation is successful or not can rely on multiple objectives such as user satisfaction, business value, and societal issues. In addition, the long-term happiness (along with short-term excitements and delight) of the users is critical for a recommender system to be considered successful. MORS workshop brings together researchers and practitioners to discuss the importance of these aspects of recommender systems and find ways to develop algorithms to build multi-objective recommenders and also evaluation metrics to assess their success.

Historically, the main criterion for a successful recommender system was the relevance of the recommended items to the user. In other words, the only objective for the recommendation algorithm was to learn user’s preferences for different items and generate recommendations accordingly. However, real-world recommender systems are well beyond a simple objective and often need to take into account multiple objectives simultaneously. These objectives can be either from the users’ perspective or they could come from other stakeholders such as item providers or any party that could be impacted by the recommendations. Such multi-objective and multi-stakeholder recommenders present unique challenges and these challenges were the focus of the MORS workshop.

User interactions can be considered to constitute different feedback channels, for example, view, click, like or follow, that provide implicit information on users’ preferences. Each implicit feedback channel typically carries a unary, positive-only signal that can be exploited by collaborative filtering models to generate lists of personalized recommendations. This article investigates how a learning-to-rank recommender system can best take advantage of implicit feedback signals from multiple channels. We focus on Factorization Machines (FMs) with Bayesian Personalized Ranking (BPR), a pairwise learning-to-rank method, that allows us to experiment with different forms of exploitation. We perform extensive experiments on three datasets with multiple types of feedback to arrive at a series of insights. We compare conventional, direct integration of feedback types with our proposed method, which exploits multiple feedback channels during the sampling process of training. We refer to our method as multi-channel sampling. Our results show that multi-channel sampling outperforms conventional integration, and that sampling with the relative “level” of feedback is always superior to a level-blind sampling approach. We evaluate our method experimentally on three datasets in different domains and observe that with our multi-channel sampler the accuracy of recommendations can be improved considerably compared to the state-of-the-art models. Further experiments reveal that the appropriate sampling method depends on particular properties of datasets such as popularity skewness.

Pairwise learning-to-rank algorithms have been shown to allow recommender systems to leverage unary user feedback. We propose Multi-feedback Bayesian Personalized Ranking (MF-BPR), a pairwise method that exploits different types of feedback with an extended sampling method. The feedback types are drawn from different "channels", in which users interact with items (e.g., clicks, likes, listens, follows, and purchases). We build on the insight that different kinds of feedback, e.g., a click versus a like, reflect different levels of commitment or preference. Our approach differs from previous work in that it exploits multiple sources of feedback simultaneously during the training process. The novelty of MF-BPR is an extended sampling method that equates feedback sources with "levels" that reflect the expected contribution of the signal. We demonstrate the effectiveness of our approach with a series of experiments carried out on three datasets containing multiple types of feedback. Our experimental results demonstrate that with a right sampling method, MF-BPR outperforms BPR in terms of accuracy. We find that the advantage of MF-BPR lies in its ability to leverage level information when sampling negative items.

In this paper, we investigate the usefulness of deep links for improving video search results. Deep links are time-coded comments with which viewers express their reactions to the content at specific time-points of a video that they find noteworthy. The rationale underlying our work is that deep links can open up an interesting new perspective on the relevance of a video, namely focusing on individual video segments, in addition to the existing ones that typically concern a video as a whole. In this perspective, deep-link comments provide non-linear access to videos via their time-codes, which can match alternate dimensions of user needs that extend beyond topical and affective relevance. We explore the different types of deep-link comments and develop a viewer expressive reaction variety (VERV) typology that captures how viewers deep-link on YouTube. We validate this typology through a user study on Amazon Mechanical Turk to show that it is a typology human annotators can agree upon. We then demonstrate, through experiments, that deep-link comments can automatically be classified into VERV categories and show the potential of our proposed usage of deep-link comments for video search through a user study.

In this paper we propose a new dataset, Div400, that was designed to support shared evaluation in different areas of social media photo retrieval, e.g., machine analysis (re-ranking, machine learning), human-based computation (crowdsourcing) or hybrid approaches (relevance feedback, machine-crowd integration). Div400 comes with associated relevance and diversity assessments performed by human annotators. 396 landmark locations are represented via 43,418 Flickr photos and metadata, Wikipedia pages and content descriptors for text and visual modalities. To facilitate distribution, only Creative Commons content was included in the dataset. The proposed dataset was validated during the 2013 Retrieving Diverse Social Images Task at the MediaEval Benchmarking Initiative for Multimedia Evaluation.

In this work, we present a new social image dataset related to the fashion and clothing domain. The dataset contains more than 32000 images, their context and social metadata. Furthermore the dataset is enriched with several types of annotations collected from the Amazon Mechanical Turk (AMT) crowdsourcing platform, which can serve as ground truth for various content analysis algorithms. This dataset has been successfully used at the Crowdsourcing task of the 2013 MediaEval Multimedia Benchmarking initiative. The dataset contributes to several research areas such as Crowdsourcing, multimedia content and context analysis as well as hybrid human/automatic approaches. In this paper, the dataset is described in detail and the dataset collection strategy, statistics, applications of dataset and its contribution to MediaEval 2013 is discussed.

Incorporating social trust in Matrix Factorization (MF) methods demonstrably improves accuracy of rating prediction. Such approaches mainly use the trust scores explicitly expressed by users. However, it is often challenging to have users provide explicit trust scores of each other. There exist quite a few works, which propose Trust Metrics (TM) to compute and predict trust scores between users based on their interactions. In this paper, we first evaluate several TMs to find out which one can best predict trust scores compared to the actual trust scores explicitly expressed by users. And, second, we propose to incorporate these trust scores inferred from the candidate TMs into social matrix factorization (MF). We investigate if incorporating the implicit trust scores in MF can make rating prediction as accurate as the MF on explicit trust scores. The reported results support the idea of employing implicit trust into MF whenever explicit trust is not available, since the performance of both models is similar.

WrapRec is an easy-to-use Recommender Systems toolkit, which allows users to easily implement or wrap recommendation algorithms from other frameworks. The main goals of WrapRec are to provide a flexible I/O, evaluation mechanism and code reusability. WrapRec provides a rich data model which makes it easy to implement algorithms for different recommender system problems, such as context-aware and cross-domain recommendation. The toolkit is written in C# and the source code is publicly available on Github under the GPL license.

The advantage of Factorization Machines over other factorization models is their ability to easily integrate and efficiently exploit auxiliary information to improve Collaborative Filtering. Until now, this auxiliary information has been drawn from external knowledge sources beyond the user-item matrix. In this paper, we demonstrate that Factorization Machines can exploit additional representations of information inherent in the user-item matrix to improve recommendation performance. We refer to our approach as 'Free Lunch' enhancement since it leverages clusters that are based on information that is present in the user-item matrix, but not otherwise directly exploited during matrix factorization. Borrowing clustering concepts from codebook sharing, our approach can also make use of 'Free Lunch' information inherent in a user-item matrix from a auxiliary domain that is different from the target domain of the recommender. Our approach improves performance both in the joint case, in which the auxiliary and target domains share users, and in the disjoint case, in which they do not. Although 'Free Lunch' enhancement does not apply equally well to any given domain or domain combination, our overall conclusion is that Factorization Machines present an opportunity to exploit information that is ubiquitously present, but commonly under-appreciated by Collaborative Filtering algorithms.

The 2014 ACM Recommender Systems Challenge invited researchers and practitioners to work towards a common goal, this goal being the prediction of users engagement in movie ratings expressed on Twitter. More than 200 participants sought to join the challenge and work on the new dataset released in its scope. The participants were asked to develop new algorithms to predict user engagement and evaluate them in a common setting, ensuring that the comparison was objective and unbiased, within the challenge.

This paper studies a new way of accessing videos in a non-linear fashion. Existing non-linear access methods allow users to jump into videos at points that depict specific visual concepts or that are likely to elicit affective reactions. We believe that deep-link comments, which occur unprompted on social video sharing platforms, offer a new opportunity beyond existing methods. With deep-link comments, viewers express themselves about a particular moment in a video by including a time-code. Deep-link comments are special because they reflect viewer perceptions of noteworthiness, that include, but extend beyond depicted conceptual content and induced affective reactions. Based on deep-link comments collected from YouTube, we develop a Viewer Expressive Reaction Variety (VERV) taxonomy that captures how viewers deep-link. We validate the taxonomy with a user study on a crowdsourcing platform and discuss how it extends conventional relevance criteria. We carry out experiments which show that deep-link comments can be automatically filtered and sorted into VERV categories.

In this work, we present a fashion-focused Creative Commons dataset, which is designed to contain a mix of general images as well as a large component of images that are focused on fashion (i.e., relevant to particular clothing items or fashion accessories). The dataset contains 4810 images and related metadata. Furthermore, a ground truth on image's tags is presented. Ground truth generation for large-scale datasets is a necessary but expensive task. Traditional expert based approaches have become an expensive and non-scalable solution. For this reason, we turn to crowdsourcing techniques in order to collect ground truth labels; in particular we make use of the commercial crowdsourcing platform, Amazon Mechanical Turk (AMT). Two different groups of annotators (i.e., trusted annotators known to the authors and crowdsourcing workers on AMT) participated in the ground truth creation. Annotation agreement between the two groups is analyzed. Applications of the dataset in different contexts are discussed. This dataset contributes to research areas such as crowdsourcing for multimedia, multimedia content analysis, and design of systems that can elicit fashion preferences from users.

An important component of question answering systems is question classification. The task of question classification is to predict the entity type of the answer of a natural language question. Question classification is typically done using machine learning techniques. Most approaches use features based on word unigrams which leads to large feature space. In this work we applied Latent Semantic Analysis (LSA) technique to reduce the large feature space of questions to a much smaller and efficient feature space. We used two different classifiers: Back-Propagation Neural Networks (BPNN) and Support Vector Machines (SVM). We found that applying LSA on question classification can not only make the question classification more time efficient, but it also improves the classification accuracy by removing the redundant features. Furthermore, we discovered that when the original feature space is compact and efficient, its reduced space performs better than a large feature space with a rich set of features. In addition, we found that in the reduced feature space, BPNN performs better than SVMs which are widely used in question classification. Our result on the well known UIUC dataset is competitive with the state-of-the-art in this field, even though we used much smaller feature spaces.