Recommender systems are inherently decision-making systems, taking actions that have consequences for the world around them.Some consequences might be desirable (for example, growing the user base for an online platform), others might be unintended (for example, amplifying inequality among item providers).In order to reason about these consequences, we need to resort to methods from the literature on causal inference.Whilst this research area has seen a growing interest in recent years, there is an abundance of open research questions from how we should model large-scale recommender systems in such causal frameworks, to what the limitations are for causal identifiability in general settings, and how we can properly handle confounding variables.The CONSEQUENCES workshop series aims to bring together researchers and practitioners who are interested in this research topic, and wish to help shape its future.

Recommender systems are inherently decision-making systems, taking actions that have consequences for the world around them. Some consequences might be desirable (for example, growing the user base for an online platform), others might be unintended (for example, amplifying inequality among item providers). In order to reason about these consequences, we need to resort to methods from the literature on causal inference. Whilst this research area has seen a growing interest in recent years, there is an abundance of open research questions from how we should model large-scale recommender systems in such causal frameworks, to what the limitations are for causal identifiability in general settings, and how we can properly handle confounding variables. The CONSEQUENCES workshop series aims to bring together researchers and practitioners who are interested in this research topic, and wish to help shape its future.

Recommender systems make algorithmic decisions about what will be shown to whom, billions of times every day across the web. These decisions have consequences that can often be far-reaching. Indeed, users that are exposed to certain items, might be convinced to explore interests that are novel to them. At the same time, exposure is often linked to economic incentives for the item producer, which platform-level metrics will be impacted by as well. Feedback loops in existing systems also imply that algorithmic decisions made by the model itself, have an impact on the data future model iterations will be trained and evaluated on.

How can we build and optimize a recommender system that must rapidly fill slates (i.e. banners) of personalized recommendations? The combination of deep learning stacks with fast maximum inner product search (MIPS) algorithms have shown it is possible to deploy flexible models in production that can rapidly deliver personalized recommendations to users. Albeit promising, this methodology is unfortunately not sufficient to build a recommender system which maximizes the reward, e.g. the probability of click. Usually instead a proxy loss is optimized and A/B testing is used to test if the system actually improved performance. This tutorial takes participants through the necessary steps to model the reward and directly optimize the reward of recommendation engines built upon fast search algorithms to produce high-performance reward-optimizing recommender systems.

Recommender systems are more and more often modelled as repeated decision making processes – deciding which (ranking of) items to recommend to a given user. Each decision to recommend or rank an item has a significant impact on immediate and future user responses, long-term satisfaction or engagement with the system, and possibly valuable exposure for the item provider. This interactive and interventionist view of the recommender uncovers a plethora of unanswered research questions, as it complicates the typically adopted offline evaluation or learning procedures in the field. We need an understanding of causal inference to reason about (possibly unintended) consequences of the recommender, and a notion of counterfactuals to answer common “what if”-type questions in learning and evaluation. Advances at the intersection of these fields can foster progress in effective, efficient and fair learning and evaluation from logged data. These topics have been emerging in the Recommender Systems community for a while, but we firmly believe in the value of a dedicated forum and place to learn and exchange ideas. We welcome contributions from both academia and industry and bring together a growing community of researchers and practitioners interested in sequential decision making, offline evaluation, batch policy learning, fairness in online platforms, as well as other related tasks, such as A/B testing.

Conventional approaches to recommendation often do not explicitly take into account information on previously shown recommendations and their recorded responses. One reason is that, since we do not know the outcome of actions the system did not take, learning directly from such logs is not a straightforward task. Several methods for off-policy or counterfactual learning have been proposed in recent years, but their efficacy for the recommendation task remains understudied. Due to the limitations of offline datasets and the lack of access of most academic researchers to online experiments, this is a non-trivial task. Simulation environments can provide a reproducible solution to this problem.

A common task for recommender systems is to build a profile of the interests of a user from items in their browsing history and later to recommend items to the user from the same catalog. The users' behavior consists of two parts: the sequence of items that they viewed without intervention (the organic part) and the sequences of items recommended to them and their outcome (the bandit part).

The REVEAL workshop1 focuses on framing the recommendation problem as a one of making personalized interventions, e.g. deciding to recommend a particular item to a particular user. Moreover, these interventions sometimes depend on each other, where a stream of interactions occurs between the user and the system, and where each decision to recommend something will have an impact on future steps and long-term rewards. This framing creates a number of challenges we will discuss at the workshop. How can recommender systems be evaluated offline in such a context? How can we learn recommendation policies that are aware of these delayed consequences and outcomes?

We develop the value based approach to Recommender systems. The value approach is a model based approach that allows forecasting of actual A/B test performance. It contrasts with the proxy based approach, which attempt to order the performance of different recommendation systems, but not forecast actual performance. It also contrasts with policy based approaches which also produce a performance forecast but use propensity scores to by-pass the requirement for a model. Value based approaches are a state of the art approach for combining organic and bandit signals that can utilise the three fundamental distances of recommendation. Their deployment requires sophisticated modelling and Bayesian computation. This tutorial develops the theory of value based recommendation and demonstrates the approach with examples in python notebooks.

The REVEAL workshop1 focuses on framing the recommendation problem as a one of making personalized interventions. Moreover, these interventions sometimes depend on each other, where a stream of interactions occurs between the user and the system, and where each decision to recommend something will have an impact on future steps and long-term rewards. This framing creates a number of challenges we will discuss at the workshop. How can recommender systems be evaluated offline in such a context? How can we learn recommendation policies that are aware of these delayed consequences and outcomes?

In recent years, the softmax model and its fast approximations have become the de-facto loss functions for deep neural networks when dealing with multi-class prediction. This loss has been extended to language modeling and recommendation, two fields that fall into the framework of learning from Positive and Unlabeled data.

Many current applications use recommendations in order to modify the natural user behavior, such as to increase the number of sales or the time spent on a website. This results in a gap between the final recommendation objective and the classical setup where recommendation candidates are evaluated by their coherence with past user behavior, by predicting either the missing entries in the user-item matrix, or the most likely next event. To bridge this gap, we optimize a recommendation policy for the task of increasing the desired outcome versus the organic user behavior. We show this is equivalent to learning to predict recommendation outcomes under a fully random recommendation policy. To this end, we propose a new domain adaptation algorithm that learns from logged data containing outcomes from a biased recommendation policy and predicts recommendation outcomes according to random exposure. We compare our method against state-of-the-art factorization methods, in addition to new approaches of causal recommendation and show significant improvements.

Deep learning is now an integral part of recommender systems, but the research is still in its early phase. New research topics pop up frequently and established topics are extended in new, interesting directions. DLRS 2018 is a venue for pioneering work in the intersection of deep learning and recommender systems research.

The inaugural REVEAL workshop1 focuses on revisiting the offline evaluation problem for recommender systems. Being able to perform offline experiments is key to rapid innovation; however practitioners often observe significant differences between offline results and the outcome of an online experiment, where users are actually exposed to the resulting recommendations. This is unfortunate because online experiments take time, can be costly, and require access to a live recommender system, when offline experiments are inherently scalable. How can we bridge that gap between offline and online experiments?

We propose Meta-Prod2vec, a novel method to compute item similarities for recommendation that leverages existing item metadata. Such scenarios are frequently encountered in applications such as content recommendation, ad targeting and web search. Our method leverages past user interactions with items and their attributes to compute low-dimensional embeddings of items. Specifically, the item metadata is injected into the model as side information to regularize the item embeddings. We show that the new item representations lead to better performance on recommendation tasks on an open music dataset.