In streaming services, recommendations are vital for guiding users to content that suits their preferences. The homepage plays a key role in helping users quickly find something they’ll enjoy, but the challenge lies in curating a vast catalog within limited screen space while catering to diverse individual interests. Typically, the homepage is organized into thematic rows that users can scroll through horizontally or vertically. The challenge of optimizing this layout to maximize user serendipity involves determining how to select the most relevant rows for each user, populate those rows with appropriate videos, and arrange them within the constrained page space to ensure intuitive video selection. To address this challenge, this paper introduces a scalable framework designed to generate personalized thematic rails and rank the generated rails vertically per users’ taste. Central to this framework is the utilization of a Graph Neural Network (GNN), which learns item representations from rich item graphs infused with metadata. By harnessing users’ historical interactions alongside these learned item representations, the framework constructs nuanced user profiles, capturing their evolving preferences and behaviors. The primary objective of this framework is to enhance the Normalized 2-Dimensional Discounted Cumulative Gain (N2DCG) metric, a key measure of user engagement with recommended content. This is achieved by iteratively refining the vertical ranking of the generated rails per user. Rigorous offline evaluations and consequent online experiments prove the effectiveness of our framework. Our findings not only affirm the potency of personalized thematic rails in driving user engagement but also reinforce the potential of leveraging advanced techniques such as Graph Neural Networks in enhancing recommendation systems within the streaming landscape.

In the realm of recommender systems research, offline evaluation metrics like NDCG [4], Recall [1], or Precision [1] are often used to measure the impact.On the other hand, common industry practices suggest evaluating new ideas/models through A/B tests where decisions are made based on business metrics like the overall engagement of users.A new model may show improvement in offline metrics but performance loss in online metrics.One reason that leads to this phenomenon is the counterfactual nature of the recommendation problem which can be addressed by off-policy evaluation methods [6][3].Another reason is the degree of causal connection between offline evaluation metrics and observed online metrics.In this work, we will share our learnings from two set of A/B tests that we conducted at Max 1 where we observed a mismatch between online and offline metrics due to a weak causal connection between online and offline metrics.Thanks to learnings from A/B tests, we discovered and quantified the impact of series to movie ratio at recommendations.Our experiments show that there is an optimal amount of series to movies ratio that provides the best possible results for user engagement.Production Model: Personalization model at Max powers the horizontal and vertical ranking of items on the homepage.

In many applications that rely on machine learning, the availability of labelled data is a matter of primary importance. However, when tackling new tasks, labels are usually missing and must be collected from scratch by the users. In this work, we address the problem of learning classifiers when the amount of labels is very scarce. We do so by learning multiple vectors, called prototypes, that represent relevant semantic concepts for the task at hand. We propose a theoretically inspired mechanism that computes probabilities of matching between the prototypes and the input elements, and we combine these probabilities to increase the expressiveness of the classifier. Moreover, by leveraging low-cost extra annotations in the training data, a simple error-boosting technique guides the learning process and provides substantial performance improvements. Empirical results confirm the benefits of the proposed approach in both balanced and unbalanced datasets. Our methodology is thus of practical use when gathering and labelling new examples is more expensive than annotating what we already have.

Predicting the click-through rate of an advertisement is a critical component of online advertising platforms. In sponsored search, the click-through rate estimates the probability that a displayed advertisement is clicked by a user after she submits a query to the search engine. Commercial search engines typically rely on machine learning models trained with a large number of features to make such predictions. This inevitably requires a lot of engineering efforts to define, compute, and select the appropriate features. In this paper, we propose two novel approaches (one working at character level and the other working at word level) that use deep convolutional neural networks to predict the click-through rate of a query-advertisement pair. Specifically, the proposed architectures consider as input only the textual content appearing in a query-advertisement pair and the page position at which the advertisement appears on the search result page of the query, and produce as output a click-through rate prediction. By comparing the character-level model with the word-level model, we show that language representation can be learnt from scratch at character level when trained on enough data. Through extensive experiments using billions of query-advertisement pairs of a popular commercial search engine, we demonstrate that both approaches significantly outperform a baseline model built on well-selected text features and a state-of-the-art word2vec-based approach. We also show the importance of the position feature in the proposed approaches in improving the prediction accuracy. When combining the predictions of the deep models introduced in this study with the prediction of the model in production of the same commercial search engine, we significantly improve the accuracy and the calibration of the click-through rate prediction of the production system. We also show the potential of leveraging the CTR prediction of the proposed deep learning models for query-ad relevance modeling and query-ad matching tasks in sponsored search.

Predicting the click-through rate of an advertisement is a critical component of online advertising platforms. In sponsored search, the click-through rate estimates the probability that a displayed advertisement is clicked by a user after she submits a query to the search engine. Commercial search engines typically rely on machine learning models trained with a large number of features to make such predictions. This inevitably requires a lot of engineering efforts to define, compute, and select the appropriate features. In this paper, we propose two novel approaches (one working at character level and the other working at word level) that use deep convolutional neural networks to predict the click-through rate of a query-advertisement pair. Specifically, the proposed architectures only consider the textual content appearing in a query-advertisement pair as input, and produce as output a click-through rate prediction. By comparing the character-level model with the word-level model, we show that language representation can be learnt from scratch at character level when trained on enough data. Through extensive experiments using billions of query-advertisement pairs of a popular commercial search engine, we demonstrate that both approaches significantly outperform a baseline model built on well-selected text features and a state-of-the-art word2vec-based approach. Finally, by combining the predictions of the deep models introduced in this study with the prediction of the model in production of the same commercial search engine, we significantly improve the accuracy and the calibration of the click-through rate prediction of the production system.