Item-to-item recommendations are crucial for user content discovery and engagement on online platforms, often showcased in prominent areas like "You May Also Like." These models typically leverage metadata and user engagement data to generate recommendations; however, data sparsity presents challenges, particularly when new movies or shows are released, limiting the ability to provide optimal recommendations early on. Additionally, as users access content across various devices with different screen sizes, it is essential to optimize the ranking of recommendations to ensure the most relevant items appear at the top. Finally, with platforms serving millions of users and an ever-changing inventory of items, scalable methodologies are necessary to effectively address these challenges. In this paper, we propose a scalable multi-stage item-to-item recommendations model called SLLIM-Rank: Similarity with Large Language Improved Model using Learning-to-Rank. The approach utilizes (a) temporal and contextual features to capture dynamic trends in item similarity, (b) a Learning-to-Rank model to prioritize items based on implicit user feedback and, (c) large language models (LLMs) to generate supplementary metadata for catalog items. We discuss effective strategies for offline evaluation of the model. Additionally, these offline findings lead to substantial improvements in key engagement metrics on a content streaming platform, specially improving the quality of cold item recommendations, demonstrating the high effectiveness of our approach in a real-world context.

In streaming services, recommendations are vital for guiding users to content that suits their preferences. The homepage plays a key role in helping users quickly find something they’ll enjoy, but the challenge lies in curating a vast catalog within limited screen space while catering to diverse individual interests. Typically, the homepage is organized into thematic rows that users can scroll through horizontally or vertically. The challenge of optimizing this layout to maximize user serendipity involves determining how to select the most relevant rows for each user, populate those rows with appropriate videos, and arrange them within the constrained page space to ensure intuitive video selection. To address this challenge, this paper introduces a scalable framework designed to generate personalized thematic rails and rank the generated rails vertically per users’ taste. Central to this framework is the utilization of a Graph Neural Network (GNN), which learns item representations from rich item graphs infused with metadata. By harnessing users’ historical interactions alongside these learned item representations, the framework constructs nuanced user profiles, capturing their evolving preferences and behaviors. The primary objective of this framework is to enhance the Normalized 2-Dimensional Discounted Cumulative Gain (N2DCG) metric, a key measure of user engagement with recommended content. This is achieved by iteratively refining the vertical ranking of the generated rails per user. Rigorous offline evaluations and consequent online experiments prove the effectiveness of our framework. Our findings not only affirm the potency of personalized thematic rails in driving user engagement but also reinforce the potential of leveraging advanced techniques such as Graph Neural Networks in enhancing recommendation systems within the streaming landscape.

Recommendation systems are used widely across many industries, such as e-commerce, multimedia content platforms, and social networks, to provide suggestions that users will most likely consume or connect, thus improving the user experience. This motivates people in industry and research organizations to focus on personalization and recommendation algorithms, resulting in many research papers. While academic research mostly focuses on the performance of recommendation algorithms in terms of ranking quality or accuracy, it often neglects key factors that impact how a recommendation system will perform in a real-world environment, including but not limited to business metric definition and evaluation, scalability, recommendation quality control, robustness, fairness, and resource limitations, such as computing and memory resources budgets, engineering workforce cost, etc. The gap in constraints and requirements between academic research and industry limits the broad applicability of many of academia's contributions to industrial recommendation systems. This workshop aspires to bridge this gap by bringing together researchers from both academia and industry. Its goal is to serve as a venue for industrial researchers to share practical insights and for academic researchers to become aware of the additional factors of algorithm adoption in real production systems.

In the realm of recommender systems research, offline evaluation metrics like NDCG [4], Recall [1], or Precision [1] are often used to measure the impact.On the other hand, common industry practices suggest evaluating new ideas/models through A/B tests where decisions are made based on business metrics like the overall engagement of users.A new model may show improvement in offline metrics but performance loss in online metrics.One reason that leads to this phenomenon is the counterfactual nature of the recommendation problem which can be addressed by off-policy evaluation methods [6][3].Another reason is the degree of causal connection between offline evaluation metrics and observed online metrics.In this work, we will share our learnings from two set of A/B tests that we conducted at Max 1 where we observed a mismatch between online and offline metrics due to a weak causal connection between online and offline metrics.Thanks to learnings from A/B tests, we discovered and quantified the impact of series to movie ratio at recommendations.Our experiments show that there is an optimal amount of series to movies ratio that provides the best possible results for user engagement.Production Model: Personalization model at Max powers the horizontal and vertical ranking of items on the homepage.