Large Language Models (LLMs) empower recommendation systems through their advanced reasoning and planning capabilities. However, the dynamic nature of user interests and content poses a significant challenge: While initial fine-tuning aligns LLMs with domain knowledge and user preferences, it fails to capture such real-time changes, necessitating robust update mechanisms. This paper investigates strategies for updating LLM-powered recommenders, focusing on the trade-offs between ongoing fine-tuning and Retrieval-Augmented Generation (RAG). Using an LLM-powered user interest exploration system as a case study, we perform a comparative analysis of these methods across dimensions like cost, agility, and knowledge incorporation. We propose a hybrid update strategy that leverages the long-term knowledge adaptation of periodic fine-tuning with the agility of low-cost RAG. We demonstrate through live A/B experiments on a billion-user platform that this hybrid approach yields statistically significant improvements in user satisfaction, offering a practical and cost-effective framework for maintaining high-quality LLM-powered recommender systems.

Exposure bias and its induced feedback loop effect are well-known problems in recommender systems. Exploration is believed to be the key to break such feedback loops. While classical contextual bandit algorithms such as Upper-Confidence-Bound and Thompson Sampling have been successful in addressing the exploration-exploitation trade-off in the single-task settings with one clear reward signal, modern recommender systems often leverage multiple rich sources of feedback such as clicks, likes, dislikes, shares, satisfaction survey responses, and employ multi-task learning in practice. It is unclear how one can incorporate exploration in the multi-task setup with different objectives. In this paper, we study an efficient bandit algorithm tailored to multi-task recommender systems, named Multi-task Neural Linear Bandit (mtNLB). In particular, we investigate efficient feature embeddings in the multi-task setups that could be used as contextual features in the Neural Linear Bandit, a contextual bandit algorithm that nicely combines the representation power from DNN and simplicity in uncertainty calculation from linear models. We further study cost-effective approximations of the uncertainty estimate and principled ways to incorporate uncertainty into the multi-task scoring of items. To showcase the efficacy of our proposed method, we conduct live experiments on a large-scale commercial recommendation platform that serves billions of users. We evaluate the quality of the uncertainty estimate and demonstrate its ability to improve exploration across the different dimensions of the reward signals in comparison to baseline approaches.

Traditional recommendation systems are subject to a strong feedback loop by learning from and reinforcing past user-item interactions, which in turn limits the discovery of novel user interests. To address this, we introduce a hybrid hierarchical framework combining Large Language Models (LLMs) and classic recommendation models for user interest exploration. The framework controls the interfacing between the LLMs and the classic recommendation models through “interest clusters”, the granularity of which can be explicitly determined by algorithm designers. It recommends the next novel interests by first representing “interest clusters” using language, and employs a fine-tuned LLM to generate novel interest descriptions that are strictly within these predefined clusters. At the low level, it grounds these generated interests to an item-level policy by restricting classic recommendation models, in this case a transformer-based sequence recommender to return items that fall within the novel clusters generated at the high level. We showcase the efficacy of this approach on an industrial-scale commercial platform serving billions of users. Live experiments show a significant increase in both exploration of novel interests and overall user enjoyment of the platform.

Effective exploration is believed to positively influence the long-term user experience on recommendation platforms. Determining its exact benefits, however, has been challenging. Regular A/B tests on exploration often measure neutral or even negative engagement metrics while failing to capture its long-term benefits. We here introduce new experiment designs to formally quantify the long-term value of exploration by examining its effects on content corpus, and connecting content corpus growth to the long-term user experience from real-world experiments. Once established the values of exploration, we investigate the Neural Linear Bandit algorithm as a general framework to introduce exploration into any deep learning based ranking systems. We conduct live experiments on one of the largest short-form video recommendation platforms that serves billions of users to validate the new experiment designs, quantify the long-term values of exploration, and to verify the effectiveness of the adopted neural linear bandit algorithm for exploration.

Recommendation system serves as a conduit connecting users to an incredibly large, diverse and ever growing collection of contents. In practice, missing information on fresh contents needs to be filled in order for them to be exposed and discovered by their audience. In this context, we are delighted to share our success stories in building a dedicated fresh content recommendation stack on a large commercial platform and also shed a light on the utilization of Large Language Models (LLMs) for fresh content recommendations within an industrial framework. To nominate fresh contents, we built a multi-funnel nomination system that combines (i) a two-tower model with strong generalization power for coverage, and (ii) a sequence model with near real-time update on user feedback for relevance, which effectively balances between coverage and relevance. Beyond that, by harnessing the reasoning and generalization capabilities of LLMs, we are presented with exciting prospects to enhance recommendation systems. We share our initial efforts on employing LLMs as data augmenters to bridge the knowledge gap on cold-start items during the training phase. This innovative approach circumvents the costly generation process during inference, presenting a model-agnostic, forward-looking solution for fresh content recommendation.

Recommendation system serves as a conduit connecting users to an incredibly large, diverse and ever growing collection of contents. In practice, missing information on fresh (and tail) contents needs to be filled in order for them to be exposed and discovered by their audience. We here share our success stories in building a dedicated fresh content recommendation stack on a large commercial platform. To nominate fresh contents, we built a multi-funnel nomination system that combines (i) a two-tower model with strong generalization power for coverage, and (ii) a sequence model with near real-time update on user feedback for relevance. The multi-funnel setup effectively balances between coverage and relevance. An in-depth study uncovers the relationship between user activity level and their proximity toward fresh contents, which further motivates a contextual multi-funnel setup. Nominated fresh candidates are then scored and ranked by systems considering prediction uncertainty to further bootstrap content with less exposure. We evaluate the benefits of the dedicated fresh content recommendation stack, and the multi-funnel nomination system in particular, through user corpus co-diverted live experiments. We conduct multiple rounds of live experiments on a commercial platform serving billion of users demonstrating efficacy of our proposed methods.

Complementary item recommendation finds products that go well with one another (e.g., a camera and a specific lens). While complementary items are ubiquitous, the dimensions by which items go together can vary by both product and category, making it difficult to detect complementary items at scale. Moreover, in practice, user preferences for complementary items can be complex combinations of item quality and evidence of complementarity. Hence, we propose a new neural complementary recommender Encore that can jointly learn complementary item relationships and user preferences. Specifically, Encore (i) effectively combines and balances both stylistic and functional evidence of complementary items across item categories; (ii) naturally models item latent quality for complementary items through Bayesian inference of customer ratings; and (iii) builds a novel neural network model to learn the complex (non-linear) relationships between items for flexible and scalable complementary product recommendations. Through experiments over large Amazon datasets, we find that Encore effectively learns complementary item relationships, leading to an improvement in accuracy of 15.5% on average versus the next-best alternative.

Understanding user interests and expertise is a vital component toward creating rich user models for information personalization in social media, recommender systems and web search. To capture the pair-wise interactions between geo-location and user's topical profile in social-spatial systems, we propose the modeling of fine-grained and multi-dimensional user geo-topic profiles. We then propose a two-layered Bayesian hierarchical user factorization generative framework to overcome user heterogeneity and another enhanced model integrated with user's contextual information to alleviate multi-dimensional sparsity. Through extensive experiments, we find the proposed model leads to a 5\textasciitilde13% improvement in precision and recall over the alternative baselines and an additional 6\textasciitilde11% improvement with the integration of user's contexts.

We propose a new model toward improving the quality of image recommendations in social sharing communities like Pinterest, Flickr, and Instagram. Concretely, we propose Neural Personalized Ranking (NPR) -- a personalized pairwise ranking model over implicit feedback datasets -- that is inspired by Bayesian Personalized Ranking (BPR) and recent advances in neural networks. We further build an enhanced model by augmenting the basic NPR model with multiple contextual preference clues including user tags, geographic features, and visual factors. In our experiments over the Flickr YFCC100M dataset, we demonstrate the proposed NPR model is more effective than multiple baselines. Moreover, the contextual enhanced NPR model significantly outperforms the base model by 16.6% and a contextual enhanced BPR model by 4.5% in precision and recall.

User interests and expertise are valuable but often hidden resources on social media. For example, Twitter Lists and LinkedIn's Skill Tags provide a partial perspective on what users are known for (by aggregating crowd tagging knowledge), but the vast majority of users are untagged; their interests and expertise are essentially hidden from important applications such as personalized recommendation, community detection, and expert mining. A natural approach to overcome these limitations is to intelligently learn user topical profiles by exploiting information from multiple, heterogeneous footprints: for instance, Twitter users who post similar hashtags may have similar interests, and YouTube users who upvote the same videos may have similar preferences. And yet identifying "similar" users by exploiting similarity in such a footprint space often provides conflicting evidence, leading to poor-quality user profiles. In this paper, we propose a unified model for learning user topical profiles that simultaneously considers multiple footprints. We show how these footprints can be embedded in a generalized optimization framework that takes into account pairwise relations among all footprints for robustly learning user profiles. Through extensive experiments, we find the proposed model is capable of learning high-quality user topical profiles, and leads to a 10-15% improvement in precision and mean average error versus a cross-triadic factorization state-of-the-art baseline.

This paper tackles the geospatial tag estimation problem, which is of critical importance for location-based search, retrieval, and mining applications. However, tag estimation is challenging due to massive sparsity, uncertainty in the tags actually used, as well as diversity across locations and times. Toward overcoming these challenges, we propose a community-based smoothing approach that seeks to uncover hidden conceptual communities which link multiple related locations by their common interests in addition to their proximity. Through extensive experiments over a sample of millions of geotagged Twitter posts, we demonstrate the effectiveness of the smoothing approach and validate the intuition that geo-locations have the tendency to share similar “ideas” in the formation of conceptual communities.

We address the challenge of personalized recommendation of high quality content producers in social media. While some candidates are easily identifiable (say, by being "favorited" many times), there is a long-tail of potential candidates for whom we have little evidence. Through careful modeling of contextual factors like the geo-spatial, topical, and social preferences of users, we propose a tensor-based personalized expert recommendation framework that integrates these factors for revealing latent connections between homogeneous entities (e.g., users and users) and between heterogeneous entities (e.g., users and experts). Through extensive experiments over geo-tagged Twitter data, we find that the proposed framework can improve the quality of recommendation by over 30% in both precision and recall compared to the state-of-the-art.

Discovering what people are known for is valuable to many important applications such as recommender systems. Unlike an individual's personal interests, what a user is known for is reflected by the views of others, and is often not easily discerned for a long-tail of the vast majority of users. In this paper, we tackle the problem of discovering what users are known for through a probabilistic model called Bayesian Contextual Poisson Factorization. Moving beyond just modeling user's content, it naturally models and integrates additional contextual factors, concretely, user's geo-spatial footprints and social influence, to overcome noisy online activities and social relations. Through GPS-tagged social media datasets, we find that the proposed method can improve known-for prediction performance by 17.5% in precision and 20.9% in recall on average, and that it can capture the implicit relationships between a user's known-for profile and her content, geo-spatial and social influence.

We propose a lightweight system for (i) semi-automatically discovering and tracking bias themes associated with opposing sides of a topic; (ii) identifying strong partisans who drive the online discussion; and (iii) inferring the opinion bias of "regular" participants. By taking just two hand-picked seeds to characterize the topic-space (e.g., "pro-choice" and "pro-life") as weak labels, we develop an efficient optimization-based opinion bias propagation method over the social/information network. We show how this approach leads to a 20% accuracy improvement versus a next-best alternative for bias estimation, as well as uncovering the opinion leaders and evolving themes associated with these topics. We also demonstrate how the inferred opinion bias can be integrated into user recommendation, leading to a 26% improvement in precision.

Experts are important for providing reliable and authoritative information and opinion, as well as for improving online reviews and services. While considerable previous research has focused on finding topical experts with broad appeal -- e.g., top Java developers, best lawyers in Texas -- we tackle the problem of personalized expert recommendation, to identify experts who have special personal appeal and importance to users. One of the key insights motivating our approach is to leverage the geo-spatial preferences of users and the variation of these preferences across different regions, topics, and social communities. Through a fine-grained GPS-tagged social media trace, we characterize these geo-spatial preferences for personalized experts, and integrate these preferences into a matrix factorization-based personalized expert recommender. Through extensive experiments, we find that the proposed approach can improve the quality of recommendation by 24% in precision compared to several baselines. We also find that users' geo-spatial preference of expertise and their underlying social communities can ameliorate the cold start problem by more than 20% in precision and recall.