Explore Further @ Bing is a webpage-to-webpage recommendation product, enhancing the search experience on Bing by surfacing engaging webpage recommendations tied to the search result URLs. In this paper, we present our approach for leveraging Large Language Models (LLMs) for enhancing our web-scale recommendation system. We describe the development and validation of our LLM-powered recommendation quality metric RecoDCG. We discuss our core techniques for utilizing LLMs to make our ranking stage quality-aware. Furthermore, we detail Q' recall, a recall path that enhances our system's candidate generation stage by leveraging LLMs to produce complementary and engaging recommendation candidates. We also address how we optimize our system for multiple objectives, balancing recommendation quality with click metrics. We deploy our work to production, achieving a significant improvement in recommendation quality. We share results from offline and online experiments as well as insights and steps we took to ensure our approaches scale effectively for our web-scale needs.

Explore Further @ Bing (Web Recommendations) is a web-scale query independent webpage-to-webpage recommendation system with an index size of over 200 billion webpages. Due to the significant variability in webpage quality across the web and the reliance of our system on learning soleley user behavior (clicks), our production system was susceptible to serving clickbait and low-quality recommendations. Our team invested several months in developing and shipping several improvements that utilize LLM-generated recommendation quality labels to enhance our ranking stack to improve the nature of the recommendations we show to our users. Another key motivation behind our efforts was to go beyond merely surfacing relevant webpages, focusing instead on prioritizing more useful and authoritative content that delivers value to users based on their implied intent. We demonstrate how large language models (LLMs) offer a powerful tool for product teams to gain deeper insights into shifts in product experience and user behavior following significant improvements or changes to a production system. In this work, to enable our analysis, we also showcase the use of a small language model (SLM) to generate better-quality webpage text features and summaries at scale and describe our approach to mitigating position bias in user interaction logs."

Recommender Systems is one of the main machine learning applications for an e-commerce platforms such as Etsy, a two sided marketplace. A frequent usage of such system is for item-to-item recommendations that show similar items (also referred as listings) based on the listing a user is currently viewing. Item-to-item recommendations typically take into account information that are only associated with the target listing and other listings in the inventory. However, other contextual information such as user intents, queries and seasonality, are often not taken into account. In this talk, we will present two approaches we developed to utilize additional contextual information in the form of queries in generating item-to-item recommendations. Moreover, we will present our journey in migrating Etsy’s rankers from linear to non-linear models. Additionally, we propose new metrics to evaluate candidate sets that accesses diversity and price spread, while not compromising relevance. The proposed metrics can also be used beyond the current application. Our proposed candidate set generation approach outperforms the model in production as well as yielding significant lift in conversion rate and other engagement metrics as indicated by several A/B tests.