The mission of major e-commerce platforms is to enable their customers to find the best products for their needs. In the common case of large inventories, complex User Interfaces (UIs) are required to allow a seamless navigation. However, as UIs often contain many widgets of different relevance, the task of constructing an optimal layout arises in order to improve the customer's experience. This is a challenging task, especially in the typical industrial setup where multiple independent teams conflict by adding and modifying UI widgets. It becomes even more challenging due to the customer preferences evolving over time, bringing the need for adaptive solutions. In a previous work [6], we addressed this task by introducing a UI governance framework powered by Machine Learning (ML) algorithms that automatically and continuously search for the optimal layout. Nevertheless, we highlighted that naive algorithmic choices exhibit several issues when implemented in the industry, such as widget dependency, combinatorial solution space and cold start problem. In this work, we demonstrate how we deal with these issues using Combinatorial Bandits, an extension of Multi-Armed Bandits (MAB) where the agent selects not only one but multiple arms at the same time. We develop two novel approaches to model combinatorial bandits, inspired by the Natural Language Processing (NLP) and the Evolutionary Algorithms (EA) fields and present their ability to enable scalable UI optimization.

Booking.com is the world’s leading online travel platform where users make many decisions supported by our recommendations, such as destinations, travel dates, facilities, etc. This leads to a complex User Interface (UI) containing many widgets of different relevance for different users. We address the problem of constructing an optimal UI, a non-trivial problem, mainly due to user preferences evolving over time and multiple independent teams collaboratively building the UI. Our goal is to provide a personalized User Experience (UX) which adapts to changes in the environment and ensures governable, collaborative product development. The solution relies on a Multi Armed Bandits (MAB) framework currently allowing product teams to collaborate on the construction of UIs and serving millions of users every day. We present examples of our solution and lessons learned during their implementation.

Online Travel Platforms are virtual two-sided marketplaces where guests search for accommodations and accommodation providers list their properties such as hotels and vacation rentals. The large majority of hotels are rated by official institutions with a number of stars indicating the quality of service they provide. It is a simple and effective mechanism that contributes to match supply with demand by helping guests to find options meeting their criteria and accommodation suppliers to market their product to the right segment directly impacting the number of transactions on the platform. Unfortunately, no similar rating system exists for the large majority of vacation rentals, making it difficult for guests to search and compare options and hard for vacation rentals suppliers to market their product effectively. In this work we describe a machine learned quality rating system for vacation rentals. The problem is challenging, mainly due to explainability requirements and the lack of ground truth. We present techniques to address these challenges and empirical evidence of their efficacy. Our system was successfully deployed and validated through Online Controlled Experiments performed in Booking.com, a large Online Travel Platform, and running for more than one year, impacting more than a million accommodations and millions of guests.

Promotions and discounts have become key components of modern e-commerce platforms. For online travel platforms (OTPs), popular promotions include room upgrades, free meals and transportation services. By offering these promotions, customers can get more value for their money, while both the OTP and its travel partners may grow their loyal customer base. However, the promotions usually incur a cost that, if uncontrolled, can become unsustainable. Consequently, for a promotion to be viable, its associated costs must be balanced by incremental revenue within set financial constraints. Personalized treatment assignment can be used to satisfy such constraints. This paper introduces a novel uplift modeling technique, relying on the Knapsack Problem formulation, that dynamically optimizes the incremental treatment’s outcome subject to the required Return on Investment (ROI) constraints. The technique leverages Retrospective Estimation, a modeling approach that relies solely on data from positive outcome examples. The method also addresses training data bias, long term effects, and seasonality challenges via online-dynamic calibration. This approach was tested via offline experiments and online randomized controlled trials at Booking.com - a leading OTP with millions of customers worldwide, resulting in a significant increase in the target outcome while staying within the required financial constraints and outperforming other approaches.

Booking.com is the world's largest online travel agent where millions of guests find their accommodation and millions of accommodation providers list their properties including hotels, apartments, bed and breakfasts, guest houses, and more. During the last years we have applied Machine Learning to improve the experience of our customers and our business. While most of the Machine Learning literature focuses on the algorithmic or mathematical aspects of the field, not much has been published about how Machine Learning can deliver meaningful impact in an industrial environment where commercial gains are paramount. We conducted an analysis on about 150 successful customer facing applications of Machine Learning, developed by dozens of teams in Booking.com, exposed to hundreds of millions of users worldwide and validated through rigorous Randomized Controlled Trials. Following the phases of a Machine Learning project we describe our approach, the many challenges we found, and the lessons we learned while scaling up such a complex technology across our organization. Our main conclusion is that an iterative, hypothesis driven process, integrated with other disciplines was fundamental to build 150 successful products enabled by Machine Learning.

Recommendation based on user preferences is a common task for e-commerce websites. New recommendation algorithms are often evaluated by offline comparison to baseline algorithms such as recommending random or the most popular items. Here, we investigate how these algorithms themselves perform and compare to the operational production system in large scale online experiments in a real-world application. Specifically, we focus on recommending travel destinations at Booking.com, a major online travel site, to users searching for their preferred vacation activities. To build ranking models we use multi-criteria rating data provided by previous users after their stay at a destination. We implement three methods and compare them to the current baseline in Booking.com: random, most popular, and Naive Bayes. Our general conclusion is that, in an online A/B test with live users, our Naive-Bayes based ranker increased user engagement significantly over the current online system.