Promotional campaigns are essential drivers of customer engagement and revenue in e-commerce. Maintaining these campaigns within budget constraints requires targeted allocation, traditionally achieved through causal uplift models that rely on vast datasets of user interactions, including non-converted sessions, which introduce challenges such as noisy data, attribution complexity and imbalanced outcomes. We propose a novel approach using converted-only data, which reduces training data size, simplifies attribution, improves efficiency, and mitigates the impact of non-converted interactions. We present a generalized framework for budget constrained promotion allocation with converted-only data and validate it through a benchmarking study and multiple large-scale deployments at Booking.com, positively impacting the experience of millions of customers worldwide. Our results demonstrate that the proposed method is competitive with standard modeling approaches and, in some cases, significantly outperforms them.

With the rapid integration of Machine Learning (ML) in business applications and processes, it is crucial to ensure the quality, reliability and reproducibility of such systems. We suggest a methodical approach towards ML system quality assessment and introduce a structured Maturity framework for governance of ML. We emphasize the importance of quality in ML and the need for rigorous assessment, driven by issues in ML governance and gaps in existing frameworks. Our primary contribution is a comprehensive open-sourced quality assessment method, validated with empirical evidence, accompanied by a systematic maturity framework tailored to ML systems. Drawing from applied experience at Booking.com, we discuss challenges and lessons learned during large-scale adoption within organizations. The study presents empirical findings, highlighting quality improvement trends and showcasing business outcomes. The maturity framework for ML systems, aims to become a valuable resource to reshape industry standards and enable a structural approach to improve ML maturity in any organization.

Machine learning applications are rapidly adopted by industry leaders in any field.The growth of investment in AI-driven solutions,including the emerging field of General AI (GenAI), has created new challenges in managing Data Science and ML resources, people and projects as a whole.The discipline of managing applied machine learning teams, requires a healthy mix between agile product development tool-set and a long term research oriented mindset.The abilities of investing in deep research while at the same time connecting the outcomes to significant business results create a large knowledge based on management methods and best practices in the field.The Third KDD Workshop on Applied Machine Learning Management brings together applied research managers from various fields to share methodologies and case-studies on management of ML teams, products, and projects, achieving business impact with advanced AI-methods.

Uplift modeling is a collection of machine learning techniques for estimating causal effects of a treatment at the individual or subgroup levels. Over the last years, causality and uplift modeling have become key trends in personalization at online e-commerce platforms, enabling the selection of the best treatment for each user in order to maximize the target business metric. Uplift modeling can be particularly useful for personalized promotional campaigns, where the potential benefit caused by a promotion needs to be weighed against the potential costs. In this tutorial we will cover basic concepts of causality and introduce the audience to state-of-the-art techniques in uplift modeling. We will discuss the advantages and the limitations of different approaches and dive into the unique setup of constrained uplift modeling. Finally, we will present real-life applications and discuss challenges in implementing these models in production.

Machine learning applications are rapidly adopted by industry leaders in any field. The growth of investment in AI-driven solutions created new challenges in managing Data Science and ML resources, people and projects as a whole. The discipline of managing applied machine learning teams, requires a healthy mix between agile product development tool-set and a long term research oriented mindset. The abilities of investing in deep research while at the same time connecting the outcomes to significant business results create a large knowledge based on management methods and best practices in the field. The Second KDD Workshop on Applied Machine Learning Management brings together applied research managers from various fields to share methodologies and case-studies on management of ML teams, products, and projects, achieving business impact with advanced AI-methods.

The Workshop on Recommenders in Tourism (RecTour) 2023, which is held in conjunction with the 17th issue of the ACM Conference on Recommender Systems (RecSys) in Singapore, addresses specific challenges for recommender systems in the tourism domain. In this overview paper, we summarize our motivations to organize the RecTour workshop and present the main topic areas of RecTour submissions. These include context-aware recommendations, group recommender systems, recommending composite items, decision making and user interaction issues, different information sources and various application scenarios.

Promotions and discounts are essential components of modern e-commerce platforms, where they are often used to incentivize customers towards purchase completion. Promotions also affect revenue and may incur a monetary loss that is often limited by a dedicated promotional budget. We propose an Online Constrained Multiple-Choice Promotions Personalization framework, driven by causal incremental estimations achieved by uplift modeling. Our work formalizes the problem as an Online Multiple-Choice Knapsack Problem and extends the existent literature by addressing cases with negative weights and values as a result from causal estimations. Our real-time adaptive method guarantees budget constraints compliance achieving above 99.7% of the potential optimal impact on various datasets. It was deployed in a large-scale experimental study at Booking.com - one of the leading online travel platforms in the world. The application resulted in 162% improvement in sales while complying a zero-budget constraint, enabling long-term self-sponsored promotional campaigns.

Machine learning applications are rapidly adopted by industry leaders in any field. The growth of investment in AI-driven solutions created new challenges in managing Data Science and ML resources, people and projects as a whole. The discipline of managing applied machine learning teams, requires a healthy mix between agile product development tool-set and a long term research oriented mindset. The abilities of investing in deep research while at the same time connecting the outcomes to significant business results create a large knowledge based on management methods and best practices in the field. The Workshop on Applied Machine Learning Management brings together applied research managers from various fields to share methodologies and case-studies on management of ML teams, products, and projects, achieving business impact with advanced AI-methods.

Modern e-commerce platforms make use of promotional offers such as discounts and rewards to encourage customers to complete purchases. While offering the promotions has a great effect on the sales, it also generates a monetary loss. By utilizing causal machine learning and optimization, our team at Booking.com was able to personalize the promotions allocation to customers, while efficiently controlling the spend within a given budget. In this talk we’ll share the personalized promotion assignment techniques, such as uplift modeling and constrained optimization, which helped us to predict the outcomes of discounts offering and allocate them efficiently. This solution allowed us to unlock promotional campaigns to bring more value to the customers and grow our business.

The Workshop on Recommenders in Tourism (RecTour) 2022, which is held in conjunction with the 16th ACM Conference on Recommender Systems (RecSys), addresses specific challenges for recommender systems in the tourism domain. In this overview paper, we summarize our motivations to organize the RecTour workshop and present the main topic areas of RecTour submissions. These include context-aware recommendations, group recommender systems, recommending composite items, decision making and user interaction issues, different information sources and various application scenarios.

The influence maximization problem aims to find the best seeding set of nodes in a network to increase the influence spread, under various information diffusion models. Recent advances have shown the importance of the timing of the seeding and introduced the sequential seeding approach, determining a step-by-step cascade of activations. Our study explores a novel Deterministic Influence Maximization Approach (DIMA) for time-based sequential seeding dynamics in a threshold-based model. We examine the problem characteristics and formulate solutions optimizing a scheduled sequential seeding strategy. Based on a set of empirical simulations we demonstrate the properties of the deterministic sequential problem, incorporate three different mathematical programming formulations and provide an initial benchmark for optimization techniques.

Personalization is omnipresent in our life, with applications ranging from entertainment and commercial uses to smart devices and medical treatments. The integration of personalization in various products turned rapidly from an unnecessary luxury to a commodity that is expected by customers. While different machine learning fields present state-of-the-art advances and super-human performance, personalization applications are often late-adopters of novel solutions due to their complex framing and multiple stakeholders' with different business goals. The role of personalisation applications is also ambiguous: it is unclear, for instance, whether models just predict a user's next action or proactively affect the user's selections. This talk focuses on examining the role of recommenders and their ability to adapt to customer feedback. Key topics such as causality and active exploration are depicted with real examples and demonstrated alongside business considerations and implementation challenges. It relies on recent advances in the field and on work conducted at Booking.com, where we implement personalization models on one of the world's leading online travel platform.

We introduce a novel dataset of real multi-destination trips booked through Booking.com's online travel platform. The dataset consists of 1.5 million reservations representing 359,000 unique journeys made across 39,000 destinations. As such, the data is particularly well suited to model sequential recommendation and retrieval problems in a high cardinality target space. To preserve user privacy and protect business-sensitive statistics, the data is fully anonymized, sampled and limited to five user origin markets. Even so, the dataset is representative of the general travel purchase behavior and therefore presents a uniquely valuable resource for Machine Learning and information retrieval researchers. This work provides an overview of the dataset. It reports several benchmark results for relevant recommendation problems obtained as part of the recently held Booking.com data challenge during the WSDM WebTour workshop.

Personalization is one of the key applications in machine learning with widespread usage across e-commerce, entertainment, production, healthcare and many other industries. While various machine learning techniques present novel state-of-the-art advances and super-human performance year-over-year, personalization and recommender-systems applications are often late-adopters of novel solutions due to problem hardness and implementation complexity. This tutorial presents recent advances across the personalization industry and demonstrates their practical applications in real case-studies of world-leading online platforms. Key trends such as deep learning, causality and active exploration with bandits are depicted with real examples and demonstrated alongside their business considerations and implementation challenges.Rising topics like explainability, fairness, natural interfaces and content generation are covered, touching on aspects of both technology and user experience. Our tutorial relies on recent advances in the field and on work conducted at Booking.com, where we implement personalization models on one of the world's leading online travel platform.

Over the years, the Web has become a premier source of information in almost every area we can think about. When considering tourism, the Web became the primary source of information for travelers. When planning trips, people search for information about destinations, accommodations, attractions, means of transportation, in short, everything related to their future trip. Once done searching they reserve almost everything online. The blessing of the easily accessible information comes with the curse of information overload. This brings Web search techniques and recommendation systems come into play. This is especially true recently with the appearance of COVID-19 and the uncertainty and transformative power it brings to travelling. WebTour 2021 brings together researchers and practitioners working on developing and improving tools and techniques for improving users ability to better find relevant information that matches their needs.

Promotions and discounts have become key components of modern e-commerce platforms. For online travel platforms (OTPs), popular promotions include room upgrades, free meals and transportation services. By offering these promotions, customers can get more value for their money, while both the OTP and its travel partners may grow their loyal customer base. However, the promotions usually incur a cost that, if uncontrolled, can become unsustainable. Consequently, for a promotion to be viable, its associated costs must be balanced by incremental revenue within set financial constraints. Personalized treatment assignment can be used to satisfy such constraints. This paper introduces a novel uplift modeling technique, relying on the Knapsack Problem formulation, that dynamically optimizes the incremental treatment’s outcome subject to the required Return on Investment (ROI) constraints. The technique leverages Retrospective Estimation, a modeling approach that relies solely on data from positive outcome examples. The method also addresses training data bias, long term effects, and seasonality challenges via online-dynamic calibration. This approach was tested via offline experiments and online randomized controlled trials at Booking.com - a leading OTP with millions of customers worldwide, resulting in a significant increase in the target outcome while staying within the required financial constraints and outperforming other approaches.

One highly studied topic in the field of social networks is the search for influential nodes, that when seeded (i.e. infected intentionally), may infect a large portion of the network through a viral process. However, when it comes to the spread of new products, such viral processes are rather rare. Social influence is indeed an important factor when it comes to the act of adopting a new product. However, this influence is usually latent and does not trigger the purchase action by itself, it therefore requires an additional sales effort. We propose a model and a method that better fit the product adoption scenario. Our method allocates the seeding efforts not only to precise nodes but also at precise points in time, such that the product adoption rate increases. By conducting a set of empirical simulations, we show that under realistic assumptions, our method improves the product adoption rate by 25%-50%.