At Peloton, we are challenged to not just surface relevant recommendations of fitness classes to our members, but also timely ones. As our fitness content library expands, we continually produce classes on certain themes which are most timely during a narrow time window. To address this challenge, we provide some control over our recommendations to external stakeholders, such as production and marketing teams. They enter timed boosts of certain classes during the windows they are relevant in. We have built out algorithms which take these desired classes and elevate the number of impressions for them, while preserving members’ engagement with our recommendations. In this paper, we discuss the system, the algorithms and some results from a few A/B tests showing how boosting works in practice.

Online controlled experiments, in which different variants of a product are compared based on an Overall Evaluation Criterion (OEC), have emerged as a gold standard for decision making in online services. It is vital that the OEC is aligned with the overall goal of stakeholders for effective decision making. However, this is a challenge when the overall goal is not immediately observable. For instance, we might want to understand the effect of deploying a feature on long-term retention, where the outcome (retention) is not observable at the end of an A/B test.