Technology firms conduct randomized controlled experiments ("A/B tests") to learn which actions to take to improve business outcomes.In firms with mature experimentation platforms, experimentation programs can consist of many thousands of tests.To effectively scale experimentation, firms rely on decision rules: standard operating procedures for mapping the results of an experiment to a choice of treatment arm to launch to the general user population.Despite the critical role of decision rules in translating experimentation into business decisions, rigorous guidance on how to evaluate and choose decision rules is scarce.This paper proposes to evaluate decision rules based on their cumulative returns to business north star metrics.Although intuitive and easy to explain to decisionmakers, this quantity can be difficult to estimate, especially when experiments have weak signal-to-noise ratios.We develop a crossvalidation estimator that is much less biased than the naive plug-in estimator under conditions realistic to digital experimentation.We demonstrate the efficacy of our approach via a case study of 123 historical A/B tests at Netflix, where we used it to show that a new decision rule would have increased cumulative returns to the north star metric by an estimated 33%, directly leading to the adoption of the new rule.

Experimentation in online digital platforms is often used to inform decision making. Specifically, the goal of many experiments is to optimize a metric of interest. Null hypothesis statistical testing can be ill-suited to this task, as it is indifferent to the magnitude of effect sizes and opportunity costs. Moreover, in practice, related experiments can often be grouped into portfolios called experimentation programs where the goal is to optimize a single metric. Past tests in an experimentation program inform useful priors on treatment effects. In these settings, we discuss how experimentation practice should change when the goal is optimization. We survey the literature on Bayesian and empirical Bayes analyses of A/B test portfolios, and single out the A/B Testing Problem [Azevedo et al., 2020] as a starting point, which treats experimentation as a constrained optimization problem. We emphasize that the framework can be implemented by appropriately tuning p-value thresholds, a result which seems under-utilized by practitioners. Furthermore, we develop several extensions of the A/B testing problem to managing multiple experimentation programs, managing a single program over time, and allowing units to be enrolled in multiple tests. Each of these can be solved by dynamic programming, and together they extend the utility of the framework in practice. We also consider the case of new experimentation programs without a strong prior on treatment effects. Finally, we discuss the implications of these results on experimentation programs in industry. For example, under no-cost assumptions, firms should be testing many more ideas, reducing test allocation sizes, and relaxing p-value thresholds away from p = 0.05. These results are exemplified with an application to data from Netflix experimentation programs. A draft of the paper may be found at https://arxiv.org/abs/2412.05508.

When primary objectives are insensitive or delayed, experimenters may instead focus on proxy metrics derived from secondary outcomes. For example, technology companies often infer the long-term impacts of product interventions from their effects on short-term user engagement signals. We consider the meta-analysis of many historical experiments to learn the covariance of treatment effects on these outcomes, which can support the construction of such proxies. Even when experiments are plentiful, if treatment effects are weak, the covariance of estimated treatment effects across experiments can be highly biased. We overcome this with techniques inspired by weak instrumental variable analysis. We show that Limited Information Maximum Likelihood (LIML) learns a parameter equivalent to fitting total least squares to a transformation of the scatterplot of treatment effects, and that Jackknife Instrumental Variables Estimation (JIVE) learns another parameter computable from the average of Jackknifed covariance matrices across experiments. We also present a total covariance estimator for the latter estimand under homoskedasticity, which is equivalent to a k-class estimator. We show how these parameters can be used to construct unbiased proxy metrics under various structural models. Lastly, we discuss the real-world application of our methods at Netflix.