Technology firms conduct randomized controlled experiments ("A/B tests") to learn which actions to take to improve business outcomes.In firms with mature experimentation platforms, experimentation programs can consist of many thousands of tests.To effectively scale experimentation, firms rely on decision rules: standard operating procedures for mapping the results of an experiment to a choice of treatment arm to launch to the general user population.Despite the critical role of decision rules in translating experimentation into business decisions, rigorous guidance on how to evaluate and choose decision rules is scarce.This paper proposes to evaluate decision rules based on their cumulative returns to business north star metrics.Although intuitive and easy to explain to decisionmakers, this quantity can be difficult to estimate, especially when experiments have weak signal-to-noise ratios.We develop a crossvalidation estimator that is much less biased than the naive plug-in estimator under conditions realistic to digital experimentation.We demonstrate the efficacy of our approach via a case study of 123 historical A/B tests at Netflix, where we used it to show that a new decision rule would have increased cumulative returns to the north star metric by an estimated 33%, directly leading to the adoption of the new rule.

Whilst absorptive capacity has been identified as an important contributor to the effective implementation of IT systems, previous studies have failed to explicitly consider the contribution of individual and organizational knowledge processes. Nine case studies of Enterprise Resource Planning (ERP) implementation were undertaken. The case studies were all undertaken in SMEs in a developing country since this is an important but under researched area for the application of the concept of absorptive capacity. A particular implication of the findings is that firms lacking knowledge of IT implementation cannot simply seek this from external sources but must develop internal organizational knowledge processes if their implementations of IT systems are to be effective. This finding is particularly pertinent to the developing country and SME context of this study, where low levels of experience within the firm and the loss of experienced staff are found to impact on the development of absorptive capacity.