Experimentation in online digital platforms is often used to inform decision making. Specifically, the goal of many experiments is to optimize a metric of interest. Null hypothesis statistical testing can be ill-suited to this task, as it is indifferent to the magnitude of effect sizes and opportunity costs. Moreover, in practice, related experiments can often be grouped into portfolios called experimentation programs where the goal is to optimize a single metric. Past tests in an experimentation program inform useful priors on treatment effects. In these settings, we discuss how experimentation practice should change when the goal is optimization. We survey the literature on Bayesian and empirical Bayes analyses of A/B test portfolios, and single out the A/B Testing Problem [Azevedo et al., 2020] as a starting point, which treats experimentation as a constrained optimization problem. We emphasize that the framework can be implemented by appropriately tuning p-value thresholds, a result which seems under-utilized by practitioners. Furthermore, we develop several extensions of the A/B testing problem to managing multiple experimentation programs, managing a single program over time, and allowing units to be enrolled in multiple tests. Each of these can be solved by dynamic programming, and together they extend the utility of the framework in practice. We also consider the case of new experimentation programs without a strong prior on treatment effects. Finally, we discuss the implications of these results on experimentation programs in industry. For example, under no-cost assumptions, firms should be testing many more ideas, reducing test allocation sizes, and relaxing p-value thresholds away from p = 0.05. These results are exemplified with an application to data from Netflix experimentation programs. A draft of the paper may be found at https://arxiv.org/abs/2412.05508.

Identifying structural differences among proteins can be a non-trivial task. When contrasting ensembles of protein structures obtained from molecular dynamics simulations, biologically-relevant features can be easily overshadowed by spurious fluctuations. Here, we present SINATRA Pro, a computational pipeline designed to robustly identify topological differences between two sets of protein structures. Algorithmically, SINATRA Pro works by first taking in the 3D atomic coordinates for each protein snapshot and summarizing them according to their underlying topology. Statistically significant topological features are then projected back onto a user-selected representative protein structure, thus facilitating the visual identification of biophysical signatures of different protein ensembles. We assess the ability of SINATRA Pro to detect minute conformational changes in five independent protein systems of varying complexities. In all test cases, SINATRA Pro identifies known structural features that have been validated by previous experimental and computational studies, as well as novel features that are also likely to be biologically-relevant according to the literature. These results highlight SINATRA Pro as a promising method for facilitating the non-trivial task of pattern recognition in trajectories resulting from molecular dynamics simulations, with substantially increased resolution.