In large-scale online experimentation platforms, experimenters aim to discover the best treatment (arm) among multiple candidates. Traditional A/B testing and multi-armed bandits (MAB) algorithms are two popular designs. The former usually achieves a higher power but may hurt the customers' satisfaction when always recommending a poor arm, while the latter aims at improving the customers' experience (collecting more rewards) but faces the loss of testing power. Recently, [26] combine the advantage of A/B testing and MAB algorithms to maximize the testing power while maintaining more rewards for experiments with two-arm and Bernoulli rewards. However, in practice, the number of arms is usually larger than two and the reward type also varies. In multi-arm experiments, the required sample size to find the optimal arm blows up to guarantee a false discovery rate with the increase of arm numbers, bringing high opportunity costs to experimenters. To save the cost during the long experimental process, we propose a more efficient sequential test framework named Soptima that can work with general reward types. Inspired by the design of traditional MAB algorithms in chasing rewards and A/B testing in maximizing power, we propose an Elimination-type strategy adapted to this framework to dynamically adjust the traffic split on arms. This strategy cooperating with Soptima simultaneously maintains the advantage of the A/B testing in maximizing the testing power, the sequential test methods in saving the sample size, and the MAB algorithms in collecting rewards. The theoretical analysis gives guarantees on the Type-I, Type-II, and optimality error rates of the proposed approach. A series of experiments from both simulation and industrial historical data sets are conducted to verify the superiority of our approach compared with available baselines.

Online controlled experiments, often referred to as A/B tests, are extensively conducted by major technology companies to evaluate the effectiveness of product strategies and inform product decision-making. The sampling process in A/B tests is not instantaneous; subjects, such as users of online platforms, arrive at the platform over time and are recruited continuously throughout the experiment. This ongoing nature of sampling can lead to shifts in sample characteristics over the experimental duration, raising issues of external validity. In other words, the causal findings derived from an experiment of a particular duration may not be applicable to the target population, potentially biasing decision-making.

Recommender systems are essential for content-sharing platforms by curating personalized content. To evaluate updates of recommender systems targeting content creators, platforms frequently engage in creatorside randomized experiments to assess their performance. These experiments help estimate treatment effects, defined as the difference in outcomes when a new (vs. the status quo) algorithm is deployed on the platform. We show that the standard difference-in-means estimator can lead to a biased treatment effect estimate. This bias can occur in either direction and may even cause a reversal in the estimated sign, leading to incorrect decision-making. This bias arises because of recommender interference, which occurs when treated and control creators compete for exposure through the recommender system. We propose a "recommender choice model" that captures how an item is chosen among a pool comprised of both treated and control content items. By combining a structural choice model with neural networks, the framework directly models the interference pathway in a microfounded way while accounting for rich viewer-content heterogeneity. This model enables counterfactual evaluations of treatment effects under alternative treatment assignments (e.g., all treated or all control) for both the entire population and specific subgroups. Within this modeling framework, we further construct a double/debiased estimator of the treatment effect that is consistent and asymptotically normal under regularity conditions. We demonstrate its empirical performance with a field experiment on Weixin short-video platform. Besides the standard creator-side experiment, we implement a costly blocked double-sided randomization design to obtain a benchmark estimate of the treatment effect without interference bias. We show that the proposed estimator significantly reduces bias in treatment effect estimates compared to the standard difference-in-means estimator. The full paper is available at https://arxiv.org/abs/2406.14380.