An emerging dilemma that faces practitioners in large scale online experimentation for e-commerce is whether to use Multi-Armed Bandit (MAB) algorithms for testing or traditional A/B testing (A/B). This paper provides a comprehensive comparison between the two, from the perspectives of confidence intervals, hypothesis test powers, and their relationships with traffic split and sample size both theoretically and numerically. We first make comparisons between MAB with A/B tests in terms of conditions under which disjoint confidence intervals occur, and analyze their connection with the traffic split. Then we explore the relationship between the two in terms of sample sizes needed to achieve the required hypothesis test power, and analyze under what conditions MAB could have a higher test power than A/B given the same sample size. Based on the theoretical analysis, we propose two new MAB algorithms that combine the strengths of traditional MAB and A/B together, with higher (or equal) test power and higher (or equal) expected rewards than A/B testing under certain common conditions in e-commerce. Last, we evaluate and compare the performance among the classical MAB algorithms, our newly proposed MAB algorithms, and A/B testing in terms of their accuracy of identifying ground truth winner with practical significance, power rewards trade-off, sample sizes etc. in both simulated datasets and industrial historical datasets. We hope the work can not only facilitate a better understanding of pros and cons of MAB and A/B testing, but also help build the connections between the two and provide possible approaches that can leverage the best from both worlds.

On e-commerce platforms, predicting if two products are compatible with each other is an important functionality to achieve trustworthy product recommendation and search experience for consumers. However, accurately predicting product compatibility is difficult due to the heterogeneous product data and the lack of manually curated training data. We study the problem of discovering effective labeling rules that can enable weakly-supervised product compatibility prediction. We develop AMRule, a multi-view rule discovery framework that can (1) adaptively and iteratively discover novel rulers that can complement the current weakly-supervised model to improve compatibility prediction; (2) discover interpretable rules from both structured attribute tables and unstructured product descriptions. AMRule adaptively discovers labeling rules from large-error instances via a boosting-style strategy, the high-quality rules can remedy the current model's weak spots and refine the model iteratively. For rule discovery from structured product attributes, we generate composable high-order rules from decision trees; and for rule discovery from unstructured product descriptions, we generate prompt-based rules from a pre-trained language model. Experiments on 4 real-world datasets show that AMRule outperforms the baselines by $5.98%$ on average and improves rule quality and rule proposal efficiency.

Recommendation systems have been crucial in driving revenue in e-commerce especially in online retail. Complementary item recommendation is a challenging problem within this field due to the inherent difficulty in defining how products relate to each other. In this paper we design and propose a complementary item recommendation system that uses multi-modal embeddings (both text-based and image-based). Our system is carefully designed and validated by domain experts. We provide extensive insights into our design choices and what does not work in terms of model choice and features used. We successfully demonstrate our system on two challenging datasets from a home improvement retailer: one involving outdoor furniture (patio) and the second involving bathroom products. When deployed live on the website, our model resulted in +170% increase in engaged visits to the recommendation container.