Recommender systems play a crucial role in online platforms, providing personalized recommendations for purchases, content consumption, and interpersonal connections. These systems involve two sides: producers (sellers, content creators, service providers, etc.) and consumers (buyers, viewers, customers, etc.). To optimize online recommender systems, A/B tests serve as the golden standard for comparing different ranking models and evaluating their impacts on both sides. While consumer-side experiments are relatively straightforward to design and commonly employed to assess ranking changes' effects on the behavior of consumers (buyers, viewers, etc.), designing producer-side experiments for an online recommender/ranking system is notably more complex. This complexity arises from the necessity of ranking producer items in the treatment and control groups by different models and then merging them into a unified ranking for presentation to each consumer. Existing design solutions in the literature lack rigorous guiding principles, leading to ad hoc approaches. In this paper, we address the limitations of current methods and propose the principles of consistency and monotonicity for designing producer-side experiments in online recommender systems. Building upon these principles, we also present a systematic solution based on counterfactual interleaving designs to accurately measure the impacts of ranking changes on the producers (sellers, content creators, etc.).

A/B tests have been widely adopted across industries as the golden rule that guides decision making. However, the long-term true north metrics we ultimately want to drive through A/B test may take a long time to mature. In these situations, a surrogate metric which predicts the long-term metric is often used instead to conclude whether the treatment is effective. However, because the surrogate rarely predicts the true north perfectly, a regular A/B test based on surrogate metrics tends to have high false positive rate and the treatment variant deemed favorable from the test may not be the winning one. In this paper, we discuss how to adjust the A/B testing comparison to ensure experiment results are trustworthy. We also provide practical guidelines on the choice of good surrogate metrics. To provide a concrete example of how to leverage surrogate metrics for fast decision making, we present a case study on developing and evaluating the predicted confirmed hire surrogate metric in LinkedIn job marketplace.