Crowdsourcing platforms use various truth discovery algorithms to aggregate annotations from multiple labelers. In an online setting, however, the main challenge is to decide whether to ask for more annotations for each item to efficiently trade off cost (i.e., the number of annotations) for quality of the aggregated annotations. In this paper, we propose a novel approach for general complex annotation (such as bounding boxes and taxonomy paths), that works in an online crowdsourcing setting. We prove that the expected average similarity of a labeler is linear in their accuracy conditional on the reported label. This enables us to infer reported label accuracy in a broad range of scenarios. We conduct extensive evaluations on real-world crowdsourcing data from Meta and show the effectiveness of our proposed online algorithms in improving the cost-quality trade-off.

Binary classification models are ubiquitous, and reliably measuring their performance is critical for their proper usage. Ideally, the performance of supervised models is measured using high-quality labeled datasets that are sufficiently large and representative of the population. However, obtaining labels for all segments of the population can be difficult, and model performance typically varies across different segments of the population (e.g., in different countries). In this work, we present a novel methodology to estimate the performance of a binary classifier in segments of the population where labels are unavailable. The main idea is that if two segments are "similar,'' then the performance of the classifier in these two segments would also be "similar.'' Specifically, we define a way to measure similarity between segments, and propose a statistical model that describes the performance of the model in unlabeled segments as a function of the performance in labeled segments. With extensive numerical experiments on synthetic and real-world datasets, we demonstrate that the proposed method substantially improves over existing methods in both estimation accuracy and computational efficiency. We also showcase the application of our method on the Instagram Adult Classifier to improve the geographic coverage and usability of the model.

Online platforms regularly rely on human annotators to make real-time operational decisions for tasks such as content moderation. While crowdsourcing models have been proposed for aggregating noisy labels, they do not generalize well when annotators produce a labels in a large space, e.g., generated from complex review trees. We study a novel crowdsourcing setting with D possible operational decisions or outcomes, but annotators produce labels in a larger space of size L > D which are mapped to decisions through a known mapping function. For content moderation, such labels can correspond to violation reasons (e.g. nudity, violence), while the space of decisions is binary: remove the content or keep it up. In this setting, it is more important to make the right decision rather than estimating the correct underlying label. Existing methods typically separate out the labels to decisions mapping from the modeling of annotators, leading to sub-optimal statistical inference efficiency and excessive computation complexity. We propose a novel confusion matrix model for each annotator that leverages this mapping. Our model is parameterized in a hierarchical manner with both population parameters shared across annotators to model shared confusions and individual parameters to admit heterogeneity among annotators. With extensive numerical experiments, we demonstrate that the proposed model substantially improves accuracy over existing methods and scales well for moderate and large L. In a real-world application on content moderation at Meta, the proposed method offers a 13% improvement in AUC over prior methods, including Meta's existing model in production.