Multi-annotator learning (MAL) aims to model annotator-specific labeling patterns. However, existing methods face a critical challenge: they simply skip updating annotator-specific model parameters when encountering missing labels—a common scenario in real-world crowdsourced datasets where each annotator labels only small subsets of samples. This leads to inefficient data utilization and overfitting risks. To this end, we propose a novel similarity-weighted semi-supervised learning framework (SimLabel) that leverages inter-annotator similarities to generate weighted soft labels for missing annotations, enabling the utilization of unannotated samples rather than skipping them entirely. We further introduce a confidence-based iterative refinement mechanism that combines maximum probability with entropy-based uncertainty to prioritize predicted high-quality pseudo-labels to impute missing labels, jointly enhancing similarity estimation and model performance over time. For evaluation, we contribute a new multimodal multi-annotator dataset, AMER2, with high and more variable missing rates, reflecting real-world annotation sparsity and enabling evaluation across different sparsity levels. Extensive experiments validate the effectiveness of our method.

Multi-annotator learning traditionally aggregates diverse annotations to approximate a single “ground truth”, treating disagreements as noise. However, this paradigm faces fundamental challenges: subjective tasks often lack absolute ground truth, and sparse annotation coverage makes aggregation statistically unreliable. We introduce a paradigm shift from sample-wise aggregation to annotator-wise behavior modeling. By treating annotator disagreements as valuable information rather than noise, modeling annotator-specific behavior patterns can reconstruct unlabeled data to reduce annotation cost, enhance aggregation reliability, and explain annotator decision behavior. To this end, we propose QuMAB (Query-based Multi-Annotator Behavior Pattern Learning), which uses lightweight queries to model individual annotators while capturing inter-annotator correlations as implicit regularization, preventing overfitting to sparse individual data while maintaining individualization and improving generalization, with a visualization of annotator focus regions offering an explainable analysis of behavior understanding. We contribute two large-scale datasets with dense per-annotator labels: STREET (4,300 labels/annotator) and AMER (average 3,118 labels/annotator), the first multimodal multi-annotator dataset. Extensive experiments demonstrate the superiority of our QuMAB in modeling individual annotators’ behavior patterns, their utility for consensus prediction, and applicability under sparse annotations.

Popularity ranking of Internet services is an important metric for network operators, because it enables mid- to-long term planning of their network facilities and root cause analysis for unexpected traffic. The service-oriented traffic monitoring is much helpful to infer the popularity, hence it has been gathering much attention from both researchers and practitioners. Lately, service identification of a given flow has become very difficult due to the rapid growth of CDNs and/or encrypted traffic, while some research works employed preceding DNS traffic as a hint. However, because of its cache mechanism, the DNS message count deviates from the actual number of flows, which can greatly degrade the ranking reliability. We propose a theoretical model for inferring the user's number of accesses per domain name by exploiting the characteristics of the DNS message count. To the best of our knowledge, this paper is the first attempt to formulate the effect of user's stub resolvers; previous studies were focused on analyzing the effect of cache servers. We evaluated the precision of our model with a real dataset of traffic of thousands of users. By analyzing the top-50 domain names by the number of users, we can infer the number of flows within a 24% error rate on average in 42 out of 50 FQDNs.