Crowdsourcing has been playing an essential role in machine learning since it can obtain a large number of labels in an economical and fast manner for training increasingly complex learning models. However, the application of crowdsourcing learning still faces several challenges such as the low quality of crowd labels and the urgent requirement for learning models adapting to the label noises. There have been many studies focusing on truth inference algorithms to improve the quality of labels obtained by crowdsourcing. Comparably, end-to-end predictive model learning in crowdsourcing scenarios, especially using cutting-edge deep learning techniques, is still in its infant stage. In this paper, we propose a novel graph convolutional network-based framework, namely CGNNAT, which models the correlation of instances by combining the GCN model with an attention mechanism to learn more representative node embeddings for a better understanding of the bias tendency of crowd workers. Furthermore, a specific projection processing layer is employed in CGNNAT to model the reliability of each crowd worker, which makes the model an end-to-end neural network directly trained by noisy crowd labels. Experimental results on several real-world and synthetic datasets show that the proposed CGNNAT outperforms state-of-the-art and classical methods in terms of label prediction.

Crowdsourcing provides a cost-effective labeling solution for the acquisition of labeled training samples for machine learning by employing workers on the Internet. A common approach to improving the label quality is to employ a truth inference method to infer integrated labels for samples from their multiple noisy labels obtained from different crowd workers. Although the quality of integrated labels is significantly improved compared with that of the original noisy ones, it still cannot completely eliminate the noises inevitably existing in the integrated labels. To further improve the label quality, this paper proposes a novel label noise correction method for crowdsourcing based on dynamic resampling (DRNC). DRNC first divides the dataset with inferred labels into a clean set and a noisy set through a filter. According to a certain proportion, the clean set and the noisy set are resampled to train multiple heterogeneous classifiers, which form an ensemble classifier . Then, the dataset is divided by the ensemble classifier into a new sub-noisy set and a sub-clean set. The whole process repeats multiple rounds, generating multiple sub-clean sets. Finally, these sub-clean sets are used to train classifiers, which jointly correct the wrong labels in the dataset by voting. Experimental results on 25 simulated and 4 real-world datasets consistently show that the proposed DRNC averagely improves the quality of labels as well as the quality of learned models in the range of 1 to 10 percentage points, compared with four state-of-the-art crowdsourcing noise correction methods.

Various correlations hidden in crowdsourcing annotation tasks bring opportunities to further improve the accuracy of label aggregation. However, these relationships are usually extremely difficult to be modeled. Most existing methods can merely make use of one or two correlations. In this paper, we propose a novel graph neural network model, namely LAGNN, which models five different correlations in crowdsourced annotation tasks by utilizing deep graph neural networks with convolution operations and derives a high label aggregation performance. Utilizing the group of high quality workers through labeling similarity, LAGNN can efficiently revise the preference among workers. Moreover, by injecting a little ground truth in its training stage, the label aggregation performance of LAGNN can be further significantly improved. We evaluate LAGNN on a large number of simulated datasets generated through varying six degrees of freedom and on eight real-world crowdsourcing datasets in both supervised and unsupervised (agnostic) modes. Experiments on data leakage is also contained. Experimental results consistently show that the proposed LAGNN significantly outperforms six state-of-the-art models in terms of label aggregation accuracy.

Multilabel annotation is a critical step to generate training sets when learning classification models in various application domains, but asking domain experts to provide labels is usually time-consuming and expensive, which cannot meet the current requirement of the fast evolution of the models in the big data era. Although crowdsourcing provides a fast solution to acquire labels for multilabel learning, it faces the risk of high data acquisition cost and low label quality. This article proposes a novel one-coin label-dependent active crowdsourcing (OCLDAC) method to iteratively query noisy labels from crowd workers and learn multilabel classification models. In each iteration of active learning, integrated labels of instances are first inferred by a novel one-coin label-dependent model, which utilizes a mixture of multiple independent Bernoulli distributions to explore and exploit correlations among the labels to increase the accuracy of truth inference. Then, instances, labels, and workers are selected according to the novel strategies that incorporate the distribution of noisy labels, the prediction probability of learning models, label correlations, and the reliability of crowd workers. Simulations on eight multilabel datasets and evaluation on one real-world crowdsourcing dataset consistently show that the proposed OCLDAC significantly outperforms the state-of-the-art methods and their variants.

As a representative classification model, decision tree has been extensively applied in data mining. It generates a series of if-then rules based on the homogeneity of class distribution. In a society where data spreads everywhere for knowledge discovery, the privacy of the data respondents is likely to be leaked and abused. Based on this concern, we propose an overview of the rapidly evolving research results focusing on privacy preserving decision tree induction. The research results are summarised according to the characteristics of related privacy preservation techniques, which include data perturbation, cryptography, and data anonymisation. In addition, we demonstrate the comparison between the merits and demerits of these methods considering the specific property of decision tree induction. At last, we conclude the future trend of privacy preserving techniques.

Traditional learning from crowdsourced labeled data consists of two stages: inferring true labels for instances from their multiple noisy labels and building a learning model using these instances with the inferred labels. This straightforward two-stage learning scheme suffers from two weaknesses: (1) the accuracy of inference may be very low; (2) useful information may be lost during inference. In this paper, we proposed a novel ensemble method for learning from crowds. Our proposed method is a meta-learning scheme. It first uses a bootstrapping process to create M sub-datasets from an original crowdsourced labeled dataset. For each sub-dataset, each instance is duplicated with different weights according to the distribution and class memberships of its multiple noisy labels. A base classifier is then trained from this extended sub-dataset. Finally, unlabeled instances are predicted by aggregating the outputs of these M base classifiers. Because the proposed method gets rid of the inference procedure and uses the full dataset to train learning models, it preserves the useful information for learning as much as possible. Experimental results on nine simulated and two real-world crowdsourcing datasets consistently show that the proposed ensemble learning method significantly outperforms five state-of-the-art methods.