Label completion serves as a preprocessing approach to handling the sparse crowdsourced label matrix problem, significantly boosting the effectiveness of the downstream label aggregation. In recent advances, worker modeling has been proved to be a powerful strategy to further improve the performance of label completion. However, in real-world scenarios, workers typically annotate only a few instances, leading to insufficient worker modeling and thus limiting the improvement of label completion. To address this issue, we propose a novel transfer learning-based label completion (TLLC) method. Specifically, we first identify all high-confidence instances from the whole crowdsourced data as a source domain and use it to pretrain a Siamese network. The abundant annotated instances in the source domain provide essential knowledge for worker modeling. Then, we transfer the pretrained network to the target domain with the instances annotated by each worker separately, ensuring worker modeling captures unique characteristics of each worker. Finally, we leverage the new embeddings learned by the transferred network to complete each worker’s missing labels. Extensive experiments on several widely used real-world datasets demonstrate the effectiveness of TLLC. Our codes and datasets are available at https://github.com/jiangliangxiao/TLLC.

In real-world crowdsourcing scenarios, most workers often annotate a few instances only, which results in a significantly sparse crowdsourced label matrix and subsequently harms the performance of label integration algorithms. Recent work called worker similarity-based label completion (WSLC) has been proven to be an effective algorithm to addressing this issue. However, WSLC considers solely the correlation of the labels annotated by different workers on per individual instance while totally ignoring the correlation of the labels annotated by different workers among similar instances. To fill this gap, we propose a novel label distribution propagation-based label completion (LDPLC) algorithm. At first, we use worker similarity weighted majority voting to initialize a label distribution for each missing label. Then, we design a label distribution propagation algorithm to enable each missing label of each instance to iteratively absorb its neighbors’ label distributions. Finally, we complete each missing label based on its converged label distribution. Experimental results on both real-world and simulated crowdsourced datasets show that LDPLC significantly outperforms WSLC in enhancing the performance of label integration algorithms. Our codes and datasets are available at https://github.com/jiangliangxiao/LDPLC.

In crowdsourcing scenarios, we can obtain multiple noisy labels for an instance from crowd workers and then aggregate these labels to infer the unknown true label of this instance. Due to the lack of expertise of workers, obtained labels usually contain a degree of noise. Existing studies usually focus on the crowdsourcing scenarios with low noise ratios but rarely focus on the crowdsourcing scenarios with high noise ratios. In this paper, we focus on the crowdsourcing scenarios with high noise ratios and propose a novel label aggregation algorithm called enhanced label distribution propagation (ELDP). First, ELDP harnesses an internal worker weighting method to estimate the weights of workers and then performs the first label distribution enhancement. Then, for instances not covered in the first enhancement, ELDP performs the second enhancement using a class membership estimation method based on the intra-cluster distance. Finally, ELDP propagates enhanced label distributions from accurately enhanced instances to inaccurately enhanced instances. Experimental results on both simulated and real-world crowdsourced datasets show that ELDP significantly outperforms all the other state-of-the-art label aggregation algorithms.

In real-world crowdsourcing scenarios, it is a common phenomenon that each worker only annotates a few instances, resulting in a significantly sparse crowdsourcing label matrix. Consequently, only a small number of workers influence the inferred integrated label of each instance, which may weaken the performance of label integration algorithms. To address this problem, we propose a novel label completion algorithm called Worker Similarity-based Label Completion (WSLC). WSLC is grounded on the assumption that workers with similar cognitive abilities will annotate similar labels on the same instances. Specifically, we first construct a data set for each worker that includes all instances annotated by this worker and learn a feature vector for each worker. Then, we define a metric based on cosine similarity to estimate worker similarity based on the learned feature vectors. Finally, we complete the labels for each worker on unannotated instances based on the worker similarity and the annotations of similar workers. The experimental results on one real-world and 34 simulated crowdsourced data sets consistently show that WSLC effectively addresses the problem of the sparse crowdsourcing label matrix and enhances the integration accuracies of label integration algorithms.

In recommender systems, exploring user intents allows for a better understanding and exploration of user preferences, thereby improving recommendation performance. However, existing methods for modeling user intents often do so by statically setting the intent count, which can result in redundancy or insufficiency in capturing the full range of user intents. In order to solve this problem, this paper proposes a model named I ntent-aware Recommendation Based on P rincipal C omponent A nalysis (Intent PCA). Intent PCA is a novel application of PCA in the field of recommender systems. This model defines intents as users’ preferences for some specific relations shown on a knowledge graph, and constructs a user-relation matrix to calculate users’ preferences for relations. Then PCA is applied on the user-relation matrix to extract user intents. Benefit from good characteristics of PCA, our PCA-based user intent extraction model can comprehensively model user intents while simultaneously avoid intent redundancy. Moreover, by combining the user intents, this paper designs an intent-based information propagation method to differentially aggregate information from surrounding neighbor nodes. Experiments conducted on three datasets validate the effectiveness of the proposed Intent PCA model.

In recent years, a large number of algorithms for label integration and noise correction have been proposed to infer the unknown true labels of instances in crowdsourcing. They have made great advances in improving the label quality of crowdsourced datasets. However, due to the presence of intractable instances, these algorithms are usually not as significant in improving the model quality as they are in improving the label quality. To improve the model quality, this paper proposes an instance weighting-based bias-variance trade-off (IWBVT) approach. IWBVT at first proposes a novel instance weighting method based on the complementary set and entropy, which mitigates the impact of intractable instances and thus makes the bias and variance of trained models closer to the unknown true results. Then, IWBVT performs probabilistic loss regressions based on the bias-variance decomposition, which achieves the bias-variance trade-off and thus reduces the generalization error of trained models. Experimental results indicate that IWBVT can serve as a universal post-processing approach to significantly improving the model quality of existing state-of-the-art label integration algorithms and noise correction algorithms.

To reduce annotation costs, it is common in crowdsourcing to collect only a few noisy labels from different crowd workers for each instance. However, the limited noisy labels restrict the performance of label integration algorithms in inferring the unknown true label for the instance. Recent works have shown that leveraging neighbor instances can help alleviate this problem. Yet, these works all assume that each instance has the same neighborhood size, which defies common sense. To address this gap, we propose a novel label integration algorithm called K-free nearest neighbor (KFNN). In KFNN, the neighborhood size of each instance is automatically determined based on its attributes and noisy labels. Specifically, KFNN initially estimates a Mahalanobis distance distribution from the attribute space to model the relationship between each instance and all classes. This distance distribution is then utilized to enhance the multiple noisy label distribution of each instance. Subsequently, a Kalman filter is designed to mitigate the impact of noise incurred by neighbor instances. Finally, KFNN determines the optimal neighborhood size by the max-margin learning. Extensive experimental results demonstrate that KFNN significantly outperforms all the other state-of-the-art algorithms and exhibits greater robustness in various crowdsourcing scenarios.

In crowdsourcing scenarios, we can obtain multiple noisy labels from different crowd workers on the Internet for each instance and then infer its unknown true label via a label integration method. However, noisy labels often have a serious negative impact on label integration. In this case, most existing works always focus on designing more complex label integration methods to infer unknown true labels more accurately from multiple noisy labels, but little attention has been paid to another perspective, i.e., purifying noisy labels before label integration. In this paper, we aim to purify noisy labels for existing label integration methods and propose a label consistency-based worker filtering (LCWF) algorithm. In LCWF, we consider that if all low-quality workers are filtered out and only high-quality workers remain, the label consistency should be high. Therefore, we utilize label consistency to filter out low-quality workers. Firstly, we directly transform the worker filtering problem into a discrete optimization problem and utilize label consistency to define the fitness function for this problem. Then, we search for the optimal solution to this problem by a genetic algorithm. Finally, we filter out all labels from low-quality workers according to the optimal solution we obtained. Experimental results on simulated and real-world datasets demonstrate that LCWF can effectively purify noisy labels and improve the integration accuracy of existing label integration methods.

Crowdsourcing services provide a fast and cheap way to annotate instances by employing crowd workers on the Internet. As a result, many learning from crowds (LFC) methods have been proposed in recent years. However, almost all these methods assume that all instances are benign, which makes them vulnerable to adversarial attacks. To improve the model's robustness to adversarial attacks, adversarial LFC (A-LFC) has attracted remarkable attention. A-LFC iteratively updates the true labels' estimations and the trained model by adversarial learning and uses the model's predictions in turn to help estimate the true labels. In A-LFC, to the best of our knowledge, the true labels' estimations and the model's predictions are inaccurate and the stopping condition for iterations is rough, which limit the model's performance. To further improve A-LFC, this paper proposes weighted A-LFC (WA-LFC). To reduce the impact of misinformation in the true labels' estimations and model's predictions, our method weights instances for adversarial learning and weights the model's predictions for estimating the true labels. Our method iteratively updates these weights and uses instance-weighted cross-entropy loss to decide when the iterative process should be stopped. Experiments on three real-world datasets show that our method substantially improves the trained model's performance. On average, the test accuracy of our method outperforms that of A-LFC by 10.06% and 27.55% in white-box and black-box attack settings, respectively.

Naive Bayes (NB) continues to be one of the top 10 data mining algorithms due to its simplicity, efficiency and efficacy. Numerous enhancements have been proposed to weaken its attribute conditional independence assumption. However, all of them only focus on the raw attribute view, which is hard to reflect all the data characteristics in real-world applications. To portray data characteristics more comprehensively, in this study, we construct two label views from the raw attributes and propose a novel model called multi-view attribute weighted naive Bayes (MAWNB). In MAWNB, we first build multiple super-parent one-dependence estimators (SPODEs) as well as random trees (RTs), then we utilize each of them to classify each training instance in turn and use all their predicted class labels to construct two label views. Next, to avoid attribute redundancy, we optimize the weight of each attribute value for each class by minimizing the negative conditional log-likelihood (CLL) in each view. Finally, the estimated class-membership probabilities by three views are fused to predict the class label for each test instance. Extensive experiments show that MAWNB significantly outperforms NB and all the other existing state-of-the-art competitors.

In crowdsourcing scenarios, we can often obtain each instance's multiple noisy labels from different crowd workers and then use a label integration method to infer its integrated label. In spite of the effectiveness of label integration methods, a certain level of label noise still exists in integrated labels. To reduce the impact of label noise, noise correction has attracted much attention from researchers, and therefore a certain number of noise correction methods were proposed in recent years. Among them, Between-class Margin-based Noise Correction (BMNC) has demonstrated remarkable denoising performance. However, BMNC can only handle binary classification tasks. For multiclass classification tasks, we propose an effective but very simple noise correction method in this paper. We refer to our proposed method as Label Distribution-based Noise Correction (LDNC). At first, LDNC transforms each instance's multiple noisy labels into a label distribution. Then, LDNC uses the margin between the first- and second-largest label probabilities in the label distribution to identify and filter each possible noise instance and thus obtains a clean set and a noise set. Finally, LDNC builds a classifier on the clean set to relabel all instances in the noise set. The experimental results on 16 simulated and one real-world multiclass crowdsourced data sets show that LDNC significantly outperforms all the other existing state-of-the-art noise correction methods.

Crowdsourcing services provide a fast, efficient, and cost-effective way to obtain large labeled data for supervised learning. Unfortunately, the quality of crowdsourced labels cannot satisfy the standards of practical applications. Ground-truth inference, simply called label integration, designs proper aggregation methods to infer the unknown true label of each instance (sample) from the multiple noisy label set provided by ordinary crowd labelers (workers). However, nearly all existing label integration methods focus solely on the multiple noisy label set per individual instance while totally ignoring the intercorrelation among multiple noisy label sets of different instances. To solve this problem, a multiple noisy label distribution propagation (MNLDP) method is proposed in this article. MNLDP at first estimates the multiple noisy label distribution of each instance from its multiple noisy label set and then propagates its multiple noisy label distribution to its nearest neighbors. Consequently, each instance absorbs a fraction of the multiple noisy label distributions from its nearest neighbors and yet simultaneously maintains a fraction of its own original multiple noisy label distribution. Empirical studies on a collection of an artificial dataset, six simulated UCI datasets, and three real-world crowdsourced datasets show that MNLDP outperforms all other existing state-of-the-art label integration methods in terms of the integration accuracy and classification accuracy.

In the printed circuit board (PCB) industry, cosmetic defect detection is an essential process to ensure product quality. However, existing PCB cosmetic defect detection approaches have a high false alarm rate, which lead to expensive labor costs of manual confirmation. To solve this problem, some traditional machine learning-based approaches have been proposed, but they just utilize hand-crafted features to build classifiers and thus are rough and sub-optimal. Recently, due to its powerful capability in automatic feature extraction, convolutional neural network (CNN) has been widely used in PCB cosmetic defect detection. However, few of them pay attention to the imbalanced class distribution as well as the different misclassification costs of real and pseudo defects, both of which are common problems in the PCB industry. To this end, in this study, we propose a novel model called cost-sensitive residual convolutional neural network (CS-ResNet) by adding a cost-sensitive adjustment layer in the standard ResNet. Specifically, we assign larger weights to minority real defects based on the class-imbalance degree and then optimize CS-ResNet by minimizing the weighted cross-entropy loss function. We conducted a series of experiments by comparing CS-ResNet with the standard ResNet, state-of-the-art CNN-based approach Auto-VRS and traditional machine learning-based approach HOG-SVM on a real-world PCB cosmetic defect dataset. Experimental results show that CS-ResNet achieves the highest Sensitivity (0.89), G-mean (0.91) and the lowest misclassification costs.

Crowdsourcing services provide an economic and efficient means of acquiring multiple noisy labels for each training instance in supervised learning. Ground truth inference methods, also known as consensus methods, are then used to obtain the integrated labels of training instances. Although consensus methods are effective, there still exists a level of noise in the set of integrated labels. Therefore, it is necessary to handle noise in the integrated labels to improve label and model quality. In this paper, we propose a resampling-based noise correction method (simply RNC). Different from previous label noise correction methods for crowdsourcing, RNC first employs a filter to obtain a clean set and a noisy set and then repeatedly resamples the clean and noisy sets several times according to a certain proportion. Finally, multiple classifiers built on the resampled data sets are used to re-label the training data. Experimental results based on 18 simulated data sets and five real-world data sets demonstrate that RNC rarely degrades the label and model quality compared to other three state-of-the-art noise correction methods and, in many cases, improves quality dramatically.

Feature selection is an optional preprocessing procedure and is frequently used to improve the classification accuracy of a machine learning algorithm by removing irrelevant and/or redundant features. However, in many real-world applications, the test cost is also required for making optimal decisions, in addition to the classification accuracy. To the best of our knowledge, thus far, few studies have been conducted on test-cost-sensitive feature selection (TCSFS). In TCSFS, the objectives are twofold: 1) to improve the classification accuracy and 2) to decrease the test cost. Therefore, in fact, it constitutes a multiobjective optimization problem. In this paper, we transformed this multiobjective optimization problem into a single-objective optimization problem by utilizing a new evaluation function and in this paper, we propose a new general wrapper framework for TCSFS. Specifically, in our proposed framework, we add a new term to the evaluation function of a wrapper feature selection method so that the test cost of measuring features is taken into account. We experimentally tested our proposed framework, using 36 classification problems from the University of California at Irvine (UCI) repository, and compared it to some other state-of-the-art feature selection frameworks. The experimental results showed that our framework allows users to select an optimal feature subset with the minimal test cost, while simultaneously maintaining a high classification accuracy.

Due to its simplicity, efficiency, and efficacy, naive Bayes (NB) has continued to be one of the top 10 algorithms in the data mining and machine learning community. Of numerous approaches to alleviating its conditional independence assumption, feature weighting has placed more emphasis on highly predictive features than those that are less predictive. In this paper, we argue that for NB highly predictive features should be highly correlated with the class (maximum mutual relevance), yet uncorrelated with other features (minimum mutual redundancy). Based on this premise, we propose a correlation-based feature weighting (CFW) filter for NB. In CFW, the weight for a feature is a sigmoid transformation of the difference between the feature-class correlation (mutual relevance) and the average feature-feature intercorrelation (average mutual redundancy). Experimental results show that NB with CFW significantly outperforms NB and all the other existing state-of-the-art feature weighting filters used to compare. Compared to feature weighting wrappers for improving NB, the main advantages of CFW are its low computational complexity (no search involved) and the fact that it maintains the simplicity of the final model. Besides, we apply CFW to text classification and have achieved remarkable improvements.

Handling text data is a challenge for machine learning because text data is high dimensional in many cases. Feature selection has been approved to be an effective approach to handle high-dimensional data. Feature selection approaches can be broadly divided into two categories: filter approaches and wrapper approaches. Generally, wrapper approaches have superior accuracy compared to filters, but filters always run faster than wrapper approaches. In order to integrate the advantages of filter approaches and wrapper approaches, we propose a gain ratio-based hybrid feature selection approach to naive Bayes text classifiers. The hybrid feature selection approach uses base classifiers to evaluate feature subsets like wrapper approaches, but it need not repeatedly search feature subsets and build base classifiers. The experimental results on large suite of benchmark text datasets show that the proposed hybrid feature selection approach significantly improves the classification accuracy of the original naive Bayes text classifiers while does not incur the high time complexity that characterizes wrapper approaches.

In many real-world applications, it is often the case that the class distribution of instances is imbalanced and the costs of misclassification are different. Thus, the class-imbalanced cost-sensitive learning has attracted much attention from researchers. Sampling is one of the widely used techniques in dealing with the class-imbalance problem, which alters the class distribution of instances so that the minority class is well represented in the training data. In this paper, we propose a novel Minority Cloning Technique (MCT) for class-imbalanced cost-sensitive learning. MCT alters the class distribution of training data by cloning each minority class instance according to the similarity between it and the mode of the minority class. The experimental results on a large number of UCI datasets show that MCT performs much better than Minority Oversampling with Replacement Technique (MORT) and Synthetic Minority Oversampling TEchnique (SMOTE) in terms of the total misclassification costs of the built classifiers.

A high quality distance function that measures the difference between instances is essential in many real-world applications and research fields. For example, in instance-based learning, the distance function plays the most important role. A large number of distance functions have been proposed. For nominal attributes, Value Difference Metric (VDM) is one of the state-of-the-art and widely used distance functions. However, it needs to estimate the conditional probabilities, which drops its efficiency in computing the distance between instances. Besides, a practical issue that arises in estimating the conditional probabilities is that the denominators can be zero or very small. This makes them either undefined or very large. Therefore, an efficient distance function that can measure the difference between two instances but without the practical issue confronting VDM is desirable. In this paper, we propose a novel distance function: Frequency Difference Metric (FDM). FDM is just based on the joint frequencies of class labels and attribute values, instead of the conditional probabilities. Extensive empirical studies show that FDM performs almost as well as VDM in terms of accuracy, but significantly outperforms VDM in terms of efficiency. This work provides a very simple, efficient, and effective distance function that can be widely used in many real-world applications and research fields.

In many real-world applications, it is often the case that the class distribution of instances is imbalanced and the costs of misclassification are different. Thus, class-imbalance and cost-sensitive learning have attracted much attention from researchers. Sampling is one of the widely used approaches in dealing with the class imbalance problem, which alters the class distribution of instances so that the minority class is well represented in the training data. In this paper, we study the effect of sampling the natural training data on state-of-the-art Bayesian network classifiers, such as Naive Bayes (NB), Tree Augmented Naïve Bayes (TAN), Averaged One-Dependence Estimators (AODE), Weighted Average of One-Dependence Estimators (WAODE), and Hidden naive Bayes (HNB) and propose sampled Bayesian network classifiers. Our experimental results on a large number of UCI datasets show that our sampled Bayesian network classifiers perform much better than the ones trained from the natural training data especially when the natural training data is highly imbalanced and the cost ratio is high enough.

Classification is an important task in data mining, while accurate class probability estimation is also desirable in real-world applications. Some probability-based classifiers, such as the k-nearest neighbor algorithm (KNN) and its variants, can estimate the class membership probabilities of the test instance. Unfortunately, a good classifier is not always a good class probability estimator. In this paper, we try to improve the class probability estimation performance of KNN and its variants. As we all know, KNN and its variants are all of the distance-related algorithms and their performance is closely related to the used distance metric. Value Difference Metric (VDM) is one of the widely used distance metrics for nominal attributes. Thus, in order to scale up the class probability estimation performance of the distance-related algorithms such as KNN and its variants, we propose an Attribute Weighted Value Difference Metric (AWVDM) in this paper. AWVDM uses the mutual information between the attribute variable and the class variable to weight the difference between two attribute values of each pair of instances. Experimental results on 36 UCI benchmark datasets validate the effectiveness of the proposed AWVDM.

A large number of distance metrics have been proposed to measure the difference of two instances. Among these metrics, Short and Fukunaga metric (SFM) and minimum risk metric (MRM) are two probability-based metrics which are widely used to find reasonable distance between each pair of instances with nominal attributes only. For simplicity, existing works use naive Bayesian (NB) classifiers to estimate class membership probabilities in SFM and MRM. However, it has been proved that the ability of NB classifiers to class probability estimation is poor. In order to scale up the classification performance of NB classifiers, many augmented NB classifiers are proposed. In this paper, we study the class probability estimation performance of these augmented NB classifiers and then use them to estimate the class membership probabilities in SFM and MRM. The experimental results based on a large number of University of California, Irvine (UCI) data-sets show that using these augmented NB classifiers to estimate the class membership probabilities in SFM and MRM can significantly enhance their generalisation ability.

Traditionally, the performance of a classifier is measured by its classification accuracy or error rate. In fact, probability-based classifiers also produce the class probability estimation (the probability that a test instance belongs to the predicted class). This information is often ignored in classification, as long as the class with the highest class probability estimation is identical to the actual class. In many data mining applications, however, classification accuracy and error rate are not enough. For example, in direct marketing, we often need to deploy different promotion strategies to customers with different likelihood (class probability) of buying some products. Thus, accurate class probability estimations are often required to make optimal decisions. In this paper, we firstly review some state-of-the-art probability-based classifiers and empirically investigate their class probability estimation performance. From our experimental results, we can draw a conclusion: C4.4 is an attractive algorithm for class probability estimation. Then, we present a locally weighted version of C4.4 to scale up its class probability estimation performance by combining locally weighted learning with C4.4. We call our improved algorithm locally weighted C4.4, simply LWC4.4. We experimentally test LWC4.4 using the whole 36 UCI data sets selected by Weka. The experimental results show that LWC4.4 significantly outperforms C4.4 in terms of class probability estimation.