Label completion serves as a preprocessing approach to handling the sparse crowdsourced label matrix problem, significantly boosting the effectiveness of the downstream label aggregation. In recent advances, worker modeling has been proved to be a powerful strategy to further improve the performance of label completion. However, in real-world scenarios, workers typically annotate only a few instances, leading to insufficient worker modeling and thus limiting the improvement of label completion. To address this issue, we propose a novel transfer learning-based label completion (TLLC) method. Specifically, we first identify all high-confidence instances from the whole crowdsourced data as a source domain and use it to pretrain a Siamese network. The abundant annotated instances in the source domain provide essential knowledge for worker modeling. Then, we transfer the pretrained network to the target domain with the instances annotated by each worker separately, ensuring worker modeling captures unique characteristics of each worker. Finally, we leverage the new embeddings learned by the transferred network to complete each worker’s missing labels. Extensive experiments on several widely used real-world datasets demonstrate the effectiveness of TLLC. Our codes and datasets are available at https://github.com/jiangliangxiao/TLLC.

Due to its simplicity, effectiveness and robustness, naive Bayes (NB) has continued to be one of the top 10 data mining algorithms. To improve its performance, a large number of improved algorithms have been proposed in the last few decades. However, in addition to Gaussian naive Bayes (GNB), there is little work on numerical attributes. At the same time, none of them takes into account the correlations among instances. To fill this gap, we propose a novel algorithm called instance correlation graph-based naive Bayes (ICGNB). Specifically, it first uses original attributes to construct an instance correlation graph (ICG) to represent the correlations among instances. Then, it employs a variational graph auto-encoder (VGAE) to generate new attributes from the constructed ICG and uses them to augment original attributes. Finally, it weights each augmented attribute to alleviate the attribute redundancy and builds GNB on the weighted attributes. The experimental results on tens of datasets show that ICGNB significantly outperforms its deserved competitors.Our codes and datasets are available at https://github.com/jiangliangxiao/ICGNB.

In real-world crowdsourcing scenarios, most workers often annotate a few instances only, which results in a significantly sparse crowdsourced label matrix and subsequently harms the performance of label integration algorithms. Recent work called worker similarity-based label completion (WSLC) has been proven to be an effective algorithm to addressing this issue. However, WSLC considers solely the correlation of the labels annotated by different workers on per individual instance while totally ignoring the correlation of the labels annotated by different workers among similar instances. To fill this gap, we propose a novel label distribution propagation-based label completion (LDPLC) algorithm. At first, we use worker similarity weighted majority voting to initialize a label distribution for each missing label. Then, we design a label distribution propagation algorithm to enable each missing label of each instance to iteratively absorb its neighbors’ label distributions. Finally, we complete each missing label based on its converged label distribution. Experimental results on both real-world and simulated crowdsourced datasets show that LDPLC significantly outperforms WSLC in enhancing the performance of label integration algorithms. Our codes and datasets are available at https://github.com/jiangliangxiao/LDPLC.

In crowdsourcing scenarios, we can obtain multiple noisy labels for an instance from crowd workers and then aggregate these labels to infer the unknown true label of this instance. Due to the lack of expertise of workers, obtained labels usually contain a degree of noise. Existing studies usually focus on the crowdsourcing scenarios with low noise ratios but rarely focus on the crowdsourcing scenarios with high noise ratios. In this paper, we focus on the crowdsourcing scenarios with high noise ratios and propose a novel label aggregation algorithm called enhanced label distribution propagation (ELDP). First, ELDP harnesses an internal worker weighting method to estimate the weights of workers and then performs the first label distribution enhancement. Then, for instances not covered in the first enhancement, ELDP performs the second enhancement using a class membership estimation method based on the intra-cluster distance. Finally, ELDP propagates enhanced label distributions from accurately enhanced instances to inaccurately enhanced instances. Experimental results on both simulated and real-world crowdsourced datasets show that ELDP significantly outperforms all the other state-of-the-art label aggregation algorithms.

In real-world crowdsourcing scenarios, it is a common phenomenon that each worker only annotates a few instances, resulting in a significantly sparse crowdsourcing label matrix. Consequently, only a small number of workers influence the inferred integrated label of each instance, which may weaken the performance of label integration algorithms. To address this problem, we propose a novel label completion algorithm called Worker Similarity-based Label Completion (WSLC). WSLC is grounded on the assumption that workers with similar cognitive abilities will annotate similar labels on the same instances. Specifically, we first construct a data set for each worker that includes all instances annotated by this worker and learn a feature vector for each worker. Then, we define a metric based on cosine similarity to estimate worker similarity based on the learned feature vectors. Finally, we complete the labels for each worker on unannotated instances based on the worker similarity and the annotations of similar workers. The experimental results on one real-world and 34 simulated crowdsourced data sets consistently show that WSLC effectively addresses the problem of the sparse crowdsourcing label matrix and enhances the integration accuracies of label integration algorithms.

Reservoir characterization (identification and parameter estimation) is critical for gas hydrate exploration and development. While machine learning techniques excel at capturing complex relationships in well-logging reservoir characterization, existing research mainly focuses on end-to-end supervised learning approaches relying on costly labeled data and lacking multi-task learning capabilities. In this paper, we introduce a self-supervised learning (SSL) framework to learn general representations of large-scale unlabeled well-logging intervals from sandy, silty, and clayey hydrate reservoirs via masked autoencoders (WirMAE). By incorporating channel-based attention mechanisms and a masked reconstruction pre-training strategy for variable tokens, WirMAE effectively extracts intrinsic multivariate correlations within logging data, enabling the generation of various missing log curves. The model is validated on data from globally distributed hydrate with diverse accumulation patterns. Compared to supervised deep learning methods and classical machine learning models, Fine-turned WirMAE achieves superior reservoir identification accuracy (average F1-score: 0.864) using only 1% labeled data in complex geological settings. Combined with domain expertise, WirMAE yields precise estimation of key reservoir parameters such as hydrate saturation and permeability, outperforming conventional petrophysical methods. Additionally, embedding visualizations and attention analyses reveal the inner workings of the model and its consistency with expert-driven geological interpretations. Our findings highlight the potential of SSL for advancing more accurate and transparent intelligent reservoir characterization using well-log data, indicating that the application of WirMAE could be extended to broader hydrocarbon reservoirs in the future.

In recommender systems, exploring user intents allows for a better understanding and exploration of user preferences, thereby improving recommendation performance. However, existing methods for modeling user intents often do so by statically setting the intent count, which can result in redundancy or insufficiency in capturing the full range of user intents. In order to solve this problem, this paper proposes a model named I ntent-aware Recommendation Based on P rincipal C omponent A nalysis (Intent PCA). Intent PCA is a novel application of PCA in the field of recommender systems. This model defines intents as users’ preferences for some specific relations shown on a knowledge graph, and constructs a user-relation matrix to calculate users’ preferences for relations. Then PCA is applied on the user-relation matrix to extract user intents. Benefit from good characteristics of PCA, our PCA-based user intent extraction model can comprehensively model user intents while simultaneously avoid intent redundancy. Moreover, by combining the user intents, this paper designs an intent-based information propagation method to differentially aggregate information from surrounding neighbor nodes. Experiments conducted on three datasets validate the effectiveness of the proposed Intent PCA model.

Crowdsourcing services provide a fast and cheap way to obtain substantial labeled data by employing crowd workers on the Internet. In crowdsourcing learning, two-stage methods have been widely used, which first infer the integrated label for each instance and then build a learning model using instances with their integrated labels. However, existing two-stage methods mainly focus on how to infer more accurate integrated labels, after that, most of them directly regard the integrated labels as class labels to build a learning model, which loses the detailed worker labeling information in multiple noisy labels and thus results in sub-optimal model accuracy. To solve this problem, in this study, we take the multiple noisy labels of each instance as its attribute value vector to construct another view in addition to the original attribute view, and propose a novel two-stage method called dual-view learning from crowds (DVLFC). In DVLFC, we first pick out workers with sufficient number of labels and augment the multiple noisy label set for each instance, then we build a supervised learning model in each view and at last we fuse their class-membership probabilities to get the final classification result. Extensive experiments on both real-world and artificial crowdsourced datasets prove the effectiveness of DVLFC.

In crowdsourcing scenarios, we can obtain each instance's multiple noisy labels from different crowd workers and then infer its unknown ground truth via a ground truth inference method. However, to the best of our knowledge, the existing ground truth inference methods always attempt to aggregate multiple noisy labels into a single consensus label as the ground truth. In this article, we aim to explore a new strategy, i.e., label selection, which directly selects the label of the highest quality worker as the ground truth. To this end, we propose a label consistency-based ground truth inference (LCGTI) method. In LCGTI, we argue that high-quality workers should have a low bias with other workers in labeling the same instances and a low variance with themselves in labeling similar instances. To estimate the bias, we calculate the label consistency of different workers on the same instances. To estimate the variance, we calculate the label consistency of the same worker on similar instances. Finally, we combine these two components to calculate the labeling quality of each worker on the inferred instance and perform label selection instead of label aggregation to achieve inference. The experimental results on 34 simulated and two real-world datasets show that LCGTI significantly outperforms all the other state-of-the-art label aggregation-based ground truth inference methods.

Crowdsourcing provides a cost-effective solution to the problem of obtaining large annotated datasets. In real-world crowdsourcing scenarios, most workers often annotate a few instances only, which results in a significantly sparse crowdsourcing label matrix and subsequently harms the performance of label integration algorithms. Probabilistic matrix factorization (PMF) has been proven to be an effective method for crowdsourcing label completion. However, its low-quality input and output labels limit its performance. To improve its performance, this paper proposes a PMF-based three-stage label completion (PMF-TLC) method. In the first stage, we design a label confidence-based strategy to estimate the quality of each raw label of each worker. Then we flip those low-quality labels in the original crowdsourcing label matrix. In the second stage, we conduct PMF on the flipped label matrix and obtain the completed label matrix with soft labels. In the third stage, we design a between-class margin-based filter to delete those low-quality soft labels in the completed label matrix. Then we convert the remaining high-quality soft labels to hard (logic) labels and obtain the final processed label matrix. Extensive experimental results on real-world and simulated crowdsourced datasets show that PMF-TLC can significantly improve label integration algorithms' performance.

In recent years, a large number of algorithms for label integration and noise correction have been proposed to infer the unknown true labels of instances in crowdsourcing. They have made great advances in improving the label quality of crowdsourced datasets. However, due to the presence of intractable instances, these algorithms are usually not as significant in improving the model quality as they are in improving the label quality. To improve the model quality, this paper proposes an instance weighting-based bias-variance trade-off (IWBVT) approach. IWBVT at first proposes a novel instance weighting method based on the complementary set and entropy, which mitigates the impact of intractable instances and thus makes the bias and variance of trained models closer to the unknown true results. Then, IWBVT performs probabilistic loss regressions based on the bias-variance decomposition, which achieves the bias-variance trade-off and thus reduces the generalization error of trained models. Experimental results indicate that IWBVT can serve as a universal post-processing approach to significantly improving the model quality of existing state-of-the-art label integration algorithms and noise correction algorithms.

To reduce annotation costs, it is common in crowdsourcing to collect only a few noisy labels from different crowd workers for each instance. However, the limited noisy labels restrict the performance of label integration algorithms in inferring the unknown true label for the instance. Recent works have shown that leveraging neighbor instances can help alleviate this problem. Yet, these works all assume that each instance has the same neighborhood size, which defies common sense. To address this gap, we propose a novel label integration algorithm called K-free nearest neighbor (KFNN). In KFNN, the neighborhood size of each instance is automatically determined based on its attributes and noisy labels. Specifically, KFNN initially estimates a Mahalanobis distance distribution from the attribute space to model the relationship between each instance and all classes. This distance distribution is then utilized to enhance the multiple noisy label distribution of each instance. Subsequently, a Kalman filter is designed to mitigate the impact of noise incurred by neighbor instances. Finally, KFNN determines the optimal neighborhood size by the max-margin learning. Extensive experimental results demonstrate that KFNN significantly outperforms all the other state-of-the-art algorithms and exhibits greater robustness in various crowdsourcing scenarios.

In crowdsourcing scenarios, we can obtain multiple noisy labels from different crowd workers on the Internet for each instance and then infer its unknown true label via a label integration method. However, noisy labels often have a serious negative impact on label integration. In this case, most existing works always focus on designing more complex label integration methods to infer unknown true labels more accurately from multiple noisy labels, but little attention has been paid to another perspective, i.e., purifying noisy labels before label integration. In this paper, we aim to purify noisy labels for existing label integration methods and propose a label consistency-based worker filtering (LCWF) algorithm. In LCWF, we consider that if all low-quality workers are filtered out and only high-quality workers remain, the label consistency should be high. Therefore, we utilize label consistency to filter out low-quality workers. Firstly, we directly transform the worker filtering problem into a discrete optimization problem and utilize label consistency to define the fitness function for this problem. Then, we search for the optimal solution to this problem by a genetic algorithm. Finally, we filter out all labels from low-quality workers according to the optimal solution we obtained. Experimental results on simulated and real-world datasets demonstrate that LCWF can effectively purify noisy labels and improve the integration accuracy of existing label integration methods.

In crowdsourcing scenarios, we can obtain multiple noisy labels from different crowd workers for each instance and then infer its integrated label via label integration. To achieve better performance, some recently published label integration methods have attempted to exploit the multiple noisy labels of inferred instances’ nearest neighbors via the K-nearest neighbor (KNN) algorithm. However, the used KNN algorithm searches inferred instances’ nearest neighbors only relying on the defined distance functions in the original attribute view and totally ignoring the valuable information hidden in the multiple noisy labels, which limits their performance. Motivated by multi-view learning, we define the multiple noisy labels as another label view of instances and propose to search inferred instances’ nearest neighbors using the joint information from both the original attribute view and the multiple noisy label view. To this end, we propose a novel label integration method called dual-view K-nearest neighbor (DVKNN). In DVKNN, we first define a new distance function to search the K-nearest neighbors of an inferred instance. Then, we define a fine-grained weight for each noisy label from each neighbor. Finally, we perform weighted majority voting (WMV) on all these noisy labels to obtain the integrated label of the inferred instance. Extensive experiments validate the effectiveness and rationality of DVKNN.

In recent years, crowd counting has garnered increasing attention due to its wide range of societal applications. However, the vast differences in crowd density distributions across various areas within a scene make it difficult for existing crowd counting methods to achieve satisfactory performance. To address this issue, this letter introduces a novel method called Dynamic Density-Region-Aware Network (DDRANet). First, a Density-Aware Module (DAM) is designed to detect density variations and dynamically segment the image into multiple density regions. Subsequently, the Region Information Encoder (RIE) transforms the obtained density region map into a latent space to enhance the information of density regions. Then, through a specially designed Region Attention Block (RAB), the encoded density region map guides the model to focus on different density regions separately, enabling the model to adapt to the density variations of the crowd within the image. Furthermore, we combine Gaussian kernel function and Density Peak Clustering to automatically generate ground truth of density region maps, supervising the optimization of DAM. Extensive experimental results on three datasets demonstrate DDRANet's superior performance over other state-of-the-art methods.

Crowdsourcing services provide a fast and cheap way to annotate instances by employing crowd workers on the Internet. As a result, many learning from crowds (LFC) methods have been proposed in recent years. However, almost all these methods assume that all instances are benign, which makes them vulnerable to adversarial attacks. To improve the model's robustness to adversarial attacks, adversarial LFC (A-LFC) has attracted remarkable attention. A-LFC iteratively updates the true labels' estimations and the trained model by adversarial learning and uses the model's predictions in turn to help estimate the true labels. In A-LFC, to the best of our knowledge, the true labels' estimations and the model's predictions are inaccurate and the stopping condition for iterations is rough, which limit the model's performance. To further improve A-LFC, this paper proposes weighted A-LFC (WA-LFC). To reduce the impact of misinformation in the true labels' estimations and model's predictions, our method weights instances for adversarial learning and weights the model's predictions for estimating the true labels. Our method iteratively updates these weights and uses instance-weighted cross-entropy loss to decide when the iterative process should be stopped. Experiments on three real-world datasets show that our method substantially improves the trained model's performance. On average, the test accuracy of our method outperforms that of A-LFC by 10.06% and 27.55% in white-box and black-box attack settings, respectively.

Multimodal learning aims to integrate complementary information from different modalities for more reliable decisions. However, existing multimodal classification methods simply integrate the learned local features, which ignore the underlying structure of each modality and the higher-order correlation across modalities. In this paper, we propose a novel Hierarchical Attention Learning Network (HALNet) for multimodal classification. Specifically, HALNet has three merits: 1) A hierarchical feature fusion module is proposed to learn multilevel features, aggregating multi-level features for a global feature representation with the attention mechanism and progressive fusion tactics. 2) A cross-modal higher-order fusion module is introduced to capture the prospective cross-modal correlations at label space. 3) A dual prediction pattern is designed to generate credible decisions. Extensive experiments on three real-world multimodal datasets demonstrate that HALNet achieves competitive performance compared to the state-of-the-art.

In crowdsourcing scenarios, each instance obtains a multiple noisy label set from different crowd workers on the Internet and then gets its integrated label via label integration. Although the label integration algorithms are often effective, a certain level of noise still remains in the integrated labels. To reduce the impact of noise on label quality, many noise correction algorithms have been proposed in recent years. However, most of them are hard to fully utilize the joint information of the original attribute view and multiple noisy label view. Motivated by multi-view learning, in this article, we propose a dual-view noise correction (DVNC) algorithm. Benefiting from the complementary and consensus principle, DVNC can fully utilize the joint information of two views and thus enhance the effect of noise correction. In DVNC, we first construct the original attribute view and multiple noisy label view for each instance and, respectively, train a classifier on each view. Then, we use the trained two classifiers to filter each noise instance and thus obtain a clean set and noise set. Finally, we train a classifier on the clean set to correct each instance in the noise set via reclassifying it as the class with the maximum posterior probability. The experimental results on simulated and real-world data sets indicate that DVNC significantly outperforms all the other noise correction algorithms used for comparison.

Naive Bayes is a classical machine learning algorithm for which discretization is commonly used to transform quantitative attributes into qualitative attributes. Of numerous discretization methods, Non-Disjoint Discretization (NDD) proposes a novel perspective by forming overlapping intervals and always locating a value toward the middle of an interval. However, existing approaches to NDD fail to adequately consider the effect of multiple occurrences of a single value — a commonly occurring circumstance in practice. By necessity, all occurrences of a single value fall within the same interval. As a result, it is often not possible to discretize an attribute into intervals containing equal numbers of training instances. Current methods address this issue in an ad hoc manner, reducing the specificity of the resulting atomic intervals. In this study, we propose a non-disjoint discretization method for NB, called Rigorous Non-Disjoint Discretization (RNDD), that handles multiple occurrences of a single value in a systematic manner. Our extensive experimental results suggest that RNDD significantly outperforms NDD along with all other existing state-of-the-art competitors.

In crowdsourcing scenarios, we can obtain each instance’s multiple noisy labels set from different crowd workers and then use a ground truth inference algorithm to infer its integrated label. Despite the effectiveness of ground truth inference algorithms, a certain level of noise still remains in the integrated labels. To reduce the impact of noise, many noise correction algorithms have been proposed in recent years. To the best of our knowledge, however, nearly all existing noise correction algorithms only exploit each instance’s own multiple noisy label sets but ignore the multiple noisy label sets of its neighbors. Here neighbors refer to the nearest instances found in the feature space based on the distance metric learning. In this article, we propose neighborhood weighted voting-based noise correction (NWVNC). In NWVNC, we at first take advantage of the multiple noisy label sets of each instance’s neighbors (including itself) to estimate the probability that it belongs to its integrated label. Then, we use the estimated probability to identify and filter noise instances and thus obtain a clean set and a noise set. Finally, we train three heterogeneous classifiers on the clean set and correct the noise instances by the consensus voting of three trained classifiers. The experimental results on 34 simulated and two real-world crowdsourced datasets show that NWVNC significantly outperforms all the other state-of-the-art noise correction algorithms used for comparison.

Naive Bayes (NB) continues to be one of the top 10 data mining algorithms due to its simplicity, efficiency and efficacy. Numerous enhancements have been proposed to weaken its attribute conditional independence assumption. However, all of them only focus on the raw attribute view, which is hard to reflect all the data characteristics in real-world applications. To portray data characteristics more comprehensively, in this study, we construct two label views from the raw attributes and propose a novel model called multi-view attribute weighted naive Bayes (MAWNB). In MAWNB, we first build multiple super-parent one-dependence estimators (SPODEs) as well as random trees (RTs), then we utilize each of them to classify each training instance in turn and use all their predicted class labels to construct two label views. Next, to avoid attribute redundancy, we optimize the weight of each attribute value for each class by minimizing the negative conditional log-likelihood (CLL) in each view. Finally, the estimated class-membership probabilities by three views are fused to predict the class label for each test instance. Extensive experiments show that MAWNB significantly outperforms NB and all the other existing state-of-the-art competitors.

In crowdsourcing scenarios, we can often obtain each instance's multiple noisy labels from different crowd workers and then use a label integration method to infer its integrated label. In spite of the effectiveness of label integration methods, a certain level of label noise still exists in integrated labels. To reduce the impact of label noise, noise correction has attracted much attention from researchers, and therefore a certain number of noise correction methods were proposed in recent years. Among them, Between-class Margin-based Noise Correction (BMNC) has demonstrated remarkable denoising performance. However, BMNC can only handle binary classification tasks. For multiclass classification tasks, we propose an effective but very simple noise correction method in this paper. We refer to our proposed method as Label Distribution-based Noise Correction (LDNC). At first, LDNC transforms each instance's multiple noisy labels into a label distribution. Then, LDNC uses the margin between the first- and second-largest label probabilities in the label distribution to identify and filter each possible noise instance and thus obtains a clean set and a noise set. Finally, LDNC builds a classifier on the clean set to relabel all instances in the noise set. The experimental results on 16 simulated and one real-world multiclass crowdsourced data sets show that LDNC significantly outperforms all the other existing state-of-the-art noise correction methods.

Crowdsourcing services provide a fast, efficient, and cost-effective way to obtain large labeled data for supervised learning. Unfortunately, the quality of crowdsourced labels cannot satisfy the standards of practical applications. Ground-truth inference, simply called label integration, designs proper aggregation methods to infer the unknown true label of each instance (sample) from the multiple noisy label set provided by ordinary crowd labelers (workers). However, nearly all existing label integration methods focus solely on the multiple noisy label set per individual instance while totally ignoring the intercorrelation among multiple noisy label sets of different instances. To solve this problem, a multiple noisy label distribution propagation (MNLDP) method is proposed in this article. MNLDP at first estimates the multiple noisy label distribution of each instance from its multiple noisy label set and then propagates its multiple noisy label distribution to its nearest neighbors. Consequently, each instance absorbs a fraction of the multiple noisy label distributions from its nearest neighbors and yet simultaneously maintains a fraction of its own original multiple noisy label distribution. Empirical studies on a collection of an artificial dataset, six simulated UCI datasets, and three real-world crowdsourced datasets show that MNLDP outperforms all other existing state-of-the-art label integration methods in terms of the integration accuracy and classification accuracy.

In the printed circuit board (PCB) industry, cosmetic defect detection is an essential process to ensure product quality. However, existing PCB cosmetic defect detection approaches have a high false alarm rate, which lead to expensive labor costs of manual confirmation. To solve this problem, some traditional machine learning-based approaches have been proposed, but they just utilize hand-crafted features to build classifiers and thus are rough and sub-optimal. Recently, due to its powerful capability in automatic feature extraction, convolutional neural network (CNN) has been widely used in PCB cosmetic defect detection. However, few of them pay attention to the imbalanced class distribution as well as the different misclassification costs of real and pseudo defects, both of which are common problems in the PCB industry. To this end, in this study, we propose a novel model called cost-sensitive residual convolutional neural network (CS-ResNet) by adding a cost-sensitive adjustment layer in the standard ResNet. Specifically, we assign larger weights to minority real defects based on the class-imbalance degree and then optimize CS-ResNet by minimizing the weighted cross-entropy loss function. We conducted a series of experiments by comparing CS-ResNet with the standard ResNet, state-of-the-art CNN-based approach Auto-VRS and traditional machine learning-based approach HOG-SVM on a real-world PCB cosmetic defect dataset. Experimental results show that CS-ResNet achieves the highest Sensitivity (0.89), G-mean (0.91) and the lowest misclassification costs.

Crowdsourcing services provide an economic and efficient means of acquiring multiple noisy labels for each training instance in supervised learning. Ground truth inference methods, also known as consensus methods, are then used to obtain the integrated labels of training instances. Although consensus methods are effective, there still exists a level of noise in the set of integrated labels. Therefore, it is necessary to handle noise in the integrated labels to improve label and model quality. In this paper, we propose a resampling-based noise correction method (simply RNC). Different from previous label noise correction methods for crowdsourcing, RNC first employs a filter to obtain a clean set and a noisy set and then repeatedly resamples the clean and noisy sets several times according to a certain proportion. Finally, multiple classifiers built on the resampled data sets are used to re-label the training data. Experimental results based on 18 simulated data sets and five real-world data sets demonstrate that RNC rarely degrades the label and model quality compared to other three state-of-the-art noise correction methods and, in many cases, improves quality dramatically.

Feature selection is an optional preprocessing procedure and is frequently used to improve the classification accuracy of a machine learning algorithm by removing irrelevant and/or redundant features. However, in many real-world applications, the test cost is also required for making optimal decisions, in addition to the classification accuracy. To the best of our knowledge, thus far, few studies have been conducted on test-cost-sensitive feature selection (TCSFS). In TCSFS, the objectives are twofold: 1) to improve the classification accuracy and 2) to decrease the test cost. Therefore, in fact, it constitutes a multiobjective optimization problem. In this paper, we transformed this multiobjective optimization problem into a single-objective optimization problem by utilizing a new evaluation function and in this paper, we propose a new general wrapper framework for TCSFS. Specifically, in our proposed framework, we add a new term to the evaluation function of a wrapper feature selection method so that the test cost of measuring features is taken into account. We experimentally tested our proposed framework, using 36 classification problems from the University of California at Irvine (UCI) repository, and compared it to some other state-of-the-art feature selection frameworks. The experimental results showed that our framework allows users to select an optimal feature subset with the minimal test cost, while simultaneously maintaining a high classification accuracy.

Scores of machine learning algorithms depend on a good distance measure to achieve high performance. The inverted specific-class distance measure, simply ISCDM, is proposed to find reasonable distance measure between each pair of instances with nominal attributes only. ISCDM does not depend on the attribute value of the training instance, which makes it less sensitive to missing values in the training set and more robust to non-class attribute noise. However, in ISCDM, all attributes are assumed to be fully independent. It is obvious that the attribute independence assumption in ISCDM is rarely true in reality, which would harm its performance in the applications with complex attribute dependencies. In this paper, we single out an improved inverted specific-class distance measure by relaxing its unrealistic attribute independence assumption. We call it averaged one-dependence inverted specific-class distance measure, simply AODISCDM. We experimentally tested it on 29 classification problems from the University of California at Irvine (UCI) repository and found that it significantly outperforms ISCDM in terms of the negative conditional log likelihood (-CLL) and the root relative squared error (RRSE). Besides, the proposed AODISCDM maintains the computational simplicity (no search involved) and robustness that characterise ISCDM.

Crowdsourcing services provide a fast, efficient, and cost-effective means of obtaining large labeled data for supervised learning. Ground truth inference, also called label integration, designs proper aggregation strategies to infer the unknown true label of each instance from the multiple noisy label set provided by ordinary crowd workers. However, to the best of our knowledge, nearly all existing label integration methods focus solely on the multiple noisy label set itself of the individual instance while totally ignoring the intercorrelation among multiple noisy label sets of different instances. To solve this problem, a multiple noisy label distribution propagation (MNLDP) method is proposed in this study. MNLDP first transforms the multiple noisy label set of each instance into its multiple noisy label distribution and then propagates its multiple noisy label distribution to its nearest neighbors. Consequently, each instance absorbs a fraction of the multiple noisy label distributions from its nearest neighbors and yet simultaneously maintains a fraction of its own original multiple noisy label distribution. Promising experimental results on simulated and real-world datasets validate the effectiveness of our proposed method.

Due to its simplicity, efficiency and efficacy, naive Bayes (NB) continues to be one of the top 10 data mining algorithms. A mass of improved approaches to NB have been proposed to weaken its conditional independence assumption. However, there has been little work, up to the present, on instance weighting filter approaches to NB. In this paper, we propose a simple, efficient, and effective instance weighting filter approach to NB. We call it attribute (feature) value frequency-based instance weighting and denote the resulting improved model as attribute value frequency weighted naive Bayes (AVFWNB). In AVFWNB, the weight of each training instance is defined as the inner product of its attribute value frequency vector and the attribute value number vector. The experimental results on 36 widely used classification problems show that AVFWNB significantly outperforms NB, yet at the same time maintains the computational simplicity that characterizes NB.

Due to its simplicity, efficiency, and efficacy, naive Bayes (NB) has continued to be one of the top 10 algorithms in the data mining and machine learning community. Of numerous approaches to alleviating its conditional independence assumption, feature weighting has placed more emphasis on highly predictive features than those that are less predictive. In this paper, we argue that for NB highly predictive features should be highly correlated with the class (maximum mutual relevance), yet uncorrelated with other features (minimum mutual redundancy). Based on this premise, we propose a correlation-based feature weighting (CFW) filter for NB. In CFW, the weight for a feature is a sigmoid transformation of the difference between the feature-class correlation (mutual relevance) and the average feature-feature intercorrelation (average mutual redundancy). Experimental results show that NB with CFW significantly outperforms NB and all the other existing state-of-the-art feature weighting filters used to compare. Compared to feature weighting wrappers for improving NB, the main advantages of CFW are its low computational complexity (no search involved) and the fact that it maintains the simplicity of the final model. Besides, we apply CFW to text classification and have achieved remarkable improvements.

Naive Bayes (NB) continues to be one of the top 10 data mining algorithms due to its simplicity, efficiency and efficacy, but the assumption of independence for attributes in NB is rarely true in reality. Attribute weighting is effective for overcoming the unrealistic assumption in NB, but it has received less attention than it warrants. Attribute weighting approaches can be broadly divided into two categories: filters and wrappers. In this paper, we mainly focus on wrapper attribute weighting approaches because they have generally higher classification performance than filter attribute weighting approaches. We propose a weight adjusted naive Bayes approach and simply denote it WANB. In WANB, the importance of each attribute in the classification of a training data set is learned and the weight vector reflecting this importance is updated. We use weight adjustment based on objective functions to find the optimal weight vector. We compare WANB with standard NB and its state-of-the-art attribute weighting approaches. Empirical studies on a collection of 36 benchmark datasets show that the classification performance of WANB significantly outperforms NB and all the existing filter approaches used to compare. Yet at the same time, compared to the existing wrapper approach called DEWANB, WANB is much more efficient and comprehensible.

Handling text data is a challenge for machine learning because text data is high dimensional in many cases. Feature selection has been approved to be an effective approach to handle high-dimensional data. Feature selection approaches can be broadly divided into two categories: filter approaches and wrapper approaches. Generally, wrapper approaches have superior accuracy compared to filters, but filters always run faster than wrapper approaches. In order to integrate the advantages of filter approaches and wrapper approaches, we propose a gain ratio-based hybrid feature selection approach to naive Bayes text classifiers. The hybrid feature selection approach uses base classifiers to evaluate feature subsets like wrapper approaches, but it need not repeatedly search feature subsets and build base classifiers. The experimental results on large suite of benchmark text datasets show that the proposed hybrid feature selection approach significantly improves the classification accuracy of the original naive Bayes text classifiers while does not incur the high time complexity that characterizes wrapper approaches.

In many real-world applications, it is often the case that the class distribution of example is imbalanced and the costs of misclassification are different. In such circumstances, not classification accuracy but misclassification cost minimization is the primary goal leading to the development of the class-imbalanced cost-sensitive learning. Under-sampling is one of the most important methods in dealing with class-imbalance problems, which uses only a subset of the majority class that is imperfect. In fact, many useful majority class examples are easy to be overlooked. To overcome this deficiency, in this paper, we propose a new adaptive method based on differential evolution for class-imbalanced cost-sensitive learning. We simply denote our proposed method by DE. DE explores the potential useful information contained in these majority class examples and creates an optimal example subset with the goal of low misclassification costs. We compare the performance of DE to its state-of-the-art competitors such as random under-sampling (US) and synthetic minority over-sampling technique (SMOTE) in terms of the total misclassification costs of the resulting base classifiers. The experimental results on 22 class-imbalanced data sets show that our proposed DE is notably better than the widely used random under-sampling (US) and the well-known synthetic minority over-sampling technique (SMOTE).

In many real-world applications, it is often the case that the class distribution of instances is imbalanced and the costs of misclassification are different. Thus, the class-imbalanced cost-sensitive learning has attracted much attention from researchers. Sampling is one of the widely used techniques in dealing with the class-imbalance problem, which alters the class distribution of instances so that the minority class is well represented in the training data. In this paper, we propose a novel Minority Cloning Technique (MCT) for class-imbalanced cost-sensitive learning. MCT alters the class distribution of training data by cloning each minority class instance according to the similarity between it and the mode of the minority class. The experimental results on a large number of UCI datasets show that MCT performs much better than Minority Oversampling with Replacement Technique (MORT) and Synthetic Minority Oversampling TEchnique (SMOTE) in terms of the total misclassification costs of the built classifiers.

A high quality distance function that measures the difference between instances is essential in many real-world applications and research fields. For example, in instance-based learning, the distance function plays the most important role. A large number of distance functions have been proposed. For nominal attributes, Value Difference Metric (VDM) is one of the state-of-the-art and widely used distance functions. However, it needs to estimate the conditional probabilities, which drops its efficiency in computing the distance between instances. Besides, a practical issue that arises in estimating the conditional probabilities is that the denominators can be zero or very small. This makes them either undefined or very large. Therefore, an efficient distance function that can measure the difference between two instances but without the practical issue confronting VDM is desirable. In this paper, we propose a novel distance function: Frequency Difference Metric (FDM). FDM is just based on the joint frequencies of class labels and attribute values, instead of the conditional probabilities. Extensive empirical studies show that FDM performs almost as well as VDM in terms of accuracy, but significantly outperforms VDM in terms of efficiency. This work provides a very simple, efficient, and effective distance function that can be widely used in many real-world applications and research fields.

In many real-world applications, it is often the case that the class distribution of instances is imbalanced and the costs of misclassification are different. Thus, class-imbalance and cost-sensitive learning have attracted much attention from researchers. Sampling is one of the widely used approaches in dealing with the class imbalance problem, which alters the class distribution of instances so that the minority class is well represented in the training data. In this paper, we study the effect of sampling the natural training data on state-of-the-art Bayesian network classifiers, such as Naive Bayes (NB), Tree Augmented Naïve Bayes (TAN), Averaged One-Dependence Estimators (AODE), Weighted Average of One-Dependence Estimators (WAODE), and Hidden naive Bayes (HNB) and propose sampled Bayesian network classifiers. Our experimental results on a large number of UCI datasets show that our sampled Bayesian network classifiers perform much better than the ones trained from the natural training data especially when the natural training data is highly imbalanced and the cost ratio is high enough.

Classification is an important task in data mining, while accurate class probability estimation is also desirable in real-world applications. Some probability-based classifiers, such as the k-nearest neighbor algorithm (KNN) and its variants, can estimate the class membership probabilities of the test instance. Unfortunately, a good classifier is not always a good class probability estimator. In this paper, we try to improve the class probability estimation performance of KNN and its variants. As we all know, KNN and its variants are all of the distance-related algorithms and their performance is closely related to the used distance metric. Value Difference Metric (VDM) is one of the widely used distance metrics for nominal attributes. Thus, in order to scale up the class probability estimation performance of the distance-related algorithms such as KNN and its variants, we propose an Attribute Weighted Value Difference Metric (AWVDM) in this paper. AWVDM uses the mutual information between the attribute variable and the class variable to weight the difference between two attribute values of each pair of instances. Experimental results on 36 UCI benchmark datasets validate the effectiveness of the proposed AWVDM.

Due to being fast, easy to implement and relatively effective, some state-of-the-art naive Bayes text classifiers with the strong assumption of conditional independence among attributes, such as multinomial naive Bayes, complement naive Bayes and the one-versus-all-but-one model, have received a great deal of attention from researchers in the domain of text classification. In this article, we revisit these naive Bayes text classifiers and empirically compare their classification performance on a large number of widely used text classification benchmark datasets. Then, we propose a locally weighted learning approach to these naive Bayes text classifiers. We call our new approach locally weighted naive Bayes text classifiers (LWNBTC). LWNBTC weakens the attribute conditional independence assumption made by these naive Bayes text classifiers by applying the locally weighted learning approach. The experimental results show that our locally weighted versions significantly outperform these state-of-the-art naive Bayes text classifiers in terms of classification accuracy.

Naive Bayes (NB) is a probability-based classification model which is based on the attribute independence assumption. However, in many real-world data mining applications, its attribute independence assumption is often violated. Responding to this fact, researchers have made a substantial amount of effort to improve the classification accuracy of NB by weakening its attribute independence assumption. For a recent example, averaged one-dependence estimators (AODE) is proposed, which weakens its attribute independence assumption by averaging all models from a restricted class of one-dependence classifiers. However, all one-dependence classifiers in AODE have same weights and are treated equally. According to our observation, different one-dependence classifiers should have different weights. Therefore, in this article, we proposed an improved model called weighted average of one-dependence estimators (WAODE) by assigning different weights to these one-dependence classifiers. In our WAODE, four different weighting approaches are designed and thus four different versions are created. For simplicity, we respectively denote them by WAODE-MI, WAODE-ACC, WAODE-CLL and WAODE-AUC. The experimental results on a large number of UCI datasets published on the main website of Weka platform show that our WAODE significantly outperform AODE.

Traditionally, the performance of a classifier is measured by its classification accuracy or error rate. In fact, probability-based classifiers also produce the class probability estimation (the probability that a test instance belongs to the predicted class). This information is often ignored in classification, as long as the class with the highest class probability estimation is identical to the actual class. In many data mining applications, however, classification accuracy and error rate are not enough. For example, in direct marketing, we often need to deploy different promotion strategies to customers with different likelihood (class probability) of buying some products. Thus, accurate class probability estimations are often required to make optimal decisions. In this paper, we firstly review some state-of-the-art probability-based classifiers and empirically investigate their class probability estimation performance. From our experimental results, we can draw a conclusion: C4.4 is an attractive algorithm for class probability estimation. Then, we present a locally weighted version of C4.4 to scale up its class probability estimation performance by combining locally weighted learning with C4.4. We call our improved algorithm locally weighted C4.4, simply LWC4.4. We experimentally test LWC4.4 using the whole 36 UCI data sets selected by Weka. The experimental results show that LWC4.4 significantly outperforms C4.4 in terms of class probability estimation.

Because learning an optimal Bayesian network classifier is an NP-hard problem, learning-improved naive Bayes has attracted much attention from researchers. In this paper, we summarize the existing improved algorithms and propose a novel Bayes model: hidden naive Bayes (HNB). In HNB, a hidden parent is created for each attribute which combines the influences from all other attributes. We experimentally test HNB in terms of classification accuracy, using the 36 UCI data sets selected by Weka, and compare it to naive Bayes (NB), selective Bayesian classifiers (SBC), naive Bayes tree (NBTree), tree-augmented naive Bayes (TAN), and averaged one-dependence estimators (AODE). The experimental results show that HNB significantly outperforms NB, SBC, NBTree, TAN, and AODE. In many data mining applications, an accurate class probability estimation and ranking are also desirable. We study the class probability estimation and ranking performance, measured by conditional log likelihood (CLL) and the area under the ROC curve (AUC), respectively, of naive Bayes and its improved models, such as SBC, NBTree, TAN, and AODE, and then compare HNB to them in terms of CLL and AUC. Our experiments show that HNB also significantly outperforms all of them.

Improving naive Bayes (simply NB)15,28 for classification has received significant attention. Related work can be broadly divided into two approaches: eager learning and lazy learning.1 Different from eager learning, the key idea for extending naive Bayes using lazy learning is to learn an improved naive Bayes for each test instance. In recent years, several lazy extensions of naive Bayes have been proposed. For example, LBR,30 SNNB,27 and LWNB.8 All these algorithms aim to improve naive Bayes' classification performance. Indeed, they achieve significant improvement in terms of classification, measured by accuracy. In many real-world data mining applications, however, an accurate ranking is more desirable than an accurate classification. Thus a natural question is whether they also achieve significant improvement in terms of ranking, measured by AUC (the area under the ROC curve).2,11,17 Responding to this question, we conduct experiments on the 36 UCI data sets18 selected by Weka12 to investigate their ranking performance and find that they do not significantly improve the ranking performance of naive Bayes. Aiming at scaling up naive Bayes' ranking performance, we present a novel lazy method ICNB (instance cloned naive Bayes) and develop three ICNB algorithms using different instance cloning strategies. We empirically compare them with naive Bayes. The experimental results show that our algorithms achieve significant improvement in terms of AUC. Our research provides a simple but effective method for the applications where an accurate ranking is desirable.

Naive Bayes (simply NB) (Langley et al., 1992) has been widely used in machine learning and data mining as a simple and effective classification algorithm. Since its conditional independence assumption is rarely true, researchers have made a substantial amount of effort to improve naive Bayes. The related research work can be broadly divided into two approaches: eager learning and lazy learning, depending on when the major computation occurs. Different from eager approach, the key idea for extending naive Bayes from the lazy approach is to learn a naive Bayes for each testing example. In recent years, some lazy extensions of naive Bayes have been proposed. For example, SNNB, LWNB, and LBR. All are aiming at improving the classification accuracy of naive Bayes. In many real-world machine learning and data mining applications, however, an accurate ranking is more desirable than an accurate classification. Responding to this fact, we present a lazy learning algorithm called instance greedily cloning naive Bayes (simply IGCNB) in this paper. Our motivation is to improve naive Bayes' ranking performance measured by AUC (Bradley, 1997; Provost and Fawcett, 1997). We experimentally tested our algorithm, using the whole 36 UCI datasets recommended by Weka, and compared it to C4.4 (Provost and Domingos, 2003), NB (Langley et al., 1992), SNNB (Xie, 2002) and LWNB (Frank, 2003). The experimental results show that our algorithm outperforms all the other algorithms used to compare significantly in yielding accurate ranking.

Naive Bayes is an effective and efficient learning algorithm in classification. In many applications, however, an accurate ranking of instances based on the class probability is more desirable. Unfortunately, naive Bayes has been found to produce poor probability estimates. Numerous techniques have been proposed to extend naive Bayes for better classification accuracy, of which selective Bayesian classifiers (SBC) (Langley & Sage, 1994), tree-augmented naive Bayes (TAN) (Friedman et al., 1997), NBTree (Kohavi, 1996), boosted naive Bayes (Elkan, 1997), and AODE (Webb et al., 2005) achieve remarkable improvement over naive Bayes in terms of classification accuracy. An interesting question is: Do these techniques also produce accurate ranking? In this paper, we first conduct a systematic experimental study on their efficacy for ranking. Then, we propose a new approach to augmenting naive Bayes for generating accurate ranking, called hidden naive Bayes (HNB). In an HNB, a hidden parent is created for each attribute to represent the influences from all other attributes, and thus a more accurate ranking is expected. HNB inherits the structural simplicity of naive Bayes and can be easily learned without structure learning. Our experiments show that HNB outperforms naive Bayes, SBC, boosted naive Bayes, NBTree, and TAN significantly, and performs slightly better than AODE in ranking.

Naive Bayes (simply NB) has been well-known as an effective and efficient classification algorithm. However, it is based on the conditional independence assumption that it is often violated in applications. In addition, in many real-world data mining applications, however, an accurate ranking of instances is often required rather than an accurate classification. For example, a ranking of customers in terms of the likelihood that they buy one's products is useful in direct marketing. In this paper, we firstly investigate the ranking performance of some lazy learning algorithms for extending naive Bayes. The ranking performance is measured by Hand and Till (2001) and Bradley (1997). We observe that they can not significantly improve naive Bayes' ranking performance. Motivated by this fact and aiming at improving naive Bayes with accurate ranking, we present a new lazy learning algorithm, called lazy naive Bayes (simply LNB), to extend naive Bayes for ranking. We experimentally tested our algorithm, using the whole 36 UCI data sets (Blake and Merz, 2000) recommended by Weka, and compared it to NB and C4.4 (Provost and Domingos, 2003) measured by AUC. The experimental results show that our algorithm significantly outperforms both NB and C4.4.