With online crowdsourcing platforms, labels can be acquired at relatively low costs from massive nonexpert workers. To improve the quality of labels obtained from these imperfect crowdsourced workers, we usually let different workers provide labels for the same instance. Then, the true labels for all instances are estimated from these multiple noisy labels. This traditional general-purpose label aggregation process, solely relying on the collected noisy labels, cannot significantly improve the accuracy of integrated labels under a low labeling quality circumstance. This paper proposes a novel bilayer collaborative clustering (BLCC) method for the label aggregation in crowdsourcing. BLCC first generates the conceptual-level features for the instances from their multiple noisy labels and infers the initially integrated labels by performing clustering on the conceptual-level features. Then, it performs another clustering on the physical-level features to form the estimations of the true labels on the physical layer. The clustering results on both layers can facilitate in tracking the changes in the uncertainties of the instances. Finally, the initially integrated labels that are likely to be wrongly inferred on the conceptual layer can be addressed using the estimated labels on the physical layer. The clustering processes on both layers can keep providing guidance information for each other in the multiple label remedy rounds. The experimental results on 12 real-world crowdsourcing data sets show that the performance of the proposed method in terms of accuracy is better than that of the state-of-the-art methods.

Multi-label active learning for image classification has been a popular research topic. It faces several challenges, even though related work has made great progress. Existing studies on multi-label active learning do not pay attention to the cleanness of sample data. In reality, data are easily polluted by external influences that are likely to disturb the exploration of data space and have a negative effect on model training. Previous methods of label correlation mining, which are purely based on observed label distribution, are defective. Apart from neglecting noise influence, they also cannot acquire sufficient relevant information. In fact, they neglect inner relation mapping from example space to label space, which is an implicit way of modeling label relationships. To solve these issues, we develop a novel multi-label active learning with low-rank application (ENMAL) algorithm in this paper. A low-rank model is constructed to quantize noise level, and the example-label pairs that contain less noise are emphasized when sampling. A low-rank mapping matrix is learned to signify the mapping relation of a multi-label domain to capture a more comprehensive and reasonable label correlation. Integrating label correlation with uncertainty and considering sample noise, an efficient sampling strategy is developed. We extend ENMAL with automatic labeling (denoted as AL-ENMAL) to further reduce the annotation workload of active learning. Empirical research demonstrates the efficacy of our approaches.

In multi-label image classification, each image is always associated with multiple labels and labels are usually correlated with each other. The intrinsic relation among labels can definitely contribute to classifier training. However, most previous studies on active learning for multi-label image classification purely mine label correlation based on observed label distribution. They ignore the mapping relation between examples and their labels. This mapping relation also implicates label relationship. Ignoring the mapping relation leads to an uncomprehensive label correlation estimation and results in a bad performance for classification. In this paper, we propose a novel multi-label active learning with low-rank mapping for image classification, called LMMAL, to solve this issue. More precisely, we train a low-rank mapping matrix to signify the mapping relation between the feature space and the label space of a certain multi-label dataset. Using this low-rank mapping relation, we exploit a full label correlation. Subsequently, an effective sampling strategy is designed by integrating this potential information with uncertainty to select the most informative example-label pairs. In addition, we extend LMMAL with automatic labeling (denoted as AL-LMMAL) to further reduce the annotation workload of active learning. Empirical results demonstrate the effectiveness of our approaches.

Multi-label active learning for image classification has attracted great attention over recent years and a lot of relevant works are published continuously. However, there still remain some problems that need to be solved, such as existing multi-label active learning algorithms do not reflect on the cleanness of sample data and their ways on label correlation mining are defective. For one thing, sample data is usually contaminated in reality, which disturbs the estimation of data distribution and further hinders the model training. For another, previous approaches for label relationship exploration are purely based on the observed label distribution of an incomplete training set, which cannot provide sufficiently efficient information. To address these issues, we propose a novel adaptive low-rank multi-label active learning algorithm, called LRMAL. Specifically, we first use low-rank matrix recovery to learn an effective low-rank feature representation from the noisy data. In a subsequent sampling phase, we make use of its superiorities to evaluate the general informativeness of each unlabeled example-label pair. Based on an intrinsic mapping relation between the example space and the label space of a certain multi-label dataset, we recover the incomplete labels of a training set for a more comprehensive label correlation mining. Furthermore, to reduce the redundancy among the selected example-label pairs, we use a diversity measurement to diversify the sampled data. Finally, an effective sampling strategy is developed by integrating these two aspects of potential information with uncertainty based on an adaptive integration scheme. Experimental results demonstrate the effectiveness of our approach.

Multi‐label image classification has attracted considerable attention in machine learning recently. Active learning is widely used in multi‐label learning because it can effectively reduce the human annotation workload required to construct high‐performance classifiers. However, annotation by experts is costly, especially when the number of labels in a dataset is large. Inspired by the idea of semi‐supervised learning, in this study, the authors propose a novel, semi‐supervised multi‐label active learning (SSMAL) method that combines automated annotation with human annotation to reduce the annotation workload associated with the active learning process. In SSMAL, they capture three aspects of potentially useful information – classification prediction information, label correlation information, and example spatial information – and they use this information to develop an effective strategy for automated annotation of selected unlabelled example‐label pairs. The experimental results obtained in this study demonstrate the effectiveness of the authors' proposed approach.

Multilabel image classification has been a hot topic in the field of computer vision and image understanding in recent years. To achieve better classification performance with fewer labeled images, multilabel active learning is used for this scenario. Several active learning methods have been proposed for multilabel image classification. However, all of them assume that either all training images have complete labels or label correlations are given at the beginning. These two assumptions are unrealistic. In fact, it is very difficult to obtain complete labels for each example, in particular when the size of labels in a multilabel dataset is very large. Typically, only partial labels are available. This is one type of “weak label” problem. To solve this weak label problem inside multilabel active learning, this paper proposes a novel solution called AE-WLMAL. AE-WLMAL explores conditional label correlations on the weak label problem with the help of input features and then utilizes label correlations to construct a unified sampling strategy and evaluate the informativeness of each example-label pair in a multilabel dataset for active sampling. In addition, a pruning strategy is adopted to further improve its computation efficiency. Moreover, AE-WLAML exploits label correlations to infer labels for unlabeled images, which further reduces human labeling cost. Our experimental results on seven real-world datasets show that AE-WLMAL consistently outperforms existing approaches.

Image classification is a hot topic of pattern recognition in computer vision. In order to achieve high accuracy of classification, a certain amount of high quality pictures are needed. As a matter of fact, high quality pictures are scarce. Active learning can solve such a problem. Label dependences play an important role in multi-label active learning for image classification. The interdependences between different labels are usually different and asymmetrical. This paper first brings the asymmetrical conditional label dependences into a novel active learning method for multi-label image classification based on the asymmetrical conditional label dependence, called ACDAL. Our extensive experimental results on three image and two non-image datasets show that our new approach ACDAL significantly outperforms existing approaches.

Due to low quality of crowdsourced labelers, the integrated label of each example is usually inferred from its multiple noisy labels provided by different labelers. This paper proposes a novel algorithm, Ground Truth Inference using Clustering (GTIC), to improve the quality of integrated labels for multi-class labeling. For a K labeling case, GTIC utilizes the multiple noisy label sets of examples to generate features. Then, it uses a K-Means algorithm to cluster all examples into K different groups, each of which is mapped to a specific class. Examples in the same cluster are assigned a corresponding class label. We compare GTIC with four existing multi-class ground truth inference algorithms, majority voting (MV), Dawid & Skene's (DS), ZenCrowd (ZC) and Spectral DS (SDS), on one synthetic and eight real-world datasets. Experimental results show that the performance of GTIC is significantly superior to the others in terms of both accuracy and M-AUC. Besides, the running time of GTIC is about twenty times faster than EM-based complicated inference algorithms.

Learning from imbalanced multilabel data is a challenging task. It has attracted considerable attention recently. In this paper we propose a MultiLabel Best First Over-sampling (ML-BFO) to improve the performance of multilabel classification algorithms, based on imbalance minimization and Wilson's ENN rule. Our experimental results show that ML-BFO not only duplicates fewer samples but also reduces the imbalance level much more than two state-of-the-art multilabel sampling methods, i.e., an over-sampling method LP-ROS and an under-sampling method MLeNN. Besides, ML-BFO significantly improves the performance of multilabel classification algorithms, and performs much better than LP-ROS and MLeNN.

This paper proposes a novel framework that introduces noise correction techniques to further improve label quality after ground truth inference in crowdsourcing. In the framework, an adaptive voting noise correction algorithm (AVNC) is proposed to identify and correct the most likely noises with the help of estimated qualities of labelers provided by the ground truth inference. The experimental results on two real-world datasets show that (1) the framework can improve label quality regardless of inference algorithms, especially under the circumstance that each example has a few noisy labels; and (2) since the algorithm AVNC considers both the number of and the probability of potential noises, it outperforms a baseline noise correction algorithm.

Label correlation analysis is very important for multi-label classification. And there is no study to measure the label correlation for example-label based active learning. In this paper, from a statistical point of view, we proposed a cosine similarity based multi-label active learning (CosMAL), which uses cosine similarity to accurately evaluate the correlations between all labels. It further uses the average correlation between the potential label and the other unlabeled labels as the label information for each sample-label pair. And then we select the most informativeness example-label pairs. Our empirical results demonstrate that our proposed method CosMAL outperforms the state-of-the-art active learning for multi-label classification. It significantly reduces the labeling workload and improves the performance of a classifier learned.

Active learning is to select the most informative examples to request their labels. Most previous studies in active learning for multi-label classification didn't pay enough attention on label correlations. This leads to a bad performance for classification. In this paper, we proposed a chi-square statistics multi-label active learning (CSMAL) algorithm, which uses chi-square statistics to accurately evaluate correlations between labels. CSMAL considers not only positive relationships but also negative ones. It uses the average correlation between a potential label and its rest unlabeled labels as the label information for each sample-label pair. CSMAL further integrates uncertainty and label information to select example-label pairs to request labels. Our empirical results demonstrate that our proposed method CSMAL outperforms the state-of-the-art active learning methods for multi-label classification. It significantly reduces the labeling workloads and improves the performance of a classifier built.

In order to achieve better classification performance with even fewer labeled images, active learning is suitable for these situations. Several active learning methods have been proposed for multi-label image classification, but all of them assume that all training images with complete labels. However, as a matter of fact, it is very difficult to get complete labels for each example, especially when the size of labels in a multi-label domain is huge. Usually, only partial labels are available. This is one kind of "weak label" problems. This paper proposes an ingeniously solution to this "weak label" problem on multi-label active learning for image classification (called WLMAL). It explores label correlation on the weak label problem with the help of input features, and then utilizes label correlation to evaluate the informativeness of each example-label pair in a multi-label dataset for active sampling. Our experimental results on three real-world datasets show that our proposed approach WLMAL consistently outperforms existing approaches significantly.

Multi-label image data is becoming ubiquitous. Image semantic understanding is typically formulated as a classification problem. This paper focuses on multi-label active learning for image classification. It first extends a traditional example based active learning method for multilabel active learning for image classification. Since the traditional example based active method doesn't work well, we propose a novel example-label based multi-label active learning method. Our experimental results on two image datasets demonstrate that the proposed method significantly reduces the labeling workload and improves the performance of the built classifier. Additionally, we conduct experiments on two other types of multi-label datasets for validating the versatility of our proposed method, and the experimental results show the consistent effect.