Knowledge graph (KG) contrastive learning (CL) has garnered significant attention in the realm of recommendation systems. However, existing models often employ random masking for graph enhancement, which can introduce sampling bias and impede interpretability. Furthermore, the KG-UIG information imbalance can lead to the neglect of critical information in the user-item interaction graph (UIG) by the model. To address these challenges, we propose a novel model, diffusion-augmented graph CL (DAGCL). This model leverages a graph diffusion mechanism for data enhancement in CL, thereby ensuring that the generated diffusion graph closely resembles the original UIG and avoiding the pitfalls associated with random sampling. Additionally, DAGCL enhances the impact of UIG on predictive accuracy by implementing both intragraph and intergraph CL (GCL), effectively mitigating the information imbalance between KGs and UIG. The model also leverages the structural characteristics of the UIG to construct a structural diffusion graph, which is integrated with the information diffusion graph to produce a comprehensive diffusion representation-further enhancing the model's robustness against sampling noise and semantic dilution by preserving essential interaction patterns and structural features in the augmented graph. Experimental results across three real-world datasets demonstrate that our proposed model outperforms state-of-the-art models significantly.

Fast Gradient Sign Method (FGSM)-based ap proaches play a critical role in adversarial attacks, particularly in transfer-based attacks. Recent advancements aim to enhance adversarial transferability by refining the sign function in FGSMs using precise gradient information. However, due to the instability of gradient update directions, such precise gradient information may still converge to local minima, thereby limiting transferability. To address this issue, we propose Majority-vote Gradient Scaling (MVGS), a novel method designed to sta bilize gradient update directions. MVGS dynamically rescales momentum terms by evaluating the consistency of historical gradients at each pixel. Since MVGS operates solely through parameter-specific momentum rescaling, it can be seamlessly integrated with existing adversarial attack methods that lever age precise gradient information. Extensive comparative and ablation experiments demonstrate that MVGS significantly improves attack success rates (ASR), achieving up to a 6.1% increase over the state-of-the-art APAA method, confirming its effectiveness.

Graph Neural Networks (GNNs) have been proven to be useful for learning graph-based knowledge. However, one of the drawbacks of GNN techniques is that they may get stuck in the problem of over-squashing. Recent studies attribute to the message passing paradigm that it may amplify some specific local relations and distort long-range information under a certain GNN. To alleviate such phenomena, we propose a novel and general GNN framework, dubbed MC-GNN, which introduces the multi-channel neural architecture to learn and fuse multi-view graph-based information. The purpose of MC-GNN is to extract distinct channel-based graph features and adaptively adjust the importance of the features. To this end, we use the Hilbert-Schmidt Independence Criterion (HSIC) to enlarge the disparity between the embeddings encoded by each channel and follow an attention mechanism to fuse the embeddings with adaptive weight adjustment. MC-GNN can apply multiple GNN backbones, which provides a solution for learning structural relations from a multi-view perspective. Experimental results demonstrate that the proposed MC-GNN is superior to the compared state-of-the-art GNN methods.

Crowdsourcing has been playing an essential role in machine learning since it can obtain a large number of labels in an economical and fast manner for training increasingly complex learning models. However, the application of crowdsourcing learning still faces several challenges such as the low quality of crowd labels and the urgent requirement for learning models adapting to the label noises. There have been many studies focusing on truth inference algorithms to improve the quality of labels obtained by crowdsourcing. Comparably, end-to-end predictive model learning in crowdsourcing scenarios, especially using cutting-edge deep learning techniques, is still in its infant stage. In this paper, we propose a novel graph convolutional network-based framework, namely CGNNAT, which models the correlation of instances by combining the GCN model with an attention mechanism to learn more representative node embeddings for a better understanding of the bias tendency of crowd workers. Furthermore, a specific projection processing layer is employed in CGNNAT to model the reliability of each crowd worker, which makes the model an end-to-end neural network directly trained by noisy crowd labels. Experimental results on several real-world and synthetic datasets show that the proposed CGNNAT outperforms state-of-the-art and classical methods in terms of label prediction.

This article focuses on the private security of multiagent systems with external eavesdroppers. Note that the agent data/states finally reach the same value by using the consensus-based anti-eavesdroppers protocol with Laplace noises, which leads to the leakage possibility of the terminal information. To this end, a novel differential privacy protocol is designed based on directed signed networks, which yields the performance of$\xi$-differential privacy-preserving. To further reveal the effects of eavesdroppers on the system, the privacy disclosure proportion of groups (PDPGs) is proposed in the two cases of a single-channel eavesdropper and a multichannel eavesdropper. It is indicated that the PDPG of the resulting system does not exceed 50% even if the malicious adversary has the ability to crack the privacy-preserving mechanism. The effectiveness of the approach and its superiority over state-of-the-art counterparts are confirmed through numerical simulations.

Crowdsourcing offers an efficient means of gathering labeled data and training machine learning models. However, the exposure of crowd workers to potential privacy breaches during data collection diminishes their willingness to participate. This paper proposes TsIFedCrowd, a novel incentive mechanism within the federated learning paradigm, aimed at mitigating privacy risks for crowd workers while achieving high-quality crowdsourced data and learning models at minimal expense. TsIFedCrowd enables clients to retain privacy-sensitive data locally, requiring only the upload of trained models to the server, which aggregates them into a global model. By modeling the federated crowdsourcing process as a two-stage Stackelberg game, TsIFedCrowd incentivizes workers to complete tasks with heightened quality and efficiency. It also provides a timeliness selection mechanism, which makes TsIFedCrowd suitable for both real-time crowdsourcing, where the generation time of task results (data) needs to meet the established requirements of the application, and non-real-time crowdsourcing. TsIFedCrowd mandates clients to complete federated crowdsourcing tasks while maximizing utility for both clients and the server, achieved through Nash equilibrium resolution. TsIFedCrowd is also extended to the multiple heterogeneous federated crowdsourcing scenario. Extensive experiments demonstrate TsIFedCrowd’s effectiveness in ensuring incentive fairness and universality, accelerating convergence speed, and budget savings. Real-world crowdsourcing tasks further validate its efficacy.

Crowdsourcing provides a cost-effective labeling solution for the acquisition of labeled training samples for machine learning by employing workers on the Internet. A common approach to improving the label quality is to employ a truth inference method to infer integrated labels for samples from their multiple noisy labels obtained from different crowd workers. Although the quality of integrated labels is significantly improved compared with that of the original noisy ones, it still cannot completely eliminate the noises inevitably existing in the integrated labels. To further improve the label quality, this paper proposes a novel label noise correction method for crowdsourcing based on dynamic resampling (DRNC). DRNC first divides the dataset with inferred labels into a clean set and a noisy set through a filter. According to a certain proportion, the clean set and the noisy set are resampled to train multiple heterogeneous classifiers, which form an ensemble classifier . Then, the dataset is divided by the ensemble classifier into a new sub-noisy set and a sub-clean set. The whole process repeats multiple rounds, generating multiple sub-clean sets. Finally, these sub-clean sets are used to train classifiers, which jointly correct the wrong labels in the dataset by voting. Experimental results on 25 simulated and 4 real-world datasets consistently show that the proposed DRNC averagely improves the quality of labels as well as the quality of learned models in the range of 1 to 10 percentage points, compared with four state-of-the-art crowdsourcing noise correction methods.

The Internet of Robotic Things (IoRT) is growing rapidly with new applications. Co-operatory robotics enables the sharing of information, autonomy, and fail-safe interaction with environment, humans, and other robots. They can also self-maintain, self-aware, and self-heal. To provide reliable and robust online monitoring of the industrial manipulator joint status, this article proposes a new IoRT architecture based on transfer learning (TL) techniques to detect manipulator fault. Robotic manipulator joint status are detected with high accuracy using a hybrid 1-D multichannel convolutional neural network (1D-MCNN), including matrix kernels and recurrent neural network (MCNN-RNN) technique. Moreover, a timestamp mapping method addresses the challenges associated with inconsistencies in sensor data timestamps. Existing data-driven methods struggle with the diverse operating conditions of industrial robots, where load and speed constantly fluctuate. To address this limitation, we propose a novel TL-based MCNN-RNN approach for joint fault diagnosis under varying work conditions. This method leverages the adaptability of TL while incorporating the inherent relations between different failure modes, enhancing the TL process. To demonstrate the performance of the suggested IoRT topology, various experimental scenarios are performed with data acquisition on six degree-of-freedom (DOF) UR16e (universal robot) manipulator. Based on the results, the proposed IoRT architecture can effectively visualize the joint fault status of the manipulator. As a result, TL architecture combined with MCNN-RNN provides an excellent accuracy of 99.03%in detecting faults on manipulator joints, which is significantly higher than traditional convolutional neural network (CNN), deep belief network (DBN), domain adversarial neural network (DANN), and conditional domain-adversarial network (CDAN).

Various correlations hidden in crowdsourcing annotation tasks bring opportunities to further improve the accuracy of label aggregation. However, these relationships are usually extremely difficult to be modeled. Most existing methods can merely make use of one or two correlations. In this paper, we propose a novel graph neural network model, namely LAGNN, which models five different correlations in crowdsourced annotation tasks by utilizing deep graph neural networks with convolution operations and derives a high label aggregation performance. Utilizing the group of high quality workers through labeling similarity, LAGNN can efficiently revise the preference among workers. Moreover, by injecting a little ground truth in its training stage, the label aggregation performance of LAGNN can be further significantly improved. We evaluate LAGNN on a large number of simulated datasets generated through varying six degrees of freedom and on eight real-world crowdsourcing datasets in both supervised and unsupervised (agnostic) modes. Experiments on data leakage is also contained. Experimental results consistently show that the proposed LAGNN significantly outperforms six state-of-the-art models in terms of label aggregation accuracy.

In biochemistry, graph structures have been widely used for modeling compounds, proteins, functional interactions, etc. A common task that divides these graphs into different categories, known as graph classification, highly relies on the quality of the representations of graphs. With the advance in graph neural networks, message-passing-based methods are adopted to iteratively aggregate neighborhood information for better graph representations. These methods, though powerful, still suffer from some shortcomings. The first challenge is that pooling-based methods in graph neural networks may sometimes ignore the part-whole hierarchies naturally existing in graph structures. These part-whole relationships are usually valuable for many molecular function prediction tasks. The second challenge is that most existing methods do not take the heterogeneity embedded in graph representations into consideration. Disentangling the heterogeneity will increase the performance and interpretability of models. This paper proposes a graph capsule network for graph classification tasks with disentangled feature representations learned automatically by well-designed algorithms. This method is capable of, on the one hand, decomposing heterogeneous representations to more fine-grained elements, whilst on the other hand, capturing part-whole relationships using capsules. Extensive experiments performed on several public-available biochemistry datasets demonstrated the effectiveness of the proposed method, compared with nine state-of-the-art graph learning methods.

Recent years have seen rapid progress in network representation learning, which removes the need for burdensome feature engineering and facilitates downstream network-based tasks. In reality, networks often exhibit heterogeneity, which means there may exist multiple types of nodes and interactions. Heterogeneous networks raise new challenges to representation learning, as the awareness of node and edge types is required. In this article, we study a basic building block of general heterogeneous networks, the heterogeneous networks with two types of nodes. Many problems can be solved by decomposing general heterogeneous networks into multiple bipartite ones. Recently, to overcome the demerits of non-metric measures used in the embedding space, metric learning-based approaches have been leveraged to tackle heterogeneous network representation learning. These approaches first generate triplets of samples, in which an anchor node, a positive counterpart, and a negative one co-exist, and then try to pull closer positive samples and push away negative ones. However, when dealing with heterogeneous networks, even the simplest two-typed ones, triplets cannot simultaneously involve both positive and negative samples from different parts of networks. To address this incompatibility of triplet-based metric learning, in this article, we propose a novel quintuple-based method for learning node representations in bipartite heterogeneous networks. Specifically, we generate quintuples that contain positive and negative samples from two different parts of networks. And we formulate two learning objectives that accommodate quintuple-based learning samples, a proximity-based loss that models the relations in quintuples by sigmoid probabilities and an angular loss that more robustly maintains similarity structures. In addition, we also parameterize feature learning by using one-dimensional convolution operators around nodes’ neighborhoods. Compared with eight methods, extensive experiments on two downstream tasks manifest the effectiveness of our approach.

Multilabel annotation is a critical step to generate training sets when learning classification models in various application domains, but asking domain experts to provide labels is usually time-consuming and expensive, which cannot meet the current requirement of the fast evolution of the models in the big data era. Although crowdsourcing provides a fast solution to acquire labels for multilabel learning, it faces the risk of high data acquisition cost and low label quality. This article proposes a novel one-coin label-dependent active crowdsourcing (OCLDAC) method to iteratively query noisy labels from crowd workers and learn multilabel classification models. In each iteration of active learning, integrated labels of instances are first inferred by a novel one-coin label-dependent model, which utilizes a mixture of multiple independent Bernoulli distributions to explore and exploit correlations among the labels to increase the accuracy of truth inference. Then, instances, labels, and workers are selected according to the novel strategies that incorporate the distribution of noisy labels, the prediction probability of learning models, label correlations, and the reliability of crowd workers. Simulations on eight multilabel datasets and evaluation on one real-world crowdsourcing dataset consistently show that the proposed OCLDAC significantly outperforms the state-of-the-art methods and their variants.

In recent years, the field of unsupervised graph representation learning has witnessed the emergence of graph contrastive learning (GCL) as a highly successful approach. GCL excels in learning graph representations by effectively bringing positive sample pairs into proximity while simultaneously pushing negative sample pairs apart in the representation space, without any manual labels. Graph-structured data to be learned inherently exhibits a critical hierarchical structure, which is crucial for organizing and managing graphs. Leveraging this attribute enhances the accuracy of graph representation outcomes. However, current GCL methods tend to overlook the hierarchical structure, which can result in sampling bias during node selection. Nodes of the same semantics can potentially be sampled as negative pairs. To overcome these limitations, we present a novel framework, Hierarchical Graph Contrastive Learning via Debiasing Noise Samples with Adaptive Repelling Ratio (HGClear). Our framework enables the simultaneous learning of node representations and the graph hierarchy in an end-to-end manner. During the process of method design, we discovered that the accuracy of node category prediction significantly affects representation results. To remove the bias caused by noise samples, we introduced a module to handle boundary nodes (i.e., noise samples) that are vulnerable to mislabeling. Specifically, we introduce a hierarchy detection module that captures both coarse-grained views and category attributes of nodes. Leveraging these results, we can identify boundary nodes and establish varying repelling ratios based on category labels, replacing the conventional temperature coefficient in the contrastive loss. Simultaneously incorporating an intra-view node contrast module not only eliminates the bias resulting from noise samples but also enhances the uniqueness of node representations. Numerous experiments on node classification datasets show that HGClear produces encouraging results and outperforms some state-of-the-art methods.

In the current era of information overload, service recommendations have emerged as a valuable tool for enhancing the user experience. Among them, social recommendation models have shown promising results by incorporating social relationships to improve representation learning. However, most of these models lack fine-grained modeling of social user behavior, leading to a unified representation of users and a loss of expressiveness in user representations. To address this issue, we propose DisenHGCF, a new social recommendation approach based on disentangled hypergraph collaborative filtering, to disentangle the representations of users and items at the granularity of social users’ intents. This approach aims to provide a more nuanced understanding of the user, leading to more accurate and personalized recommendations. To be specific, DisenHGCF leverages hypergraphs to represent the complex relationships among users, friends, and items. By using the hypergraph disentangling module based on attention, it is able to disentangle the user’s intents and generate users representations in cooperating their intents for recommendation tasks. Additionally, a contrastive learning task based on intent-weight perturbation is designed to enhance representational learning. The experimental results obtained from the BeiBei and Beidian datasets demonstrate the superiority of our proposed approach in comparison to previous baseline methods, as evidenced by higher Recall and NDCG scores.

Exploiting social networks is expected to enhance the performance of recommender systems when interaction information is sparse. Existing social recommendation models focus on modeling multi-graph structures and then aggregating the information from these multiple graphs to learn potential user preferences. However, these methods often employ complex models and redundant parameters to get a slight performance improvement. Contrastive learning has been widely researched as an effective paradigm in the area of recommendation. Most existing contrastive learning-based models usually focus on constructing multi-graph structures to perform graph augmentation for contrastive learning. However, the effect of graph augmentation on contrastive learning is inconclusive. In view of these challenges, in this work, we propose a contrastive learning based graph convolution network for social recommendation (CLSR), which integrates information from both the social graph and the interaction graph. First, we propose a fusion-simplified method to combine the social graph and the interaction graph. Technically, on the basis of exploring users’ interests by interaction graph, we further exploit social connections to alleviate data sparsity. By combining the user embeddings learned through two graphs in a certain proportion, we can obtain user representation at a finer granularity. Meanwhile, we introduce a contrastive learning framework for multi-graph network modeling, where we explore the feasibility of constructing positive and negative samples of contrastive learning by conducting data augmentation on embedding representations. Extensive experiments verify the superiority of CLSR’s contrastive learning framework and fusion-simplified method of integrating social relations.

When crowdsourced workers perform annotation tasks in an unfamiliar domain, their accuracy will dramatically decline due to the lack of expertise. Transferring knowledge from relevant domains can form a better representation for instances, which benefits the estimation of workers' expertise in truth inference models. However, existing knowledge transfer processes for crowdsourcing require a considerable number of well-collected instances in source domains. This paper proposes a novel truth inference model for crowdsourcing, where (meta-)knowledge is transferred by meta-learning and used in the estimation of workers' expertise. Our preliminary experiments demonstrate that the meta-knowledge transfer significantly reduces instances in source domains and increases the accuracy of truth inference.

Resolving community structure in networks is of significant benefit for both scientific inquiries and practical applications. Recently, deep neural networks have demonstrated excellent performance on various graph mining tasks, including community detection. However, there are still some challenges that are urgent to be addressed. First, being frequently formulated in an unsupervised setting, community detection has been proved to be more resistant to the advantages of end-to-end learning. Many deep methods carry out clustering algorithms after the acquisition of node representations. Second, very few studies consider the heterogeneity of a large number of real-world networks in end-to-end community detection. For instance, the building blocks of general heterogeneous networks are the bipartite model, which is a ubiquitous structure where two types of nodes co-exist. In view of these challenges, we study the end-to-end community co-partition of two types of nodes in bipartite networks. Specifically, we extend both spectral and spatial graph convolution operators to bipartite structures for node feature encoding. Then we formulate a novel loss function with a modularity-based objective, as well as two collapsed regularizations for producing more informative community assignment matrices. Co-partitions of nodes can be directly achieved by optimization with stochastic gradient descent under the proposed framework. Comprehensive empirical analysis, compared with various types of classic and deep methods, demonstrates the efficacy and the scalability of the proposed method.

Achieving high-accuracy relation extraction usually requires a large well-annotated corpus for model training, which is extremely time-consuming and labor-intensive. Consequently, distant supervision, implemented as a multi-instance learning paradigm, was proposed to build models from bags using automated labeled corpora, where massive noises are unavoidably introduced in the bag labels, deteriorating the model performance. To deal with the above problem, this paper proposes novel multi-instance active learning for relation extraction, which aims to improve the classification performance of the learned models under the premise of using as few labeled training samples as possible. The proposed active learning strategies for bag selection are based on the similarity between bags, the improved Fisher matrix, and the combination of the two (hybrid), respectively. Extensive experiment results on two real-world datasets under three base learners consistently show that our proposed active learning methods archive better performance for relation extraction, compared with three baseline methods. Furthermore, the hybrid bag selection strategy performs the best.

Attributed graph clustering is a fundamental yet challenging task for graph analysis, which is of significance in the real world. Recently, graph contrastive learning have demonstrated excellent performance on clustering task. However, we observe that existing methods have the following three defects: a) an end-to-end graph clustering framework based on contrastive clustering has not yet been implemented, b) they require a pretext task to obtain initial labels, c) and most methods lack a specific loss function for clustering processes. To address above issues, we innovatively propose Attributed Graph Contrastive Clustering (AGCC), which is a novel self-supervised attributed graph clustering framework. AGCC can directly produce a clustering assignment matrix without calculating target distribution matrix. Specifically, in our method, an effective module based on contrastive graph representation learning is being leveraged to extract node embeddings by node features and topological structure. In the phase of data augmentation, the module can adaptively mask node features and drops edges. Meanwhile, another reliable contrastive clustering module generates clustering results through contrasting the representation of each cluster. Comprehensive empirical analysis, compared with various types of classic and deep methods, demonstrates the efficacy and the scalability of the proposed method.

Real-world networks often show heterogeneity. A frequently encountered type is the bipartite heterogeneous structure, in which two types of nodes and three types of edges exist. Recently, much attention has been devoted to representation learning in these networks. One of the essential differences between heterogeneous and homogeneous learning is that the former structure requires methods to possess awareness to node and edge types. Most existing methods, including metapath-based, proximity-based and graph neural network-based, adopt inner product or vector norms to evaluate the similarities in embedding space. However, these measures either violates the triangle inequality, or show severe sensitivity to scaling transformation. The limitations often hinder the applicability to real-world problems. In view of this, in this paper, we propose a novel angle-based method for bipartite heterogeneous network representation. Specifically, we first construct training sets by generating quintuples, which contain both positive and negative samples from two different parts of networks. Then we analyze the quintuple-based problem from a geometry perspective, and transform the comparisons between preferred and non-preferred samples to the comparisons of angles. In addition, we utilize convolution modules to extract node features. A hinge loss, as the final objective, is proposed to relax the angular constraint for learning. Extensive experiments for two typical tasks show the efficacy of the proposed method, comparing with eight competitive methods.

Embedding-based entity alignment, which represents knowledge graphs as low-dimensional embeddings and finds entities in different knowledge graphs that semantically represent the same real-world entity by measuring the similarities between entity embeddings, has achieved promising results. However, existing methods are still challenged by the error accumulation of embeddings along multi-step paths and the semantic information loss. This paper proposes a novel embedding-based entity alignment method that iteratively aligns both entities and relations with high similarities as training data. Newly-aligned entities and relations are used to calibrate the corresponding embeddings in the unified embedding space, which reduces the error accumulation. To reduce the negative impact of semantic information loss, the authors propose to use relation structural similarity instead of embedding similarity to align relations. Experimental results on five widely used real-world datasets show that the proposed method significantly outperforms several state-of-the-art methods for entity alignment.

With the advancements of the Health 2.0 technology, large-scale healthcare services are available online. Recommender systems for healthcare services have emerged for decision assistance. Most existing collaborative recommendation algorithms only mine global interactions while failing to capture the local different information of users or items. Besides, privacy concern is another significant problem to be considered in recommender systems for healthcare services. In this article, a privacy-aware factorization-based hybrid method is proposed for healthcare service recommendations. For better modeling of user preferences and service features, multiview embeddings of users and healthcare services are learned. Besides, we address the privacy problem by integrating local differential privacy and locality-sensitive hashing techniques into the recommendation model for privacy-aware neighbor searching. The final prediction is made based on a stochastic gradient descent learning-based hybrid collaborative model. Experiments demonstrate the effectiveness of the proposed method in both recommendation performance and privacy concerns.

To effectively classify graph instances, graph neural networks need to have the capability to capture the part-whole relationship existing in a graph. A capsule is a group of neurons representing complicated properties of entities, which has shown its advantages in traditional convolutional neural networks. This paper proposed novel Capsule Graph Neural Networks that use the EM routing mechanism (CapsGNNEM) to generate high-quality graph embeddings. Experimental results on a number of real-world graph datasets demonstrate that the proposed CapsGNNEM outperforms nine state-of-the-art models in graph classification tasks.

Keyphrase annotation task aims to retrieve the most representative phrases that express the essential gist of documents. In reality, some phrases that best summarize documents are often absent from the original text, which motivates researchers to develop generation methods, being able to create phrases. Existing generation approaches usually adopt the encoder-decoder framework for sequence generation. However, the widely-used recurrent neural network might fail to capture long-range dependencies among items. In addition, intuitively, as keyphrases are likely to correlate with topical words, some methods propose to introduce topic models into keyphrase generation. But they hardly leverage the global information of topics. In view of this, we employ the Transformer architecture with the pre-trained BERT model as the encoder-decoder framework for keyphrase generation. BERT and Transformer are demonstrated to be effective for many text mining tasks. But they have not been extensively studied for keyphrase generation. Furthermore, we propose a topic attention mechanism to utilize the corpus-level topic information globally for keyphrase generation. Specifically, we propose BertTKG, a keyphrase generation method that uses a contextualized neural topic model for corpus-level topic representation learning, and then enhances the document representations learned by pre-trained language model for better keyphrase decoding. Extensive experiments conducted on three public datasets manifest the superiority of BertTKG.

Graph neural networks have shown superior performance in representing graph-structured data. However, directly applying most of existing methods to heterogeneous graph learning tasks may be ill-suited or infeasible, since multiple different types of nodes, edges and attributes challenge the methods originally developed for homogeneous networks. Given this situation, heterogeneous graph embedding recently has gained much research attention. Existing heterogeneous graph embedding methods often adopt random walk-based sampling techniques to model both heterogeneity and rich attribute information. These methods strongly rely on customized sample generation strategies and usually consume a lot of memory. To avoid these restrictions, we present a novel model based on edge-aware neighborhood convolution which is designed to extract features from the neighborhood of different types of nodes. The first and second-order proximity-based objectives are formulated for learning node representations. On the task of link prediction and recommendation, experiments on five heterogeneous graph datasets show the effectiveness of our model.

When acquiring labels from crowdsourcing platforms, a task may be designed to include multiple labels and the values of each label may belong to a set of various distinct options, which is the so-called multi-class multi-label annotation. To improve the quality of labels, requesters usually let one task be independently completed by a group of heterogeneous crowdsourced workers. Then, the true values of the multiple labels of each task are inferred from these repeated noisy labels. In this paper, we propose two novel probabilistic models MCMLI and MCMLD to address the multi-class multi-label inference problem in crowdsourcing. MCMLI assumes that the labels of each task are mutually independent and MCMLD utilizes a mixture of multiple independently multinoulli distributions to capture the correlation among the labels. Both models can jointly infer multiple true labels of each instance as well as estimate the reliability of crowdsourced workers modeled by a set of confusion matrices with an expectation-maximization algorithm. Experiments with three typical crowdsourcing scenarios and a real-world dataset show that our proposed models significantly outperform existing competitive alternatives. When the labels are strongly correlated, MCMLD substantially outperforms MCMLI. Furthermore, our models can be easily simplified to the one-coin models, which show more advantageous when errors are uniformly distributed, or labels are sparse.

Learning from crowds often performs in an active learning paradigm, aiming to improve learning performance quickly as well as to reduce labeling cost by selecting proper workers to (re)label critical instances. Previous active learning methods for learning from crowds do not have any proactive mechanism to effectively improve the reliability of workers, which prevents to obtain steadily rising learning curves. To help workers improve their reliability while performing tasks, this paper proposes a novel Interactive Learning framework with Proactive Cognitive Enhancement (ILPCE) for crowd workers. The ILPCE framework includes an interactive learning mechanism: When crowd workers perform labeling tasks in active learning, their cognitive ability to the specific domain can be enhanced through learning the exemplars selected by a psychological model-based machine teaching method. A novel probabilistic truth inference model and an interactive labeling scheme are proposed to ensure the effectiveness of the interactive learning mechanism and the performance of learning models can be simultaneously improved through a fast and low-cost way. Experimental results on three real-world learning tasks demonstrate that our ILPCE significantly outperforms five representative state-of-the-art methods.

The task of link prediction for knowledge graphs is to predict missing relationships between entities. Knowledge graph embedding, which aims to represent entities and relations of a knowledge graph as low dimensional vectors in a continuous vector space, has achieved promising predictive performance. If an embedding model can cover different types of connectivity patterns and mapping properties of relations as many as possible, it will potentially bring more benefits for link prediction tasks. In this paper, we propose a novel embedding model, namely LineaRE, which is capable of modeling four connectivity patterns (i.e., symmetry, antisymmetry, inversion, and composition) and four mapping properties (i.e., one-to-one, one-to-many, many-to-one, and many-to-many) of relations. Specifically, we regard knowledge graph embedding as a simple linear regression task, where a relation is modeled as a linear function of two low-dimensional vector-presented entities with two weight vectors and a bias vector. Since the vectors are defined in a real number space and the scoring function of the model is linear, our model is simple and scalable to large knowledge graphs. Experimental results on multiple widely used real-world datasets show that the proposed LineaRE model significantly outperforms existing state-of-the-art models for link prediction tasks.

Summary With the explosive growth of cloud services, how to design effective recommendation models has become more and more important. Temporal information has been proved to be an important factor affecting recommendation performance. Actually, both of user behaviors and QoS performance of services are time‐sensitive, especially in dynamic cloud environment. However, most existing collaborative recommendation methods seldom consider temporal influence to QoS performance. Furthermore, with the ever‐increasing number of security threats in clouds, privacy preservation becomes an important problem to be addressed in recommender systems. Based on these observations, in this paper, we propose a temporal‐aware and sparsity‐tolerant hybrid collaborative recommendation method with privacy preservation, where tensor factorization‐based CF model is integrated into neighborhood‐based CF model to achieve improved recommendation performance. Specifically, temporal influence is considered into recommendation model by distinguishing short‐term QoS metrics from long‐term QoS metrics. A privacy‐aware time aggregation mechanism is adopted to preserve the sensitive time information of users. Moreover, to deal with the sparsity problem, a sparsity‐oriented data QoS rating prediction mechanism based on tensor factorization technique is applied to predict the missing or unrated QoS rating data. Finally, predictions are made based on both stable and temporal nearest neighbors. Experiments are conducted to demonstrate the prediction performance of our proposal.

With crowdsourcing systems, labels can be obtained with low cost, which facilitates the creation of training sets for prediction model learning. However, the labels obtained from crowdsourcing are often imperfect, which brings great challenges in model learning. Since 2008, the machine learning community has noticed the great opportunities brought by crowdsourcing and has developed a large number of techniques to deal with inaccuracy, randomness, and uncertainty issues when learning with crowdsourcing. This paper summarizes the technical progress in this field during past eleven years. We focus on two fundamental issues: the data (label) quality and the prediction model quality. For data quality, we summarize ground truth inference methods and some machine learning based methods to further improve data quality. For the prediction model quality, we summarize several learning paradigms developed under the crowdsourcing scenario. Finally, we further discuss several promising future research directions to attract researchers to make contributions in crowdsourcing.

Distributed machine learning has gained lots of attention due to the rapid growth of data. In this paper, we focus regularized empirical risk minimization problems, and propose two novel Distributed Accelerated Alternating Direction Method of Multipliers (D-A2DM2) algorithms for distributed classification. Based on the framework of Alternating Direction Method of Multipliers (ADMM), we decentralize the distributed classification problem as a global consensus optimization problem with a series of sub-problems. In D-A2DM2, we exploit ADMM with variance reduction for sub-problem optimization in parallel. Taking global update and local update into consideration respectively, we propose two acceleration mechanisms in the framework of D-A2DM2. In particular, inspired by Nesterov's accelerated gradient descent, we utilize it for global update to further improve time efficiency. Moreover, we also introduce Nesterov's acceleration for local update, and develop the corrected local update and symmetric dual update to accelerate the convergence with only a little change in the computational effort. Theoretically, D-A2DM2 has a linear convergence rate. Empirically, experimental results show that D-A2DM2 converge faster than existing distributed ADMM-based classification, and could be a highly efficient algorithm for practical use.

Abstract Various studies have demonstrated that convolutional neural networks (CNNs) can be directly applied to different levels of text embedding, such as character‐, word‐, or document‐levels. However, the effectiveness of different embeddings is limited in the reported result and there is a lack of clear guidance on some aspects of their use, including choosing the proper level of embedding and switching word semantics from one domain to another when appropriate. In this paper, we propose a new architecture of CNN based on multiple representations for text classification, by constructing multiple planes so that more information can be dumped into the networks, such as different parts of text obtained through named entity recognizer or part‐of‐speech tagging tools, different levels of text embedding, or contextual sentences. Various large‐scale, domain‐specific datasets are used to validate the proposed architecture. Tasks analyzed include ontology document classification, biomedical event categorization, and sentiment analysis, showing that multi‐representational CNNs, which learns to focus attention to specific representations of text, can obtain further gains in performance over state‐of‐the‐art deep neural network models.

With the crowdsourcing of small tasks becoming easier, it is possible to obtain non-expert/imperfect labels at low cost. With low-cost imperfect labeling, it is straightforward to collect multiple labels for the same data items. This paper proposes strategies of utilizing these multiple labels for supervised learning, based on two basic ideas: majority voting and pairing. We show several interesting results based on our experiments. (i) The strategies based on the majority voting idea work well under the situation where the certainty level is high. (ii) On the contrary, the pairing strategies are more preferable under the situation where the certainty level is low. (iii) Among the majority voting strategies, soft majority voting can reduce the bias and roughness, and perform better than majority voting. (iv) Pairing can completely avoid the bias by having both sides (potentially correct and incorrect/noisy information) considered. Beta estimation is applied to reduce the impact of the noise in pairing. Our experimental results show that pairing with Beta estimation always performs well under different certainty levels. (v) All strategies investigated are labeling quality agnostic strategies for real-world applications, and some of them perform better than or at least very close to the gnostic strategies.

Traditional learning from crowdsourced labeled data consists of two stages: inferring true labels for instances from their multiple noisy labels and building a learning model using these instances with the inferred labels. This straightforward two-stage learning scheme suffers from two weaknesses: (1) the accuracy of inference may be very low; (2) useful information may be lost during inference. In this paper, we proposed a novel ensemble method for learning from crowds. Our proposed method is a meta-learning scheme. It first uses a bootstrapping process to create M sub-datasets from an original crowdsourced labeled dataset. For each sub-dataset, each instance is duplicated with different weights according to the distribution and class memberships of its multiple noisy labels. A base classifier is then trained from this extended sub-dataset. Finally, unlabeled instances are predicted by aggregating the outputs of these M base classifiers. Because the proposed method gets rid of the inference procedure and uses the full dataset to train learning models, it preserves the useful information for learning as much as possible. Experimental results on nine simulated and two real-world crowdsourcing datasets consistently show that the proposed ensemble learning method significantly outperforms five state-of-the-art methods.

With online crowdsourcing platforms, labels can be acquired at relatively low costs from massive nonexpert workers. To improve the quality of labels obtained from these imperfect crowdsourced workers, we usually let different workers provide labels for the same instance. Then, the true labels for all instances are estimated from these multiple noisy labels. This traditional general-purpose label aggregation process, solely relying on the collected noisy labels, cannot significantly improve the accuracy of integrated labels under a low labeling quality circumstance. This paper proposes a novel bilayer collaborative clustering (BLCC) method for the label aggregation in crowdsourcing. BLCC first generates the conceptual-level features for the instances from their multiple noisy labels and infers the initially integrated labels by performing clustering on the conceptual-level features. Then, it performs another clustering on the physical-level features to form the estimations of the true labels on the physical layer. The clustering results on both layers can facilitate in tracking the changes in the uncertainties of the instances. Finally, the initially integrated labels that are likely to be wrongly inferred on the conceptual layer can be addressed using the estimated labels on the physical layer. The clustering processes on both layers can keep providing guidance information for each other in the multiple label remedy rounds. The experimental results on 12 real-world crowdsourcing data sets show that the performance of the proposed method in terms of accuracy is better than that of the state-of-the-art methods.

When acquiring labels from crowdsourcing platforms, a task may be designed to include multiple labels and the values of each label may belong to a set of various distinct options, which is the so-called multi-class multi-label annotation. To improve the quality of labels, one task is independently completed by a group of heterogeneous crowdsourced workers. Then, the true values of the multiple labels of each task are inferred from these repeated noisy labels. In this paper, we propose a novel probabilistic method, which includes a multi-class multi-label dependency (MCMLD) model, to address this problem. The proposed method assumes that the label-correlation exists in both unknown true labels and noisy crowdsourced labels. Thus, it introduces a mixture of multiple independently multinoulli distributions to capture the correlation among the labels. Finally, the unknown true values of the multiple labels of each task, together with a set of confusion matrices modeling the reliability of the workers, can be jointly inferred through an EM algorithm. Experiments with three simulated typical crowdsourcing scenarios and a real-world dataset consistently show that our proposed MCMLD method significantly outperforms several competitive alternatives. Furthermore, if the labels are strongly correlated, the advantage of MCMLD will be more remarkable.

Crowdsourcing systems provide a cost effective and convenient way to collect labels, but they often fail to guarantee the quality of the labels. This paper proposes a novel framework that introduces noise correction techniques to further improve the quality of integrated labels that are inferred from the multiple noisy labels of objects. In the proposed general framework, information about the qualities of labelers estimated by a front-end ground truth inference algorithm is utilized to supervise subsequent label noise filtering and correction. The framework uses a novel algorithm termed adaptive voting noise correction (AVNC) to precisely identify and correct the potential noisy labels. After filtering out the instances with noisy labels, the remaining cleansed data set is used to create multiple weak classifiers, based on which a powerful ensemble classifier is induced to correct these noises. Experimental results on eight simulated data sets with different kinds of features and two real-world crowdsourcing data sets in different domains consistently show that: 1) the proposed framework can improve label quality regardless of inference algorithms, especially under the circumstance that each instance has a few repeated labels and 2) since the proposed AVNC algorithm considers both the number of and the probability of potential label noises, it outperforms the state-of-the-art noise correction algorithms.

This paper proposes a novel general label aggregation method for both binary and multi-class labeling in crowdsourcing, namely Bi-Layer Clustering (BLC), which clusters two layers of features - the conceptual-level and the physical-level features - to infer true labels of instances. BLC first clusters the instances using the conceptual-level features extracted from their multiple noisy labels and then performs clustering again using the physical-level features. It can facilitate tracking the uncertainty changes of the instances, so that the integrated labels that are likely to be falsely inferred on the conceptual layer can be easily corrected using the estimated labels on the physical layer. Experimental results on two real-world crowdsourcing data sets show that BLC outperforms seven state-of-the-art methods.

Multi‐label image classification has attracted considerable attention in machine learning recently. Active learning is widely used in multi‐label learning because it can effectively reduce the human annotation workload required to construct high‐performance classifiers. However, annotation by experts is costly, especially when the number of labels in a dataset is large. Inspired by the idea of semi‐supervised learning, in this study, the authors propose a novel, semi‐supervised multi‐label active learning (SSMAL) method that combines automated annotation with human annotation to reduce the annotation workload associated with the active learning process. In SSMAL, they capture three aspects of potentially useful information – classification prediction information, label correlation information, and example spatial information – and they use this information to develop an effective strategy for automated annotation of selected unlabelled example‐label pairs. The experimental results obtained in this study demonstrate the effectiveness of the authors' proposed approach.

Multilabel image classification has been a hot topic in the field of computer vision and image understanding in recent years. To achieve better classification performance with fewer labeled images, multilabel active learning is used for this scenario. Several active learning methods have been proposed for multilabel image classification. However, all of them assume that either all training images have complete labels or label correlations are given at the beginning. These two assumptions are unrealistic. In fact, it is very difficult to obtain complete labels for each example, in particular when the size of labels in a multilabel dataset is very large. Typically, only partial labels are available. This is one type of “weak label” problem. To solve this weak label problem inside multilabel active learning, this paper proposes a novel solution called AE-WLMAL. AE-WLMAL explores conditional label correlations on the weak label problem with the help of input features and then utilizes label correlations to construct a unified sampling strategy and evaluate the informativeness of each example-label pair in a multilabel dataset for active sampling. In addition, a pruning strategy is adopted to further improve its computation efficiency. Moreover, AE-WLAML exploits label correlations to infer labels for unlabeled images, which further reduces human labeling cost. Our experimental results on seven real-world datasets show that AE-WLMAL consistently outperforms existing approaches.

Due to low quality of crowdsourced labelers, the integrated label of each example is usually inferred from its multiple noisy labels provided by different labelers. This paper proposes a novel algorithm, Ground Truth Inference using Clustering (GTIC), to improve the quality of integrated labels for multi-class labeling. For a K labeling case, GTIC utilizes the multiple noisy label sets of examples to generate features. Then, it uses a K-Means algorithm to cluster all examples into K different groups, each of which is mapped to a specific class. Examples in the same cluster are assigned a corresponding class label. We compare GTIC with four existing multi-class ground truth inference algorithms, majority voting (MV), Dawid & Skene's (DS), ZenCrowd (ZC) and Spectral DS (SDS), on one synthetic and eight real-world datasets. Experimental results show that the performance of GTIC is significantly superior to the others in terms of both accuracy and M-AUC. Besides, the running time of GTIC is about twenty times faster than EM-based complicated inference algorithms.

This paper proposes a novel framework that introduces noise correction techniques to further improve label quality after ground truth inference in crowdsourcing. In the framework, an adaptive voting noise correction algorithm (AVNC) is proposed to identify and correct the most likely noises with the help of estimated qualities of labelers provided by the ground truth inference. The experimental results on two real-world datasets show that (1) the framework can improve label quality regardless of inference algorithms, especially under the circumstance that each example has a few noisy labels; and (2) since the algorithm AVNC considers both the number of and the probability of potential noises, it outperforms a baseline noise correction algorithm.

The important task of correcting label noise is addressed infrequently in literature. The difficulty of developing a robust label correction algorithm leads to this silence concerning label correction. To break the silence, we propose two algorithms to correct label noise. One utilizes self-training to re-label noise, called Self-Training Correction (STC). Another is a clustering-based method, which groups instances together to infer their ground-truth labels, called Cluster-based Correction (CC). We also adapt an algorithm from previous work, a consensus-based method called Polishing that consults with an ensemble of classifiers to change the values of attributes and labels. We simplify Polishing such that it only alters labels of instances, and call it Polishing Labels (PL). We experimentally compare our novel methods with Polishing Labels by examining their improvements on the label qualities, model qualities, and AUC metrics of binary and multi-class data sets, and ultimately conclude that only CC can significantly improve label qualities, model qualities, and AUC metrics consistently. STC and PL can improve these metrics in some cases, but not as reliably. Hence, our Cluster-based Correction method is the best.

We investigate the methods of improving data quality, in terms of label accuracy, in the context of image labeling in crowdsourcing. First, we look at three consensus methods for inferring a ground-truth label from the multiple noisy labels obtained from crowdsourcing, i.e., Majority Voting (MV), Dawid Skene (DS), and KOS. We then apply three noise correction methods to correct labels inferred by these consensus methods, i.e., Polishing Labels (PL), Self-Training Correction (STC), and Cluster Correction (CC). Our experimental results show that the noise correction methods improve the labeling quality significantly.

With various online crowdsourcing platforms, it is easy to collect multiple labels for the same examples from the crowd. Consensus integration algorithms can infer the estimated ground truths from the multiple label sets of these crowdsourcing datasets. However, it couldn't be avoided that these integrated (estimated) labels still contain noises. In order to further improve the performance of a model learned from data with these integrated labels, we propose an active learning framework to further improve the data quality, such that to improve the model quality, through acquiring limited true labels from experts (the oracle). We further investigate two active learning strategies in terms of two uncertainty measures (i.e., CLUE and MUE) within the active learning framework. From our experimental results on eight simulation crowdsourcing datasets and four real-world crowdsourcing datasets with three popular consensus integration algorithms, we draw several conclusions as follows. (i) Our active learning framework with the input from the oracle significantly improves the generalization ability of the model learned from crowdsourcing data. (ii) Our two active learning strategies outperform a random active learning strategy.

With crowdsourcing systems, it is easy to collect multiple noisy labels for the same object for supervised learning. This dynamic annotation procedure fits the active learning perspective and accompanies the imbalanced multiple noisy labeling problem. This paper proposes a novel active learning framework with multiple imperfect annotators involved in crowdsourcing systems. The framework contains two core procedures: label integration and instance selection. In the label integration procedure, a positive label threshold (PLAT) algorithm is introduced to induce the class membership from the multiple noisy label set of each instance in a training set. PLAT solves the imbalanced labeling problem by dynamically adjusting the threshold for determining the class membership of an example. Furthermore, three novel instance selection strategies are proposed to adapt PLAT for improving the learning performance. These strategies are respectively based on the uncertainty derived from the multiple labels, the uncertainty derived from the learned model, and the combination method (CFI). Experimental results on 12 datasets with different underlying class distributions demonstrate that the three novel instance selection strategies significantly improve the learning performance, and CFI has the best performance when labeling behaviors exhibit different levels of imbalance in crowdsourcing systems. We also apply our methods to a real-world scenario, obtaining noisy labels from Amazon Mechanical Turk, and show that our proposed strategies achieve very high performance.

It can be easy to collect multiple noisy labels for the same object via Internet-based crowdsourcing systems. Labelers may have bias when labeling, due to lacking expertise, dedication, and personal preference. These cause Imbalanced Multiple Noisy Labeling. In most cases, we have no information about the labeling qualities of labelers and the underlying class distributions. It is important to design agnostic solutions to utilize these noisy labels for supervised learning. We first investigate how imbalanced multiple noisy labeling affects the class distributions of training sets and the performance of classification. Then, an agnostic algorithm Positive LAbel frequency Threshold (PLAT) is proposed to deal with the imbalanced labeling issue. Simulations on eight UCI data sets with different underlying class distributions show that PLAT not only effectively deals with the imbalanced multiple noisy labeling problems that off-the-shelf agnostic methods cannot cope with, but also performs nearly the same as majority voting under the circumstances without imbalance. We also apply PLAT to eight real-world data sets with imbalanced labels collected from Amazon Mechanical Turk, and the experimental results show that PLAT is efficient and better than other ground truth inference algorithms.

Multi-label image data is becoming ubiquitous. Image semantic understanding is typically formulated as a classification problem. This paper focuses on multi-label active learning for image classification. It first extends a traditional example based active learning method for multilabel active learning for image classification. Since the traditional example based active method doesn't work well, we propose a novel example-label based multi-label active learning method. Our experimental results on two image datasets demonstrate that the proposed method significantly reduces the labeling workload and improves the performance of the built classifier. Additionally, we conduct experiments on two other types of multi-label datasets for validating the versatility of our proposed method, and the experimental results show the consistent effect.

When labeling objects via Internet-based outsourcing systems, the labelers may have bias, because they lack expertise, dedication and personal preference. These reasons cause Imbalanced Multiple Noisy Labeling. To deal with the imbalance labeling issue, we propose an agnostic algorithm PLAT (Positive LAbel frequency Threshold) which does not need any information about quality of labelers and underlying class distribution. Simulations on eight real-world datasets with different underlying class distributions demonstrate that PLAT not only effectively deals with the imbalanced multiple noisy labeling problem that off-the-shelf agnostic methods cannot cope with, but also performs nearly the same as majority voting under the circumstances that labelers have no bias.

Internet-based crowdsourcing systems can be viewed as a kind of loosely coupled social networks. With these systems, it is easy to collect multiple noisy labels for the same object when conducting annotation for supervised learning. Because non-expert labelers lack expertise and dedication, and have strong personal preference, they may have bias when labeling. These cause Imbalanced Multiple Noisy Labeling. In this paper, we propose an agnostic algorithm Positive LAbel frequency Threshold (PLAT) to deal with imbalanced labeling. Because of the dynamics of social networks, in most cases no information about the qualities of labelers and underlying class distributions can be acquired. PLAT does not require prior knowledge of the labeling qualities of labelers, the underlying class distributions, and the level of labeling imbalance. Simulations on eight real-world datasets with different underlying class distributions demonstrate that PLAT not only effectively deals with the imbalanced multiple noisy labeling that off-the-shelf agnostic methods cannot cope with, but also performs nearly the same as majority voting under the circumstances that labelers have no bias.