Multimodal named entity recognition (MNER) aims to identify predefined entity types from text by incorporating auxiliary information such as images. Compared with traditional named entity recognition methods, MNER introduces visual context to recognize entities in ambiguous text. To more effectively utilize the visual modality for text understanding, most current approaches rely on static cross-modal feature alignment and fusion. However, when text-image correlations are weak, such strategies fail to align the semantic information between modalities. In addition, existing methods tend to model local and global semantic representations independently, lacking a unified framework that can jointly capture both local and global semantic information. To address these issues, we propose a unified semantic alignment and cross-graph fusion model framework (SAGF) for MNER. First, we introduce a multimodal adaptive semantic alignment mechanism that uses a trainable bilinear similarity function to adaptively align cross-modal semantic information under weak text-image correlation. Second, we propose a cross-modal graph fusion strategy. It combines local semantic representations based on attention mechanisms, with global structure features encoded in a unified graph structure. This joint local with global modeling enables the framework to capture semantic relations and contextual structures among entities. Experiments on two well-known social media datasets demonstrate the effectiveness of the SAGF model, achieving strong performance in MNER.

Contrastive learning for deep multi-view clustering aims to learn discriminative representations across multiple views. However, prevailing cluster-level alignment approaches fail to fully leverage cross-view consistency and complementarity, as they neglect instance-level semantic coherence. To address this limitation, we propose a novel bidirectional contrastive learning network for multi-view clustering. By simultaneously contrasting the inter-view semantic label matrix along the row and column directions (i.e., at instance-level and cluster-level), the labels of the same instance in different views are consistent, and instances assigned to the same cluster across different views remain consistent. Moreover, we use dual-channel MLPs to avoid information conflicts caused by bidirectional contrastive learning. The proposed framework also demonstrates strong generalization capability, serving as a plug-and-play module that can be seamlessly integrated with existing methods to improve their clustering performance. Extensive experiments on publicly available datasets demonstrate the superiority of our method over several state-of-the-art techniques.

Multi-label image classification aims to predict a set of labels that are present in an image. The key challenge of multi-label image classification lies in two aspects: modeling label correlations and utilizing spatial information. However, the existing approaches mainly calculate the correlation between labels according to co-occurrence among them. While the result is easily affected by the label noise and occasional co-occurrences. In addition, some works try to model the correlation between labels and spatial features, but the correlation among labels is not fully considered to model the spatial relationships among features. To address the above issues, we propose a novel cross-modality semantic guidance-based framework for multi-label image classification, namely CMSG. First, we design a semantic-guided attention (SGA) module, which applies the label correlation matrix to guide the learning of class-specific features, which implicitly models semantic correlations among labels. Second, we design a spatial-aware attention (SAA) module to extract high-level semantic-aware spatial features based on class-specific features obtained from the SGA module. The experiments carried out on three benchmark datasets demonstrate that our proposed method outperforms existing state-of-the-art algorithms on multi-label image classification.

Microservice architecture has sprung up over recent years for managing enterprise applications, due to its ability to independently deploy and scale services. Despite its benefits, ensuring the reliability and safety of a microservice system remains highly challenging. Existing anomaly detection algorithms based on a single data modality (i.e., metrics, logs, or traces) fail to fully account for the complex correlations and interactions between different modalities, leading to false negatives and false alarms, whereas incorporating more data modalities can offer opportunities for further performance gain. As a fresh attempt, we propose in this paper a semi-supervised graph-based anomaly detection method, MSTGAD, which seamlessly integrates all available data modalities via attentive multi-modal learning. First, we extract and normalize features from the three modalities, and further integrate them using a graph, namely MST (microservice system twin) graph, where each node represents a service instance and the edge indicates the scheduling relationship between different service instances. The MST graph provides a virtual representation of the status and scheduling relationships among service instances of a real-world microservice system. Second, we construct a transformer-based neural network with both spatial and temporal attention mechanisms to model the inter-correlations between different modalities and temporal dependencies between the data points. This enables us to detect anomalies automatically and accurately in real-time. Extensive experiments on two real-world datasets verify the effectiveness of our proposed MSTGAD method, achieving competitive performance against state-of-the-art approaches, with a 0.961 F1-score and an average increase of 4.85%. The source code of MST-GAD is publicly available at https://github.com/ant-research/microservice_system_twin_graph_based_anomaly_detection.

Multi-label learning deals with the problem that each instance is associated with multiple labels simultaneously, and many methods have been proposed by modeling label correlations in a global way to improve the performance of multi-label learning. However, the local label correlations and the influence of feature correlations are not fully exploited for multi-label learning. In real applications, different examples may share different label correlations, and similarly, different feature correlations are also shared by different data subsets. In this paper, a method is proposed for multi-label learning by modeling local label correlations and local feature correlations. Specifically, the data set is first divided into several subsets by a clustering method. Then, the local label and feature correlations, and the multi-label classifiers are modeled based on each data subset respectively. In addition, a novel regularization is proposed to model the consistency between classifiers corresponding to different data subsets. Experimental results on twelve real-word multi-label data sets demonstrate the effectiveness of the proposed method.

Computational color naming (CCN) aims to learn a mapping from pixels into semantic color names, e.g., red, green and blue. CCN has wide applications including color vision deficiency assistance and color image retrieval. Existing research on CCN mainly studies pixels collected under laboratory settings or studies images collected from the web. However, laboratory pixels are very limited such that the learned mapping may not generalize well on unseen pixels, and the mapping discovered from images is usually data-specific. In this paper, we aim to learn a universal mapping by studying pixels collected from the web. To this end, we formulate a novel classification problem that incorporates both the pixels and the RGB color model. The RGB color model is beneficial for learning the mapping because it characterizes the production of colors, e.g., the addition of red and green produces yellow. However, the characterization is rather qualitative. To solve this problem, we propose ColorMLP, which is a multilayer perceptron (MLP) embedded with graph attention networks (GATs). Here, the GATs are designed to capture color relations that we construct by referring to the RGB color model. In this way, the parameters of the MLP can be regularized to comply with the RGB model. We conduct comprehensive experiments to demonstrate the superiority of ColorMLP to alternative methods.To expand the application of CCN, we design a novel data augmentation method named partial color jitter (PCJ), which performs color jitter (CJ) on a subset of pixels belonging to the same color of an image. In this way, PCJ partially changes the color properties of images, thereby significantly increasing images’ diversity. We conduct extensive experiments on CIFAR10/100 and ImageNet datasets, showing that PCJ can consistently improve the classification performance. Our data and software can be found at https://https://github.com/yanzipei/CCN_and_ItsApp.

In-air handwriting is a new and more humanized human-computer interaction way. The existing in-air handwriting systems are mainly based on three-dimensional sensors, which are expensive, too large and not conducive for integration and application promotion. To solve this problem, this paper proposes a new in-air handwriting system using cheap and portable monocular camera which allows users writing freely in the air. Additionally we develop an end-to-end fingertip detection algorithm based on improved YOLOv5 algorithm to form in-air handwritten characters. Concretely, we first build a fingertip images dataset. After preprocessing and fingertip labeling, we use the dataset to train the improved YOLOv5 model, and then use the trained model to detect the coordinates of the fingertip in each video frame. After that, we connect the coordinates of the fingertip of each frame to form the character, and finally utilize the classifiers to recognize characters. The experimental results show that proposed in-air handwriting system allows user write freely in the air, and can obtain over 92 % in fingertip detection and character recognition.

This article tackles the problem of multilabel learning with missing labels. For this problem, it is widely accepted that label correlations can be used to recover the ground-truth label matrix. Most of the existing approaches impose the low-rank assumption on the observed label matrix to exploit label correlations by decomposing it into two matrices, which describe the latent factors of instances and labels, respectively. The quality of these latent factors highly influences the recovery of ground-truth labels and the construction of the multilabel classification model. In this article, we propose recovering the ground-truth label matrix by regularized matrix factorization. Specifically, the latent factors of instances are regularized by the local topological structure derived from the feature space, which can be further used to induce an effective multilabel model. Moreover, the latent factors of labels and the label correlations are mutually adapted via label manifold regularization. In this way, the recovery of the ground-truth label matrix and the construction of the multilabel classification model are optimized jointly and can benefit from the regularized matrix factorization. Extensive experimental studies show that the proposed approach significantly outperforms the state-of-the-art algorithms on both full-label and missing-label data.

Abstract Multi-label learning deals with data examples which are associated with multiple class labels simultaneously. Despite the success of existing approaches to multi-label learning, there is still a problem neglected by researchers, i.e., not only are some of the values of observed labels missing, but also some of the labels are completely unobserved for the training data. We refer to the problem asmulti-label learning with missing and completely unobserved labels, and argue that it is necessary to discover these completely unobserved labels in order to mine useful knowledge and make a deeper understanding of what is behind the data. In this paper, we propose a new approach named MCUL to solve multi-label learning with Missing and Completely Unobserved Labels. We try to discover the unobserved labels of a multi-label data set with a clustering based regularization term and describe the semantic meanings of them based on the label-specific features learned by MCUL, and overcome the problem of missing labels by exploiting label correlations. The proposed method MCUL can predict both the observed and newly discovered labels simultaneously for unseen data examples. Experimental results validated over ten benchmark datasets demonstrate that the proposed method can outperform other state-of-the-art approaches on observed labels and obtain an acceptable performance on the new discovered labels as well.

Existing multi-label learning (MLL) approaches mainly assume all the labels are observed and construct classification models with a fixed set of target labels (known labels). However, in some real applications, multiple latent labels may exist outside this set and hide in the data, especially for large-scale data sets. Discovering and exploring the latent labels hidden in the data may not only find interesting knowledge but also help us to build a more robust learning model. In this paper, a novel approach named DLCL (i.e., Discovering Latent Class Labels for MLL) is proposed which can not only discover the latent labels in the training data but also predict new instances with the latent and known labels simultaneously. Extensive experiments show a competitive performance of DLCL against other state-of-the-art MLL approaches.

Multilabel learning deals with examples having multiple class labels simultaneously. It has been applied to a variety of applications, such as text categorization and image annotation. A large number of algorithms have been proposed for multilabel learning, most of which concentrate on multilabel classification problems and only a few of them are feature selection algorithms. Current multilabel classification models are mainly built on a single data representation composed of all the features which are shared by all the class labels. Since each class label might be decided by some specific features of its own, and the problems of classification and feature selection are often addressed independently, in this paper, we propose a novel method which can perform joint feature selection and classification for multilabel learning, named JFSC. Different from many existing methods, JFSC learns both shared features and label-specific features by considering pairwise label correlations, and builds the multilabel classifier on the learned low-dimensional data representations simultaneously. A comparative study with state-of-the-art approaches manifests a competitive performance of our proposed method both in classification and feature selection for multilabel learning.

Binary Relevance is a well-known framework for multi-label classification, which considers each class label as a binary classification problem. Many existing multi-label algorithms are constructed within this framework, and utilize identical data representation in the discrimination of all the class labels. In multi-label classification, however, each class label might be determined by some specific characteristics of its own. In this paper, we seek to learn label-specific data representation for each class label, which is composed of label-specific features. Our proposed method LLSF can not only be utilized for multi-label classification directly, but also be applied as a feature selection method for multi-label learning and a general strategy to improve multi-label classification algorithms comprising a number of binary classifiers. Inspired by the research works on modeling high-order label correlations, we further extend LLSF to learn class-Dependent Labels in a sparse stackingway, denoted as LLSF-DL. It incorporates both second-order- and high-order label correlations. A comparative study with the state-of-the-art approaches manifests the effectiveness and efficiency of our proposed methods.

Binary relevance (BR) is a well-known framework for multi-label classification. It decomposes multi-label classification into binary (one-vs-rest) classification subproblems, one for each label. The BR approach is a simple and straightforward way for multi-label classification, but it still has several drawbacks. First, it does not consider label correlations. Second, each binary classifier may suffer from the issue of class-imbalance. Third, it can become computationally unaffordable for data sets with many labels. Several remedies have been proposed to solve these problems by exploiting label correlations between labels and performing label space dimension reduction. Meanwhile, inconsistency, another potential drawback of BR, is often ignored by researchers when they construct multi-label classification models. Inconsistency refers to the phenomenon that if an example belongs to more than one class label, then during the binary training stage, it can be considered as both positive and negative example simultaneously. This will mislead binary classifiers to learn suboptimal decision boundaries. In this paper, we seek to solve this problem by learning label specific features for each label. We assume that each label is only associated with a subset of features from the original feature set, and any two strongly correlated class labels can share more features with each other than two uncorrelated or weakly correlated ones. The proposed method can be applied as a feature selection method for multi-label learning and a general strategy to improve multi-label classification algorithms comprising a number of binary classifiers. Comparison with the state-of-the-art approaches manifests competitive performance of our proposed method.

In multi-label classification, labels often have correlations with each other. Exploiting label correlations can improve the performances of classifiers. Current multi-label classification methods mainly consider the global label correlations. However, the label correlations may be different over different data groups. In this paper, we propose a simple and efficient framework for multi-label classification, called Group sensitive Classifier Chains. We assume that similar examples not only share the same label correlations, but also tend to have similar labels. We augment the original feature space with label space and cluster them into groups, then learn the label dependency graph in each group respectively and build the classifier chains on each group specific label dependency graph. The group specific classifier chains which are built on the nearest group of the test example are used for prediction. Comparison results with the state-of-the-art approaches manifest competitive performances of our method.