Neural Collapse (NC) presents an elegant geometric structure that enables individual activations (features), class means and classifier (weights) vectors to reachoptimalinter-class separability during the terminal phase of training on abalanceddataset. Once shifted to imbalanced classification, such an optimal structure of NC can be readily destroyed by the notoriousminority collapse, where the classifier vectors corresponding to the minority classes are squeezed. In response, existing works mainly optimize classifiers in an effort to recover NC. However, we discover that this squeezing phenomenon is not only confined to classifier vectors but also occurs with class means. Consequently, reconstructing NC solely at the classifier aspect may be futile, as the class means remain compressed, leading to the violation of inherentself-dualityin NC (i.e., class means and classifier vectors converge mutually) and incidentally, an unsatisfactory collapse of individual activations towards the corresponding class means. To shake off these dilemmas, we present a unifiedAll-aroundNeuralCollapse framework (AllNC), aiming to comprehensively restore NC across multiple aspects including individual activations, class means and classifier vectors. We thoroughly analyze its effectiveness and verify its performance on multiple benchmark datasets as state-of-the-art in both balanced and imbalanced settings.

Multi-label image classification relies on a large-scale, well-maintained dataset, which may easily be mislabeled due to various subjective reasons. Existing methods for coping with noise usually focus on improving the model robustness in the case of single-label noise. However, compared with noisy single-label learning, noisy multi-label learning is more practical and challenging. To reduce the negative impact of noisy multi-annotations, we propose a universal approach for noisy multi-label learning (UNM). In UNM, we propose the label-wise embedding network which investigates the semantic alignment between label embeddings and their corresponding output features to learn robust feature representations. Meanwhile, mining the co-occurrence of multi-labels is also added to regularize the noisy network predictions. We cyclically change the fitting status of our label-wise embedding network to distinguish the noisy samples and generate pseudo labels for them. As a result, UNM provides an effective way to exploit the label-wise features and semantic label embeddings in noisy scenarios. To verify the generalizability of our method, we also test our method on Partial Multi-label Learning (PML) and Multi-label Learning with Missing Labels (MLML). Extensive experiments on benchmark datasets including Microsoft COCO, Pascal VOC, and Visual Genome explicitly validate the proposed method.

In open set recognition (OSR), almost all existing methods are designed specially for recognizing individual instances, even these instances are collectively coming in batch. Recognizers in decision either reject or categorize them to some known class using empirically-set threshold. Thus the decision threshold plays a key role. However, the selection for it usually depends on the knowledge of known classes, inevitably incurring risks due to lacking available information from unknown classes. On the other hand, a more realistic OSR system should NOT just rest on a reject decision but should go further, especially for discovering the hidden unknown classes among the reject instances, whereas existing OSR methods do not pay special attention. In this paper, we introduce a novel collective/batch decision strategy with an aim to extend existing OSR for new class discovery while considering correlations among the testing instances. Specifically, a collective decision-based OSR framework (CD-OSR) is proposed by slightly modifying the Hierarchical Dirichlet process (HDP). Thanks to HDP, our CD-OSR does not need to define the decision threshold and can implement the open set recognition and new class discovery simultaneously. Finally, extensive experiments on benchmark datasets indicate the validity of CD-OSR.

Federated Semi-Supervised Learning alleviates the necessity for fully labeled data in Federated Learning. However, it does not sufficiently prioritize model privacy or the personalized requirements of clients. To address these concerns, our key idea is to communicate no longer individual model parameters but their black-box models, implying to provide other clients with only an input-output interface of individual models. The communication mechanism is model-agnostic, thereby facilitating adaption to heterogeneous models for each client through customization. Consequently, we propose a framework called Personalized Federated Semi-Supervised Learning with Black-Box Models (B2PFSSL) to enhance the privacy of communication. To prevent negative knowledge transfer due to data heterogeneity, we design a two-stage strategy that filters at both the model and data levels, enabling clients to obtain large training datasets by including more high-quality pseudo-labeled data under conditions of scarce labeled data. The experimental results indicate that B2PFSSL achieves competitive performance while reducing the amount of information exposed during communication. Furthermore, it can foster productive collaboration among diverse model architectures in model heterogeneous Federated Learning.

Subspace clustering has attracted much attention because of its ability to group unlabeled high-dimensional data into multiple subspaces. Existing graph-based subspace clustering methods focus on either the sparsity of data affinity or the low rank of data affinity. Thus, the quality of data affinity plays an essential role in the performance of subspace clustering. However, the real-world data are generally high-dimensional, complex, and heterogeneous multi-source data, so that the data affinity learned by these methods cannot be completely dependent. Moreover, since these approaches always ignore the intrinsic structure of data, their grouping effect is relatively low. In this paper, we propose a novel unsupervised algorithm, called Structure-Aware Subspace Clustering (SASC), to address the above issues. SASC considers local and global correlation structures simultaneously to capture the intrinsic structure. Further, it integrates the captured structure into representation learning to gain a relatively precise data affinity. It is powerful to promote an all-around grouping effect and enhances the robustness and applicability of subspace clustering. Experiments on various benchmark datasets, including bioinformatics, handwritten digit, object image, and speech signal, demonstrate the effectiveness of the proposed algorithm.

Similarity-based method gives rise to a new class of methods for multi-label learning and also achieves promising performance. In this paper, we generalize this method, resulting in a new framework for classification task. Specifically, we unite similarity-based learning and generalized linear models to achieve the best of both worlds. This allows us to capture interdependencies between classes and prevent from impairing performance of noisy classes. Each learned parameter of the model can reveal the contribution of one class to another, providing interpretability to some extent. Experiment results show the effectiveness of the proposed approach on multi-class and multi-label data sets.

In open set recognition (OSR), almost all existing methods are designed specially for recognizing individual instances, even these instances are collectively coming in batch. Recognizers in decision either reject or categorize them to some known class using empirically-set threshold. Thus the decision threshold plays a key role. However, the selection for it usually depends on the knowledge of known classes, inevitably incurring risks due to lacking available information from unknown classes. On the other hand, a more realistic OSR system should NOT just rest on a reject decision but should go further, especially for discovering the hidden unknown classes among the reject instances, whereas existing OSR methods do not pay special attention. In this paper, we introduce a novel collective/batch decision strategy with an aim to extend existing OSR for new class discovery while considering correlations among the testing instances. Specifically, a collective decision-based OSR framework (CD-OSR) is proposed by slightly modifying the Hierarchical Dirichlet process (HDP). Thanks to HDP, our CD-OSR does not need to define the decision threshold and can implement the open set recognition and new class discovery simultaneously. Finally, extensive experiments on benchmark datasets indicate the validity of CD-OSR.

This paper presents the first study on Sign-aware Perturbations Regression (SaPR), where the observed response variables contain the aware sign (negative or positive) perturbations.In order to predict the non-perturbation response variables, we propose a novel parameter estimator SZOM (i.e.,Setting Zero Operator Method), which aims at taking full advantage of the aware perturbations information to correct the mistake values in the estimation process with computationally efficiency.In this paper, the two aspects of theoretical analysis are proposed to deeply understand our method.Firstly, we establish the perturbation parameter error upper bound and prove consistency guarantee in the linear regression scenario.Secondly, we introduce the generalization error bound for the proposed SZMO, which indicates that the error bound is related to the value and the number of negative and positive perturbations.The effectiveness of the proposed approach is well validated by the experimental results on both synthetic and real datasets.

Canonical correlation analysis (CCA) is a typical and useful learning paradigm in big data analysis for capturing correlation across multiple views of the same objects. When dealing with data with additional ordinal information, traditional CCA suffers from poor performance due to ignoring the ordinal relationships within the data. Such data is becoming increasingly common, as either temporal or sequential information is often associated with the data collection process. To incorporate the ordinal information into the objective function of CCA, the so-called ordinal discriminative CCA has been presented in the literature. Although ordinal discriminative CCA can yield better ordinal regression results, its performance deteriorates when data is corrupted with noise and outliers, as it tends to smear the order information contained in class centers. To address this issue, in this article we construct a robust manifold-preserved ordinal discriminative correlation regression (rmODCR). The robustness is achieved by replacing the traditional ( l 2 -norm) class centers with l p -norm centers, where p is efficiently estimated according to the moments of the data distributions, as well as by incorporating the manifold distribution information of the data in the objective optimization. In addition, we further extend the robust manifold-preserved ordinal discriminative correlation regression to deep convolutional architectures. Extensive experimental evaluations have demonstrated the superiority of the proposed methods.

In heterogeneous domain adaptation (HDA), since the feature spaces of the source and target domains are different, knowledge transfer from the source to the target domain is really challenging. How to align the different feature spaces and then adaptively transfer the related knowledge is critical for HDA. In this paper, we develop an adaptive teacher-and-student model for heterogeneous domain adaptation (AtsHDA). In AtsHDA, the source domain as a teacher and the target domain as a student are aligned or co-adapted to each other first, so that their correlation can be maximized. Then the target domain adaptively learns from the source domain. Specifically, there is a balance between the learning by the target domain itself and the instruction from the source domain. That is, when the guidance from the source domain is helpful for learning, the learning of target classifier emphasizes the instruction of source knowledge, and considers its own knowledge more, otherwise. Further, an ensemble method is designed to decide such a balance. Finally, empirical results show that AtsHDA can achieve competitive results compared with the state-of-arts.

Semi-supervised learning is especially important in data mining applications because it can make use of plentiful unlabeled data to train the high-quality learning models. Semi-Supervised Support Vector Machine (S3VM) is a powerful semi-supervised learning model. However, the high computational cost and non-convexity severely impede the S3VM method in large-scale applications. Although several learning algorithms were proposed for S3VM, scaling up S3VM is still an open problem. To address this challenging problem, in this paper, we propose a new incremental learning algorithm to scale up S3VM (IL-S3VM) based on the path following technique in the framework of Difference of Convex (DC) programming. The traditional DC programming based algorithms need multiple outer loops and are not suitable for incremental learning, and traditional path following algorithms are limited to convex problems. Our new IL-S3VM algorithm based on the path-following technique can directly update the solution of S3VM to converge to a local minimum within one outer loop so that the efficient incremental learning can be achieved. More importantly, we provide the finite convergence analysis for our new algorithm. To the best of our knowledge, our new IL-S3VM algorithm is the first efficient path following algorithm for a non-convex problem (i.e., S3VM) with local minimum convergence guarantee. Experimental results on a variety of benchmark datasets not only confirm the finite convergence of IL-S3VM, but also show a huge reduction of computational time compared with existing batch and incremental learning algorithms, while retaining the similar generalization performance.

Feature missing is a serious problem in many applications, which may lead to low quality of training data and further significantly degrade the learning performance. While feature acquisition usually involves special devices or complex process, it is expensive to acquire all feature values for the whole dataset. On the other hand, features may be correlated with each other, and some values may be recovered from the others. It is thus important to decide which features are most informative for recovering the other features as well as improving the learning performance. In this paper, we try to train an effective classification model with least acquisition cost by jointly performing active feature querying and supervised matrix completion. When completing the feature matrix, a novel objective function is proposed to simultaneously minimize the reconstruction error on observed entries and the supervised loss on training data. When querying the feature value, the most uncertain entry is actively selected based on the variance of previous iterations. In addition, a bi-objective optimization method is presented for cost-aware active selection when features bear different acquisition costs. The effectiveness of the proposed approach is well validated by both theoretical analysis and experimental study.

In the paper, we propose a class of dynamic conditional Gaussian graphical models (DCGGMs) based on a set of nonidentical distribution observations, which changes smoothly with time or condition. Specifically, the DCGGMs model the dynamic output network influenced by conditioning input variables, which are encoded by a set of varying parameters. Moreover, we propose a joint smooth graphical Lasso to estimate the DCGGMs, which combines kernel smoother with sparse group Lasso penalty. At the same time, we design an efficient accelerated proximal gradient algorithm to solve this estimator. Theoretically, we establish the asymptotic properties of our model on consistency and sparsistency under the high-dimensional settings. In particular, we highlight a class of consistency theory for dynamic graphical models, in which the sample size can be seen as n4/5for estimating a local graphical model when the bandwidth parameter h of kernel smoother is chosen as h = n-1/5for describing the dynamic. Finally, the extensive numerical experiments on both synthetic and real datasets are provided to support the effectiveness of the proposed method.

Traditional ranking mainly focuses on one type of data source, and effective modeling still relies on a sufficiently large number of labeled or supervised examples. However, in many real-world applications, in particular with the rapid growth of the Web 2.0, ranking over multiple interrelated (heterogeneous) domains becomes a common situation, where in some domains we may have a large amount of training data while in some other domains we can only collect very little. One important question is: "if there is not sufficient supervision in the domain of interest, how could one borrow labeled information from a related but heterogenous domain to build an accurate model?". This paper explores such an approach by bridging two heterogeneous domains via the latent space. We propose a regularized framework to simultaneously minimize two loss functions corresponding to two related but different information sources, by mapping each domain onto a "shared latent space", capturing similar and transferable oncepts. We solve this problem by optimizing the convex upper bound of the non-continuous loss function and derive its generalization bound. Experimental results on three different genres of data sets demonstrate the effectiveness of the proposed approach.

Multimodal recognition is an emerging technique to overcome the non-robustness of the unimodal recognition in real applications. Canonical correlation analysis (CCA) has been employed as a powerful tool for feature fusion in the realization of such multimodal system. However, CCA is the unsupervised feature extraction and it does not utilize the class information of the samples, resulting in the constraint of the recognition performance. In this paper, the class information is incorporated into the framework of CCA for combined feature extraction, and a novel method of combined feature extraction for multimodal recognition, called discriminative canonical correlation analysis (DCCA), is proposed. The experiments show that DCCA outperforms some related methods of both unimodal recognition and multimodal recognition.

Dimensionality reduction is among the keys in mining high-dimensional data. This paper studies semi-supervised dimensionality reduction. In this setting, besides abundant unlabeled examples, domain knowledge in the form of pairwise constraints are available, which specifies whether a pair of instances belong to the same class (must-link constraints) or different classes (cannot-link constraints). We propose the SSDR algorithm, which can preserve the intrinsic structure of the unlabeled data as well as both the must-link and cannot-link constraints defined on the labeled examples in the projected low-dimensional space. The SSDR algorithm is efficient and has a closed form solution. Experiments on a broad range of data sets show that SSDR is superior to many established dimensionality reduction methods.

Choosing an appropriate kernel is one of the key problems in kernel-based methods. Most existing kernel selection methods require that the class labels of the training examples are known. In this paper, we propose an adaptive kernel selection method for kernel principal component analysis, which can effectively learn the kernels when the class labels of the training examples are not available. By iteratively optimizing a novel criterion, the proposed method can achieve nonlinear feature extraction and unsupervised kernel learning simultaneously. Moreover, a non-iterative approximate algorithm is developed. The effectiveness of the proposed algorithms are validated on UCI datasets and the COIL-20 object recognition database.