Traditional Discriminant analysis (DA) is one of the classical supervised learning algorithms to reduce the dimensionality of data with Gaussian assumption. Since the unique class mean in traditional DA is intractable to estimate the non-Gaussian distrbution of data, some existing DA algorithms based on the clustering criterion focus on learning multiple means in each class so as to address the non-Gaussian issue. The clustering-based DA inevitably involved the constraint optimization problem to learn multiple means, which may lead to the locally optimal solution. To address these issues, inspired by the smooth approximation theory and the concept of Kolmogorov mean, this paper explores an unconstraint function with asymptotic property as an alternative proxy to clustering-based DA algorithms. Thus the derived DA algorithm, i.e., adaptive and asymptotic mean-based subclass discriminant analysis (AASDA), which not only leverages multiple means to represent different subclasses in same class but also adaptively and asymptotically learns the similar mean for each sample in the learned optimal subspace via the gradient-based optimizer. The asymptotic analysis of unconstraint function, the gradient analysis and convergence guarantee of proposed criterion verify the effectiveness of AASDA algorithm. Its merits are thoroughly assessed on a suite of synthetic and real world data experiments.

Neuroscientific evidence reveals that human visual recognition is not an instantaneous event but a hierarchical process, where the brain constructs a holistic perception by progressively integrating simple features like edges or texture into complex scenes. Ensemble learning successfully utilizes this principle, yet existing methods typically integrate models at the decision level, neglecting the rich, complementary information within the feature space itself and thus fundamentally limiting their potential. To address this, we introduce Synergistic Semantic Boosting (S2-Boosting), a framework that employs a self-supervised hierarchical semantic learning module to decompose an image into complementary, semantically meaningful parts autonomously. These parts guide a boosting procedure where a sequence of specialized learners, each focusing on a specific semantic partition, collaboratively corrects the ensemble's errors. We further present encouraging results on real-world image datasets, highlighting the intrinsic interpretability, paving the way for more robust and transparent models.

Federated clustering addresses the critical challenge of extracting patterns from decentralized, unlabeled data. However, it is hampered by the flaw that current approaches are forced to accept a compromise between performance and privacy: transmitting embedding representations risks sensitive data leakage, while sharing only abstract cluster prototypes leads to diminished model accuracy. To resolve this dilemma, we propose Structural Privacy-Preserving Federated Graph Clustering (SPP-FGC), a novel algorithm that innovatively leverages local structural graphs as the primary medium for privacy-preserving knowledge sharing, thus moving beyond the limitations of conventional techniques. Our framework operates on a clear client-server logic; on the client-side, each participant constructs a private structural graph that captures intrinsic data relationships, which the server then securely aggregates and aligns to form a comprehensive global graph from which a unified clustering structure is derived. The framework offers two distinct modes to suit different needs. SPP-FGC is designed as an efficient one-shot method that completes its task in a single communication round, ideal for rapid analysis. For more complex, unstructured data like images, SPP-FGC+ employs an iterative process where clients and the server collaboratively refine feature representations to achieve superior downstream performance. Extensive experiments demonstrate that our framework achieves state-of-the-art performance, improving clustering accuracy by up to 10% (NMI) over federated baselines while maintaining provable privacy guarantees.

Self-supervised 3D point cloud understanding is crucial for scene understanding, where Masked Autoencoders (MAE) have achieved excellent performance in point cloud representation learning. However, existing MAE-style methods fail to consider spatial-semantic variations in masking strategies, and joint learning with multi-view images often overlooks view redundancy. To address these challenges, we propose an MAE framework enhanced with reliable multi-view 2D-3D Key-part alignment and Reinforced masking, named as KR-MAE. Our approach comprises three key innovations: Reinforced Masking (RM) strategically samples visible tokens based on semantic saliency to enhance reconstruction fidelity; Reliable Multi-View Selector (RVS) dynamically refines the most informative image subset by filtering occluded or low-texture views, mitigating detrimental redundancy; Reliable-view 2D-3D Key-part Aligned Transformer (KAT) establishes semantic-aligned correspondence between salient 3D point cloud parts and reliable multi-view 2D image patches, leveraging rich texture cues from 2D images to compensate for sparse geometry in point cloud. Extensive experiments on 3D classification and segmentation benchmarks demonstrate that KR-MAE achieves state-of-the-art performance, surpassing prior multi-modal methods.

Time series analysis plays a vital role in fields such as finance, healthcare, industry, and meteorology, underpinning key tasks including classification, forecasting, and anomaly detection. Although deep learning models have achieved remarkable progress in these areas in recent years, constructing an efficient, multi-task compatible, and generalizable unified framework for time series analysis remains a significant challenge. Existing approaches are often tailored to single tasks or specific data types, making it difficult to simultaneously handle multi-task modeling and effectively integrate information across diverse time series types. Moreover, real-world data are often affected by noise, complex frequency components, and multi-scale dynamic patterns, which further complicate robust feature extraction and analysis. To ameliorate these challenges, we propose FusAD, a unified analysis framework designed for diverse time series tasks. FusAD features an adaptive time-frequency fusion mechanism, integrating both Fourier and Wavelet transforms to efficiently capture global-local and multi-scale dynamic features. With an adaptive denoising mechanism, FusAD automatically senses and filters various types of noise, highlighting crucial sequence variations and enabling robust feature extraction in complex environments. In addition, the framework integrates a general information fusion and decoding structure, combined with masked pre-training, to promote efficient learning and transfer of multi-granularity representations. Extensive experiments demonstrate that FusAD consistently outperforms state-of-the-art models on mainstream time series benchmarks for classification, forecasting, and anomaly detection tasks, while maintaining high efficiency and scalability. Code is available at https://github.com/zhangda1018/FusAD.

The anchor similarity matrix, widely used for efficient clustering, exhibits an imbalance between its rows and columns - only the rows are typically constrained by probabilistic properties, unlike the regular similarity matrix where both dimensions are regulated. This paper addresses the critical question of how to impose meaningful constraints on the columns to better capture the data structure. We propose a novel method, termed Multi-view Clustering via Bilaterally constrained anchor Graph (MCBG), which learns a fused anchor similarity matrix with bilateral constraints. To ensure consistency across views, we quantitatively assess their contributions and integrate them into a unified model. By applying distinct constraints to rows and columns, MCBG promotes a balanced and expressive anchor similarity distribution, avoiding degenerate cases. Furthermore, a rank constraint on the Laplacian matrix of an anchor-pairwise graph is incorporated, ensuring a one-step post-processing-free multi-view clustering framework. An efficient alternating iterative optimization algorithm is developed, adapted to the natural properties of the target problem. Extensive experiments validate the superiority of the proposed method.

Ensemble learning methods, such as Bagging and Boosting, are well-regarded for their ability to enhance model performance by combining diverse base learners. These approaches leverage the strengths of individual models to achieve more accurate and robust predictions. However, real-world datasets often contain noise, which can significantly impair model effectiveness. This paper focuses on two prevalent and challenging types: feature noise, which can lead to fitting instability and poor generalization, and label noise, which can lead to erroneous supervision and model overfitting. Recognizing the inherent properties of ensemble learning, particularly its focus on optimizing the decision margin to improve classification accuracy, we see an opportunity to bolster ensemble model robustness. To address both feature and label noise, we propose a novel approach called Dual Geometry Margin Boosting (DGMB). This method employs two key strategies: the Decision Plane Margin (DPM), which enhances class separation, and the Hyper-Sphere Margin (HSM), which effectively filters out potentially noisy samples during the learning process. Our experiments demonstrate the impressive ability of DGMB to resist both feature and label noise. Through rigorous testing on various noise-contaminated datasets, we show that DGMB maintains strong performance and outperforms other robust Ensemble methods.

Recently, Multi-View Graph Clustering (MVGC) methods have achieved significant progress, leading to their wide adoption in various applications. However, most MVGC methods merely pursue consistent information by simply fusing multi-view graphs, ignoring the cross-view interactions among them, which limits the ceiling of their performance. To make up for this deficiency, we design a credible cross-view graph enhancement module to explore the credible topological structure, while accomplishing cross-view interactions, to boost clustering performance in multi-view graph scenarios. Besides, we reconsider the graph clustering task from the perspective of graph signal processing. From this novel perspective, we adapt the high-order Graph Trend Filter to reveal the inhomogeneities in graph smoothness levels and further consider the brand-new local preference in MVGC, which provides theoretical guidance for graph clustering. Building on these insights, we propose the Enhanced Graph Trend Filter Clustering (EGTFC) method and present an effective algorithm accompanied by corresponding theoretical analyses to tackle the optimization problem inherent in EGTFC. Finally, substantial experimental results on twelve benchmark datasets demonstrate the effectiveness of our proposals and the superiority over thirteen state-of-the-art MVGC methods.

Multi-view spectral clustering (MVSC) has garnered growing interest across various real-world applications, owing to its flexibility in managing diverse data space structures. Nevertheless, the fusion of multiple $n\times n$n×n similarity matrices and the separate post-discretization process hinder the utilization of MVSC in large-scale tasks, where $n$n denotes the number of samples. Moreover, noise in different similarity matrices, along with the two-stage mismatch caused by the post-discretization, results in a reduction in clustering effectiveness. To overcome these challenges, we establish a novel fast multi-view discrete clustering (FMVDC) model via spectral embedding fusion, which integrates spectral embedding matrices ($n\times c$n×c, $c\ll n$c≪n) to directly obtain discrete sample categories, where $c$c indicates the number of clusters, bypassing the need for both similarity matrix fusion and post-discretization. To further enhance clustering efficiency, we employ an anchor-based spectral embedding strategy to decrease the computational complexity of spectral analysis from cubic to linear. Since gradient descent methods are incapable of discrete models, we propose a fast optimization strategy based on the coordinate descent method to solve the FMVDC model efficiently. Extensive studies demonstrate that FMVDC significantly improves clustering performance compared to existing state-of-the-art methods, particularly in large-scale clustering tasks.

Top-$k$k feature selection in sparse learning is a fundamental problem in machine learning. It is difficult to conquer due to the rigid $\ell _{2,0}$ℓ2,0-norm constraint. Existing literature mostly relaxes the constraint and seeks the approximation of the selection matrix, degenerating primitive models and missing the genuine solutions. This research tackles the primitive top-$k$k feature selection model in sparse learning. From the perspective of universality, we investigate both supervised and semi-supervised models of top-$k$k feature selection in sparse learning. By disassembling the feature selection matrix, it is revealed that two different objectives could be unified into one general ratio-trace problem, which is a non-convex optimization problem. The accelerated coordinate descent method is raised to efficiently solve the non-convex objective, through which the local optimal solution of top-$k$k feature indices is obtained with a competitive time cost. To verify the proposed algorithm, we design toy experiments that could visualize the advantages of the selected features. Meanwhile, experimental results on nine normal datasets and the large-scale ImageNet dataset comprehensively show the superiority of our methods compared to representative and state-of-the-art supervised and semi-supervised algorithms.

Bipartite graph-based co-clustering is efficient in modeling cluster manifold structures. However, existing methods decouple bipartite graph construction from the learning of pseudo-labels for samples and anchors, often leading to suboptimal clustering performance. Moreover, neglecting local manifold relationships among anchors yields inferior anchor pseudo-labels, which further degrades the quality of sample pseudo-labels. To overcome these limitations, we propose a novel model termed Fast Co-Clustering (FC$^{2}$2), which jointly captures both local and global correlations between samples and anchors. Specifically, to model the coupling between the one-hot pseudo-labels of samples and anchors, we construct a bipartite graph with adaptively updated weights during the clustering process. To prevent severely imbalanced cluster assignments, we prove the equivalence between maximizing pseudo-label covariance and balancing cluster proportions, and incorporate a balanced regularization term to enhance the rationality of the resulting clusters. Furthermore, the local smoothness of anchor pseudo-labels is preserved via a low-rank decomposition of a compact anchor similarity graph. These two components jointly ensure that spatially adjacent anchors tend to share similar cluster identities, and that samples and anchors in close proximity are also assigned to similar clusters. We develop an efficient iterative optimization algorithm to update all model variables. Extensive experiments on benchmark and synthetic datasets validate the superior performance and efficiency of the proposed method compared with state-of-the-art approaches.

Graph-based multi-view clustering aims to leverage the consistency and complementarity of multiple information sources (views) to enhance clustering performance. The introduction of multi-order graphs has brought significant performance gains by mitigating the sparsity of first-order graphs. However, different views, along with their derived high-order graphs, inevitably contain noise and view-specific information (i.e., diversity), which may hinder the learning of a consensus graph. To address this critical issue while retaining the benefits of multi-order structures, this paper proposes a novel framework, termed Consistency driven Decomposition for Multi-view Multi-order Graph Clustering (CDMMGC). In CDMMGC, multi-order graphs are utilized to mitigate the sparsity problem of first-order graphs and each multi-order graph from each view is decomposed into a consistency and a diversity component. Accordingly, the consensus graph is learned on the consistency component from multi-view multi-order graphs, which is expected to be more accurate in capturing the data semantics. Experiments on extensive datasets demonstrate the effectiveness and superiority of the proposed CDMMGC model in data clustering compared with the state-of-the-art methods. The code is available athttps://github.com/SunseaIU/CDMMGC

Multi-view Graph Clustering (MGC) is a crucial approach for uncovering complex data structures by leveraging multiple perspectives of data. However, existing MGC methods face two key challenges: (1) limitations in graph structure that neglect long-range dependencies, and (2) overlooking the view-cluster local structure when mining view discrepancies. To address these issues, we propose a Multi-view Graph Clustering approach based on Dual View-Cluster-Order Interactivity (DVCOI-MGC). This approach consists of three modules: (1) Multi-View Multi-Order Graph Construction, where high-order graphs are generated using matrix exponentiation to capture long-range dependencies; (2) Dual View-Cluster-Order Interactivity, which utilizes a discrete graph cut model to separately learn order-specific and view-specific clustering results from the sets of order-specific multi-view graphs and view-specific multi-order graphs, with a separate View-Cluster-Order tensor weight for each learning direction; and (3) Bidirectional Truncation Consistency Learning, which applies a sparse boolean weight vector to locally select and integrate clustering results while preserving both the view-cluster and order-cluster local structures. Additionally, we introduce an efficient iterative optimization method to solve the discrete graph cut problem and provide a theoretical analysis of its convergence and computational complexity. Extensive experiments on 8 real-world datasets demonstrate that our approach significantly improves clustering performance over 11 state-of-the-art methods.

Feature selection is one of the hot issues in machine learning. It reduces storage pressure by effectively screening features and has become a very practical data preprocessing method. At present, most feature selection algorithms apply $\ell _{2,1}$ -norm on the transformation matrix to calculate the scores for all features and then select appropriate features according to these scores. But their sparsity is limited, and meaningless regularization parameters increase the cost, making it prone to falling into local optimum. To solve the above difficulties, this article proposes a novel max-min robust unsupervised feature selection via sparse subspace (MMRUFS), which considers both the reconstruction term and variance term of data, so that the model can not only fully retain the original information of data, but also make the data more dispersed. Second, $\ell _{2,0}$ -norm constraint is used on the transformation matrix to directly select the optimal feature subset, avoiding the fine-tuning of regularization parameters. To enhance the robustness, MMRUFS carefully designs mark weight vector to make the model treat normal samples and outliers differently and achieves the effect of anomaly detection. Finally, MMRUFS is solved by designing the surrogate matrix, and its convergence is strictly guaranteed, experimental results reveal that MMRUFS outperforms other feature selection algorithms on multiple real-world datasets.

Linear discriminant analysis (LDA) is a widely used dimensionality reduction (DR) technique that is effective in extracting discriminative features across various fields. Ratio sum LDA (RSLDA), a variant of LDA, was developed to address the shortcoming of the LDA method, which tends to obtain features with weak discriminative information. However, the traditional ratio sum formulation is dominated by the maximum ratio, which makes it difficult to select highly discriminative features and contradicts the original goal of the ratio sum. In this article, we analyzed the underlying causes of the dominance problem in RSLDA. A novel discriminant feature learning method via balance ratio sum discriminant analysis (BRSDA) is proposed. BRSDA effectively balances the ratios of the model formulation, thereby mitigating the domination problem. It focuses on optimizing low-quality projection directions, thereby yielding a well-balanced solution with consistently strong projection quality. First, a minimization ratio sum (Min-RS) criterion is adopted, which leverages the balance property of the harmonic mean to balance the gaps between the ratios. Second, Min-RS is integrated with the $\ell _{p}$ -norm to further balance gaps between ratios. By amplifying the differences across projection directions, the $\ell _{p}$ -norm drives BRSDA to emphasize the optimization of low-quality directions, thus raising the lower bound of direction quality. Finally, since obtaining a closed-form solution for the BRSDA problem is challenging, the gradient descent method is employed to solve its optimization. Sufficient experimental results verify the effectiveness of BRSDA, and BRSDA can effectively solve the domination problem and extract discriminative features.

Graph-based multi-view clustering has become a powerful tool for multi-view data analysis by jointly capturing structural relationships across heterogeneous views. Despite significant advances in this field, several limitations persist. Most graph-based multi-view algorithms employ a relaxation-and-discretization two-stage paradigm to obtain clustering assignments, which may significantly undermine information preservation and clustering performance. Moreover, these methods often suffer from quadratic or even cubic time complexity, rendering them impractical for large-scale applications. Additionally, existing graph-based multi-view models struggle to adequately capture the inherent uncertainty characterized by ambiguous or overlapping clustering boundaries. To address these challenges, this paper proposes a multi-view fuzzy clustering method with anchor graph. Specifically, the proposed method learns a symmetric and doubly stochastic consensus similarity graph from multiple views and performs clustering directly on the learned consensus graph, thereby directly outputting clustering results. By imposing nonnegativity and row normalization constraints on the cluster indicator matrix, the proposed method can further reveal the underlying uncertain clustering structure. Furthermore, an iterative optimization algorithm with linear complexity is developed to solve the proposed model. Extensive experiments conducted on both synthetic and real-world benchmark datasets illustrate the effectiveness and efficiency of the proposed method against state-of-the-art models.

We propose a novel Greedy Graph Cut (GGC) algorithm to address the graph partitioning problem. The algorithm begins by treating each data point as an individual cluster and iteratively merges cluster pairs that maximize the reduction in the global objective function until the desired number of clusters is achieved. We provide a theoretical proof of the monotonic convergence of the objective function values throughout this process. To improve computational efficiency, the algorithm restricts merging operations to adjacent clusters, resulting in a computational complexity that scales nearly linearly with the sample size. A significant advantage of our greedy approach is its deterministic nature, which ensures consistent results across multiple runs. This stands in contrast to many existing algorithms that are sensitive to random initialization effects. We demonstrate the effectiveness of the proposed algorithm by applying it to the Normalized Cut (N-Cut) problem, a well-studied variant of graph partitioning. Extensive experimental results show that GGC consistently outperforms the conventional two-stage optimization approach-which involves eigendecomposition followed by k-means clustering-in solving the N-Cut problem. Furthermore, comparative analyses reveal that GGC achieves superior performance compared to several state-of-the-art clustering algorithms.

Multi-view clustering (MVC) based on anchor learning has been proven to be effective in improving clustering accuracy and efficiency. Existing MVC methods are mainly based on single-granularity anchor learning, that is, the number of anchors corresponding to different views is constant and consistent, which will lead to information redundancy or insufficient mining. In addition, aggregating anchors of varying scales from all views to obtain multi-view shared clustering results remains a problem to be further explored. To address the above problems, a novel MVC method named View-adaptive Multi-granularity Anchor Learning (VMAL) is proposed in this paper, where view-adaptive anchor pruning and view-shared sample clustering are jointly optimized. On the one hand, VMAL can dynamically adjust the optimal number of anchors for each view during optimization by exploiting the reconstruction error of samples. On the other hand, an intuitive and effective mapping-aggregation message passing strategy is cleverly designed, which first maps the anchor representations of different views to the cluster space and then transfers the obtained cluster information of anchors to the sample space through an aggregation matrix. As a byproduct, VMAL can directly obtain the discrete cluster distribution of samples without additional partitioning. Finally, an iterative optimization algorithm is developed to solve the proposed VMAL method. Experimental results on multiple datasets have demonstrated the superiority of VMAL in terms of clustering results when compared with other state-of-the-art methods.

Tensor-based multi-view clustering algorithms have attracted considerable attention due to their superior clustering performance. However, these algorithms typically treat each view independently, failing to utilize the complementary information across all views, thus lacking globality. Additionally, employing low-rank tensor constraints to extract consistent information among views may result in the loss of important information due to weak consistency constraints. These limitations significantly hinder the clustering performance. To address these issues, we propose Simple Multi-view Tensor Clustering (SimMTC), which achieves globality and strong consistency. SimMTC first applies Fast Fourier Transform (FFT) to the anchor graphs to obtain high-frequency and low-frequency information, which encode similarities between samples and anchors from all views, thereby capturing global information. Orthogonal tensor factorization is then conducted in the frequency domain. Moreover, a novel strong consistency constraint based on FFT is introduced, which enhances the extraction of consistent information in the frequency domain. What's more, an efficient alternating optimization algorithm is designed to solve the optimization problem in SimMTC. Finally, extensive experiments on real-world datasets demonstrate that SimMTC achieves state-of-the-art clustering performance. The code has been made publicly available on GitHub at: https://github.com/haonanxin/SimMTC_code.

Spectral clustering has received widespread attention for its effectiveness in handling nonconvex geometries. The classic methods of spectral clustering include Ratio Cut (Rcut), Normalized Cut (Ncut), and Min-Max Cut (MMcut). Among them, the objective function of MMcut is more reasonable. Unfortunately, existing methods cannot solve MMcut problem without relaxing the discrete or nonnegative constraints. To this end, based on coordinate descent, we propose a basic optimization algorithm to solve MMcut problem without relaxing any constraint. And then, a fast version of the solver is proposed to improve the computational efficiency. Besides, we prove the convergence of the proposed solver, evaluate its computational complexity, and discuss the connection between MMcut and NCut. Finally, extensive experiments are performed to evaluate the effectiveness of our proposed method.

Multi-view clustering (MVC) with bipartite graph has been extensively studied to rapidly handle multi-source heterogeneous information via sparse anchors. However, most existing methods follow a two-stage learning paradigm that first learns continuous label matrix and then discretizes it, not only bringing extra trade-off parameters but yielding suboptimal solutions. Also, numerous methods still exhibit limited scalability for large-scale problems. Thus, this paper proposes two novel models for discrete, trade-off parameter-free and rapid MVC. First, the Bipartite Graph Discrete Reconstruction (BGDR) model uniquely leverages the discrete label matrices of both samples and anchors to dynamically reconstruct a consensus bipartite graph across views. This concise reconstruction style eliminates redundant computations, and anchor labels enable to enrich cluster partition information during reconstruction, enhancing both accuracy and efficiency. The final clustering outcomes are directly acquired via discrete sample labels. Second, to free the optimization time overheads from the limitation of sample size, we further devise the Compact Graph Discrete Reconstruction (CGDR) model, which reconstructs a smaller compact affinity graph among anchors for significant acceleration. Original sample labels are then gained by label propagation. Systematic experiments illuminate that both models reach superior outcomes in term of efficacy and efficiency.

We propose the DPSM method, a density-based node clustering approach that automatically determines the number of clusters and can be applied in both data space and graph space. Unlike traditional density-based clustering methods, which necessitate calculating the distance between any two nodes, our proposed technique determines density through a propagation process, thereby making it suitable for a graph space. In DPSM, nodes are partitioned into small clusters based on propagated density. The partitioning technique has been proved to be sound and complete. We then extend the concept of spectral clustering from individual nodes to these small clusters, while introducing the CluCut measure to guide cluster merging. This measure is modified in various ways to account for cluster properties, thus provides guidance on when to terminate the merging process. Various experiments have validated the effectiveness of DPSM and the accuracy of these conclusions.

K-means algorithm divides samples into c classes based on their structural characteristics. However, due to the non convex nature of the clustering problem, algorithms are prone to converge to poor local minima. To address the aforementioned issues, we propose the Local Structure Preserving Clustering with Learnable Anchors (LSPC-LA) method. We assume that with a well-designed anchor selection strategy, samples near the same anchor tend to belong to the same cluster, which reveal high confidence Must-Link local structural information for clustering. Based on this observation, we first construct an anchor-based bipartite graph, transforming the sample clustering problem into anchor clustering problem by local structural information, thus reducing the solution space and minimizing the risk of poor local minima. Then we create an anchor guiding matrix to allow anchors to learn the sample structure, improving clustering performance. Subsequently, an alternating iterative algorithm is proposed to optimize the LSPC-LA model. Finally, extensive experiments demonstrate the accuracy of the local structural information and the effectiveness of LSPC-LA.

Multiview clustering aims to detect the consistent graph structure across each view, and has garnered extensive attention in recent years. However, the graph quality determines the clustering performance and cannot be updated in most methods, while the label extraction also relies on post-processing (e.g. spectral rotation). As a consequence, multiple objective functions are optimized independently, making it difficult to achieve one-pass clustering. Motivated by this, Graph-Free Multiview Clustering with Anchor (GMCA) is proposed towards coherence (in optimization), simplicity (in hyperparameters). Considering that similarity only exist among samples in the same cluster for ideal assignments, a label-based reverse framework is proposed for the first time to achieve feature approximation via graph-free factorization. Although non-negative label matrix exhibits indicative interpretability, cluster independence is overlooked, thus non-negative and orthogonal constraint is further imposed and proved beneficial for ideal graph backtracking. Besides, to prevent optimization from focusing on extreme approximating loss caused by redundant features, the Frobenius norm is employed, while allocating flexible view weights as collateral benefit. Comparison experiments with fourteen state-of-the-art methods are performed on eight real-world data sets, while our method exceeds the second-best method by up to$4.85\%$in clustering accuracy and keeps running time linear with samples number.

Deep models, characterized by complex structures and end-to-end optimization, proved effective in providing decision support based on real-world data. However, the lack of transparency in their decision-making process and the difficulty in interpreting the role of individual neurons limited their practical applicability in many critical and sensitive domains. Inspired by the parallels between neural networks and ensemble models, where performance was achieved through the collaboration of multiple weak learners, this article presents a novel perspective that reframes neural networks as hierarchical ensembles. We propose the hierarchical backpropagated ensemble (HBE) model, wherein each neuron functions both as a base learner and as part of an ensemble of preceding neurons. This framework applies ensemble learning techniques to neural networks, allowing each neuron to focus on specific subtasks while progressively constructing a network that meets global objectives. Experimental results on real-world data show that this hierarchical structure enhances the effectiveness of traditional ensemble models, and the ensemble-based explanations offer improved initialization and dynamically adjustable network structures, leading to more efficient training.

The representation learning of time series has a wide range of downstream tasks and applications in many practical scenarios. However, due to the complexity, spatiotemporality, and continuity of sequential stream data, compared with the representation learning of structural data such as images/videos, the time series self-supervised representation learning is even more challenging. Besides, the direct application of existing contrastive learning and masked autoencoder based approaches to time series representation learning encounters inherent theoretical limitations, such as ineffective augmentation and masking strategies. To this end, we propose a Language Pre-training guided Masking Representation Learning (LPMRL) for times series classification. Specifically, we first propose a novel language pre-training guided masking encoder for adaptively sampling semantic spatiotemporal patches via natural language descriptions and improving the discriminability of latent representations. Furthermore, we present the dual-information contrastive learning mechanism to explore both local and global information by meticulously designing high-quality hard negative samples of time series data samples. As a result, we also design various experiments, such as visualization of masking position and distribution and reconstruction error to verify the reasonability of proposed language guided masking technique. Last, we evaluate the performance of proposed representation learning via classification task conducted on 106 time series datasets, which demonstrates the effectiveness of proposed method.

Masked autoencoders have been extensively utilized in 3D point cloud self-supervised learning, where the fundamental approach involves masking a portion of the point cloud and subsequently reconstructing it. This process is hypothesized to enhance model learning by leveraging the inherent structure of the point cloud data. However, the information density within point clouds is inherently uneven, contrasting with the more uniform distributions found in language and 2D image data. This uneven distribution suggests that the application of random masking strategies, commonly adopted from NLP and 2D vision, may not be optimal for point cloud data, potentially leading to suboptimal learning outcomes. Based on this observation, we propose a simple yet effective Density-directed Masked Autoencoders for Point Cloud Self-supervised Learning (Point-DMAE), which learns latent semantic point cloud features using a density-directed masking strategy. Specifically, our method employs a dual-branch Transformer architecture to extract both high-level and fine-grained point features through global and local block density-directed masking, respectively. Point-DMAE demonstrates high pre-training efficiency and significantly outperforms our baseline (Point-MAE) on 3D object classification tasks within the ScanObjectNN dataset by 4.13% on OBJ-BG, 5.17% on OBJ-ONLY, and 4.17% on PB-T50-RS. Codes are available at https://github.com/jinxianglong10/Point-DMAE.

Anchor graph clustering has gained significant attention due to its effectiveness and efficiency. As the most representative points in original data, anchors are applied to connect the sample space to label space. However, when noise is present in original data, the anchor-refined label spreading mechanism may fail. To address this, we propose an Efficient Co-clustering via Anchor-refined Label Spreading (ECALS), which simultaneously clusters original data and anchors. We introduce the size constraint that ensures each cluster contains a minimum number of samples. Our method includes two variants, which are continuous and discrete model, catering to both fuzzy and discrete label matrices. Both models are applicable to out-of-sample problems and demonstrate superior performance on synthetic and real-world datasets.

Anchor-based clustering methods have gained attention for their efficiency in subspace, multi-view, and ensemble clustering tasks. Most existing methods focus on using anchors to reduce computational complexity in the original data space. However, clustering directly on anchors, followed by label propagation to the original data, can significantly improve computational efficiency. In this paper, we propose an Efficient Anchor Graph Clustering (EAGC) method that maximizes within-cluster homogeneity among anchors. Inspired by the relaxation and discretization model in spectral clustering, we propose two corresponding models, namely EAGC-R and EAGC-D. EAGC-R first obtains relaxed spectral embedding of anchors and then the embedding is discretized by K-Means. EAGC-D directly solves the discrete anchor membership matrix by coordinate descent method. Once anchor clustering results are obtained, original data labels can be obtained through anchor label transmission. Extensive experiments conducted on both synthetic and real datasets illustrate the effectiveness of proposed methods.

Ensemble learning has achieved remarkable success in machine learning, but its reliance on numerous base learners limits its application in resource-constrained environments. This paper introduces an innovative "Margin-Maximizing Fine-Grained Ensemble Method" that achieves performance surpassing large-scale ensembles by meticulously optimizing a small number of learners and enhancing generalization capability. We propose a novel learnable confidence matrix, quantifying each classifier’s confidence for each category, precisely capturing category-specific advantages of individual learners. Furthermore, we design a margin-based loss function, constructing a smooth and partially convex objective using the logsumexp technique. This approach improves optimization, eases convergence, and enables adaptive confidence allocation. Finally, we prove that the loss function is Lipschitz continuous, based on which we develop an efficient gradient optimization algorithm that simultaneously maximizes margins and dynamically adjusts learner weights. Extensive experiments demonstrate that our method outperforms traditional random forests using only one-tenth of the base learners and other state-of-the-art ensemble methods.

Generalized Category Discovery (GCD) is a recently proposed open-world problem that aims to automatically classify and discover new categories based on partially labeled data. For unlabeled data, previous research commonly considers using pseudo-labels to assist in model learning. These pseudo-labels, together with the true labels of labeled data, form the learning targets for the final classifier, leading to better predictive outcomes. However, low-quality labels can inevitably hinder the learning process of the model. To address this issue, inspired by previous methods, we propose the Calibrated Generalized Category Discovery (CGCD) framework, which incorporates a projection head, a classifier head, and a calibration head. The projection head is used for representation learning. The calibration head learns high-quality labels from the robust predictions of the classifier head, and the classifier head utilizes these high-quality labels for more efficient learning. Both heads mutually enhance each other during training, ultimately leading to a superior solution. In addition, leveraging the characteristics of both the classifier head and calibration head, we designed a classifier representation distribution regularization term to further ensure consistency in their learning processes. Extensive experimental results demonstrate that the proposed CGCD framework achieves state-of-the-art performance across five general and fine-grained visual recognition datasets by leveraging high-quality label learning.

The use of anchors often leads to better efficiency and scalability, making them highly favored. However, there is a challenge in anchor-based multi-view subspace learning. A unified anchor graph overly emphasize the commonality between views, failing to adequately capture the view-specific individuality. This has led some models to independently explore the individuality of each view before aligning and integrating them, often achieving better performance but making the process more cumbersome. Therefore, this paper proposes a new model, simultaneously capturing the individuality and commonality between anchor graphs for multi-view clustering. The model has three notable advantages: First, it allows view-specific anchor graphs to align in real-time with a common anchor graph as a reference, eliminating the need for post-alignment. Second, it enforces a cluster-wise structure among anchors and balances sample distribution among them, providing strong discriminative power. Lastly, it maintains linear complexity with respect to the numbers of samples and anchors, avoiding the significant time costs associated with their increase. Comprehensive experiments demonstrate the effectiveness and efficiency of our method compared to various state-of-the-art algorithms.

ABSTRACT Band selection in hyperspectral imaging is a burgeoning research area whose aim is to select a small number of bands in order to reduce data redundancy and noise bands. The existing ranking‐based methods face two challenges: (1) The density calculation using nearest neighbours only considers distances between bands, ignoring shared neighbours. Thus, it fails to reflect the local distribution of bands. (2) The high dimensionality of the bands limits the effectiveness of the Euclidean distance‐based metric in accurately capturing their similarity. To address the issues, we've proposed an innovative approach for selecting bands, grounded in a mass‐based metric and shared nearest neighbours called MBSNN. Initially, we leverage a mass‐based metric computation technique to supplant the conventional distance metric between disparate bands. This substitution mitigates the distortions that high‐dimensional data can inflict on distance calculations. Subsequently, the natural nearest neighbour method is combined to calculate the local density of the band, reflecting its local distribution characteristics. Finally, an information entropy and peak synergy band selection technique is constructed. To substantiate the merits of our proposed approach, we executed experiments utilising support vector machines across four benchmark datasets. The results of these experiments affirm the effectiveness of our band selection approach.

Feature selection plays an important role in a wide range of applications. Most sparsity-based feature selection methods solve a relaxed$\ell _{2,p}$-norm ($0 \lt p \leq 1$) regularized problem, which often results in a sub-optimal feature subset and requires extensive effort to tune regularization parameters. Optimizing the non-convex$\ell _{2,0}$-norm constrained problem remains an open challenge. Existing optimization algorithms for solving the$\ell _{2,0}$-norm constrained problem often rely on specific data distribution assumptions and cannot guarantee global convergence. In this article, we propose an unsupervised discriminative feature selection method using$\ell _{2,0}$-norm constrained sparse projection (SPDFS) to address these challenges. Specifically, building on the principle of supervised linear discriminant analysis, fuzzy membership learning and$\ell _{2,0}$-norm constrained projection learning are jointly performed to learn a feature-wise sparse projection for unsupervised discriminative feature selection. More importantly, we follow two optimization strategies to address the NP-hard nature of the problem: a non-iterative algorithm with a globally optimal solution is derived for a special case, and an iterative algorithm with both ascent property and approximation guarantee is employed for the general case. Additionally, we explore the relationship between our model and its potential variants. Experimental results on both synthetic and real-world datasets demonstrate the superiority of the proposed method over several state-of-the-art methods in data clustering and text classification tasks. The code is available at:https://github.com/xiadongcs/SPDFS.

In the graph-based semi-supervised learning, the Green-function method is a classical method that works by computing the Green's function in the graph space. However, when applied to large graphs, especially those sparse ones, this method performs unstably and unsatisfactorily. We make a detailed analysis on it and propose a novel method from the perspective of optimization. On fully connected graphs, the method is equivalent to the Green-function method and can be seen as another interpretation with physical meanings, while on non-fully connected graphs, it helps to explain why the Green-function method causes a mess on large sparse graphs. To solve this dilemma, we propose a workable approach to improve our proposed method. Unlike the original method, our improved method can also apply two accelerating techniques, Gaussian Elimination, and Anchored Graphs to become more efficient on large graphs. Finally, the extensive experiments prove our conclusions and the efficiency, accuracy, and stability of our improved Green's function method.

Fuzzy C-Means algorithm (FCM) is one of the most commonly used fuzzy clustering algorithm, which uses the alternating optimization algorithm to update the membership matrix and the cluster center matrix. FCM achieves effective results in clustering tasks. However, due to many constraints, the objective function is inconvenient to optimize directly and is prone to converges to a suboptimal local minimum, which affects the clustering performance. In this paper, we propose a minimization problem equivalent to FCM. Firstly, we use the optimal solution when fixing the cluster center matrix to replace the membership matrix, transforming the original constrained optimization problem into an unconstrained optimization problem, thus reducing the number of variables. We then use gradient descent instead of alternating optimization to solve the model, so we call this model UC-FCM. Extensive experimental results show that UC-FCM can obtain better local minimum and achieve superior clustering performance compared to FCM under the same initialization. Moreover, UC-FCM is also competitive compared with other advanced clustering algorithms.

In this paper, we explain the mechanism of bilinear pooling as a module of hard sample generation, and find that bilinear pooling significantly expands variances of the first-order vectors when it produces discriminative bilinear features. In conjunction with the extremely high dimensionality of the obtained bilinear features, those variances lead to overfitting in subsequent learning models. To solve this issue, we construct a bi-level optimization problem, where the high-level problem is the supervised classification loss, and the low-level problem is the principal component analysis (PCA). Then, we find that PCA on bilinear features is equivalent to spectral clustering, which allows us to mathematically prove that the first $\log _{2}(C)$log2(C) principal components can support the discriminant information of $C$C classes. By removing the rest principal components, the dimensionality and variances are simultaneously reduced. To the best of our knowledge, this is the first work providing a lower bound for dimension reduction for bilinear pooling. However, the PCA projection matrix $\mathbf{L}$L is prone to overfitting due to having many parameters. To address this issue, we propose a rank-$k$k general bilinear projection (RK-GBP) that decomposes $\mathbf{L}$L into two small matrices $\mathbf{U}$U and $\mathbf{V}$V, whose learnable parameters are smaller. Different from traditional bilinear projections used in factorized bilinear pooling (FBiP), our RK-GBP can preserve the orthogonality of columns in $\mathbf{L}$L by constraining the orthogonality of columns in $\mathbf{U}$U and $\mathbf{V}$V. For computational efficiency, we relax the PCA in the low-level task into a dictionary learning problem, obtaining the rank-$k$k orthogonal factorization bilinear pooling (RK-OFBP). The RK-OFBP can be considered as a general form of current factorization bilinear pooling methods (e.g., Hadamard product-based ones). Finally, we evaluate our approach on fine-grained images and large-scale datasets, demonstrating that our proposed method not only produces extremely low-dimensional features but also outperforms other methods in classification tasks. For example, our RK-OFBP can employ 32-dimensional vectors to achieve comparable results to B-CNN (Lin, 2015) (dimension: 512*512) for the 200-class classification task.

Clustering has long been a fundamental problem in machine learning and data mining, with the aim of grouping data samples on the basis of their intrinsic similarity. However, the consensus that different features often exhibit varying levels of discriminative power in clustering model learning is under explored sufficiently in collaboration with the pseudo-label guided unsupervised discriminative analysis. To this end, we propose an Adaptive Feature-Weighted Local-global data Clustering (AFW-LGC) model which is featured by two improvements. First, AFW-LGC takes into account both global separability (between-cluster scatter) and local compactness (within-cluster scatter) whose impacts are mediated by a learnable parameter. Second, the different contributions of features are adaptively learned in AFW-LGC for further discriminative ability enhancement. Both improvements are seamlessly integrated for feature-weighted unsupervised discriminative subspace clustering nature of AFW-LGC. Extensive experiments on eight data sets demonstrate the superior clustering performance of AFW-LGC over some SOTA methods as well as the rationality of our proposed feature importance exploration strategy.

Graph-based multi-view clustering has garnered considerable attention owing to its effectiveness. Nevertheless, despite the promising performance achieved by previous studies, several limitations remain to be addressed. Most graph-based models employ a two-stage strategy involving relaxation and discretization to derive clustering results, which may lead to deviation from the original problem. Moreover, graph-based methods do not adequately address the challenges of overlapping clusters or ambiguous cluster membership. Additionally, assigning appropriate weights based on the importance of each view is crucial. To address these problems, we propose a self-weighted multi-view fuzzy clustering algorithm that incorporates multiple graph learning. Specifically, we automatically allocate weights corresponding to each view to construct a fused similarity graph matrix. Subsequently, we approximate it as the scaled product of fuzzy membership matrices to directly derive clustering assignments. An iterative optimization algorithm is designed for solving the proposed model. Experiment evaluations conducted on benchmark datasets illustrate that the proposed method outperforms several leading multi-view clustering approaches.

The application of hyperspectral image (HSI) clustering has become widely used in the field of remote sensing. Traditional fuzzy K-means clustering methods often struggle with HSI data due to the significant levels of noise, consequently resulting in segmentation inaccuracies. To address this limitation, this letter introduces an innovative outlier indicator-based projection fuzzy K-means clustering (OIPFK) algorithm for clustering of HSI data, enhancing the efficacy and robustness of previous fuzzy K-means methodologies through a two-pronged strategy. Initially, an outlier indicator vector is constructed to identify noise and outliers by computing the distances between each data point in a reduced dimensional space. Subsequently, the OIPFK algorithm incorporates the fuzzy membership relationships between samples and clustering centers within this lower-dimensional framework, along with the integration of the outlier indicator vectors, to significantly mitigates the influence of noise and extraneous features. Moreover, an efficient iterative optimization algorithm is employed to address the optimization challenges inherent to OIPKM. Experimental results from three real-world hyperspectral image datasets demonstrate the effectiveness and superiority of our proposed method.

Fuzzy clustering and its derivatives have been widely applied for handling overlapping clusters through probabilistic membership assignment, yet their performance degrades under cumulative outlier interference. To cope with this limitation, we propose the Outlier Resistant Fuzzy Clustering via Row Sparse Discriminative Embedding Projection (RFCDE), which introduces an adaptive sample contribution vector to resist the outliers, a row-sparse membership refinement strategy to enhance normal sample attention, and a projection-guided prototype learning module to mitigate representation bias. Furthermore, a discriminative embedding objective is designed to effectively mitigate extraneous feature effects. These modules form a unified iterative architecture that improves clustering reliability in a low-dimensional framework. Comparative experiments on real-world datasets validate its broad applicability.

Deep nonnegative matrix factorization (DNMF) technique has attached much great attention in recent year, since it can effectively discover the underlying hierarchical structure of complex data. However, most existing unsupervised and semi-supervised DNMF approaches not only suffer from the noisy data seriously, but also fail to enhance the decomposition quality of DNMF obviously by using the obtained supervisory information. To overcome these drawbacks, a robust semi-supervised DNMF method, called the correntropy based semi-supervised DNMF with constraint propagation (CSDCP), is proposed in this paper for learning a compact and meaningful data representation from the original data. Particularly, instead of adopting the traditional Frobenius norm, CSDCP employs the nonlinear and local similarity measure (e.g., correntropy) as the loss function in DNMF to enhance the robustness of DNMF for the noisy data. In addition, the hypergraph based constraint propagation (HCP) algorithm is adopted in CSDCP to exploit the limited supervisory information fully for capturing good data representation. Moreover, algorithm analysis of CSDCP is presented in this paper, including convergence analysis, robustness analysis supervised information analysis, and computational complexity. Extensive experimental results have illustrated that, in comparison to the most related DNMF approaches, CSDCP usually has better clustering results on six nonnegative datasets in clustering tasks.

Ensemble clustering based on co-association matrices integrates multiple connective matrices from base clusterings to achieve superior results. However, these methods primarily focus on inter-sample relationships, neglecting variations across different base clusterings, potentially introducing noise. Additionally, they overlook interactions between samples and base clusterings, which are crucial for extracting common information and avoiding post-processing steps that may cause information loss and instability in clustering results. To address these issues, we propose the Tensorized Graph Learning for Spectral Ensemble Clustering (TGLSEC) model. TGLSEC stacks all connective matrices into a third-order tensor, employs Fast Fourier Transform (FFT) for encoding, and elucidates inter-relations in the frequency domain. By minimizing the tensor Schatten p-norm, TGLSEC extracts common information in the low-rank space, eliminating noise and improving the quality of the common shared graph. Incorporating Laplacian rank constraints, TGLSEC learns a common shared graph with c-connected components, directly representing the clustering structure and avoiding post-processing steps, leading to more stable clustering results. To enhance computational efficiency for large-scale datasets, TGLSEC has been expanded into a bipartite-graph-based model, TGLSEC-BG, reducing complexity and computational time. Extensive experiments on real-world datasets demonstrate that TGLSEC and TGLSEC-BG exhibit superior clustering performance and robustness to noise.

Deep Multi-View Clustering (MVC) methods partition multi-view data into disjoint clusters in an unsupervised manner, showing significant promise across various domains. However, current MVC methods primarily focus on capturing the consistency information shared across all views and undervalue the specificity information inherent in each view that reflects its unique characteristics. Furthermore, the underexploration of the separability of learned representations limits the overall clustering performance of existing MVC methods and leads to undesirable clustering results. In this paper, we propose a fully differentiable and end-to-end deep MVC framework, named Comprehensive Information Extraction with Separable Representation Learning (CIRSEL), to address these issues. CIRSEL recasts specificity information extraction as a high-order graph pooling process to capture the view-specific characteristics of individual views. Utilizing the cross-attention mechanism, CIRSEL adaptively fuses the consistent and view-specific representations to achieve comprehensive information extraction. Subsequently, CIRSEL maps representations into a unit hypersphere space with evenly distributed prototypes and maximizes the variational estimation of Mutual Information, which enhances the inter-cluster separability and intra-cluster compactness in the embedding space and further benefits the following clustering learning. Finally, CIRSEL introduces a nuclear norm-based balance regularization, which ensures balanced clustering results can be directly retrieved by the cosine similarity between the representations and prototypes. Extensive experiments on ten benchmark datasets demonstrate the effectiveness of CIRSEL compared to sixteen current MVC methods.

Ensemble clustering, which combines the information from multiple base clusterings to obtain a better partition result, has received extensive attention due to its effectiveness and robustness. Although many algorithms have been developed in recent years that have achieved impressive results in practical applications, two challenging issues in ensemble clustering remain. First, most algorithms assume that all base clusterings have the same impact on the clustering results, assigning them the same weight. This makes the clustering performance susceptible to the influence of redundant, low-quality base clusterings. Second, co-association matrix-based algorithms often rely on additional methods, such as hierarchical agglomerative clustering, to obtain the final clustering result after constructing the weighted co-association matrix. This not only complicates optimization process but also leads to the loss of some sample-similarity information during clustering. To address this problem, we propose a novel Toward Balance Adaptive Weighted Ensemble Clustering (TBAWEC) algorithm. This method transforms the ensemble clustering problem into an optimization problem, producing the final result without requiring additional clustering algorithms. Moreover, we introduce balanced technology into ensemble clustering for the first time, significantly improving the balance of clustering results. Extensive experiments on real datasets demonstrate that the proposed algorithm outperforms the most advanced ensemble and balanced clustering algorithms simultaneously.

Clustering based on structured graph learning involves acquiring a proximity matrix with an explicit clustering structure from the original one. However, the original proximity matrix often lacks some must-links compared to the groundtruth, constraining the upper bound of clustering performance. High-order proximity information can mitigate this limitation, yet traditional high-order proximity matrix-based methods are time-intensive. To tackle this, we propose the Tensorized High-order Bipartite Graphs-based structured proximity matrix learning method (THBG). Firstly, we introduce a high-order bipartite graph proximity matrix with a swift computation method, incorporating high-order information and significantly reducing computational overhead. Secondly, we apply tensor nuclear norm minimization to the tensor composed of high-order bipartite graphs, learning a low-rank tensor representation that effectively harnesses the consistency of high-order information. Concurrently, a structured bipartite graph proximity matrix with an explicit clustering structure is adaptively learned based on the low-rank tensor representation and Laplace rank constraint. Experimental results demonstrate the superiority and great potential of this method. Code available:https://anonymous.4open.science/r/THBG-D10D.

In recent years, unsupervised linear regression has attracted attention for its ability to directly capture the mapping relationship between samples and targets. However, existing algorithms can only utilize limited information from a single view, which often leads to unsatisfactory results. To address this problem, we propose a regression clustering model based on multi-view information fusion, called Scalable Multi-view Regression Clustering. This model consists of two parts: intra-view information fusion and inter-view information fusion. In the first part, to capture the local correlations among samples, we propose constructing view-specific bipartite graphs. Unlike traditional single-view and multi-view clustering algorithms, we treat the weights of the bipartite graph as additional features of the samples, thereby directly incorporating the local manifold structure of the samples at the feature level. Furthermore, since the original features of the samples also contain valuable information, we perform unsupervised linear regression separately on the samples represented by the original features and those represented by the bipartite graph weights in each view. The results are then integrated in a weighted manner. In the second part, we propose adaptively weighting the clustering results from each view to capture complementary information across views, thereby enhancing clustering performance. This strategy not only avoids the bipartite graph alignment issue in multi-view clustering but also enables clustering with linear time complexity, making it effective for handling large-scale data. An iterative optimization algorithm is developed to update all variables alternately. Experiments conducted on benchmark datasets demonstrate the superiority of our proposed model.

Graph-based methods have demonstrated strong performance in multi-view clustering (MVC) due to their capability to capture complex data structures. Among these, discrete spectral embedding learning has emerged as an effective strategy for directly producing clustering assignments, thereby avoiding potential suboptimality introduced by post-processing. However, most existing discrete MVC methods overlook the problem of skewed cluster assignments, which can significantly affect the quality and interpretability of clustering results in practical applications. To address this issue, we propose a novel framework for Balanced and Discrete Multi-view Clustering via Adaptive Graph Learning (BDMC-AGL). The proposed model jointly integrates adaptive graph construction and size-constrained spectral embedding learning into a unified optimization framework, enhancing the robustness of clustering while explicitly encouraging balanced partitioning. The introduction of size constraints into the discrete spectral embedding, however, poses a challenging optimization problem. To effectively solve it, we develop an efficient algorithm that guarantees convergence to a locally optimal solution. Extensive experiments conducted on eight benchmark datasets demonstrate that BDMC-AGL consistently outperforms state-of-the-art methods in terms of clustering accuracy and balance. Moreover, ablation studies validate the significant contribution of the size constraint mechanism in improving multi-view clustering performance. The source code is publicly available at: https://github.com/haha1206/BDMC-AGL.

Broad learning system (BLS) is renowned for its excellent generalization and high efficiency in data classification. However, when confronted with class imbalance problems, BLS treats all samples as equally important, resulting in performance degradation. In addition, the presence of noise and outliers in imbalanced data further complicates BLS's ability to handle real-world classification problems. To address these challenges, this article proposes a graph-embedding intuitionistic fuzzy adaptive broad learning system (GEIB). The graph embedding strategy proposed by GEIB leverages the geometric topology of the data and class-specific information, effectively capturing variability among imbalanced samples and improving class separability. Furthermore, we introduce intuitionistic fuzzy theory. The BLS integrated with it considers both the homogeneity and heterogeneity of sample neighborhoods, enabling it to address uncertainty and imprecision in the data. It further differentiates clean samples from noisy ones in imbalanced datasets, thereby enhancing model robustness. To further investigate the prior distribution information of imbalanced data, we design an adaptive class-specific penalty mechanism based on global distribution and local density information. This mechanism accounts for both intraclass and interclass density information and class global distribution. We verify the superiority of our method by conducting a comparison with current approaches using real-world datasets that include both Gaussian noise and noise-free versions.

The clustering algorithm has always been a hot spot in machine learning, which has made great progress and been widely used in different scenarios. Due to the characteristics and requirements of some application scenarios, the branch of the balanced clustering algorithm has been developed. The ideal of these algorithms is to obtain clusters containing approximately the same number of samples. However, when there are data points distributed at the boundary of different clusters, resulting in different probabilities of their belonging, hard-partitioned balanced clustering may not be able to handle these boundary data well, thus limiting their performance. Motivated by this, we propose a Fuzzy Min-Cut with Soft Balancing Effects (FCBE) method in this article. Specifically, the FCBE model utilizes fuzzy constraints to simultaneously enhance the ability of the balanced algorithm to capture boundary data members and the advantage of directly obtaining the partitioning results of graph-cut problem without postprocessing. In addition, a sparse regularization is introduced to avoid trivial solutions and maintain the separability of the relationship matrix. Furthermore, the proposed FCBE method can be viewed as a flexibly adjustable generalization pattern that not only has clear interpretability but also can become special cases with clear physical meanings under different parameter values. The feasibility of FCBE has been verified on real datasets.

Traditional partition-based fuzzy clustering algorithms are widely used for revealing possible hidden structures in data. However, high computational cost limits their applications in large-scale and high-dimensional data. Moreover, most fuzzy clustering algorithms are sensitive to noise. To tackle these issues, a robust jointly sparse fast fuzzy clustering algorithm via anchor graph (RSFCAG) is proposed and analyzed in this paper. Specifically, we first propose a fast k-means method integrated shadowed set and balanced ternary tree, which serves as a fast hierarchical clustering approach by partitioning every cluster into three subclusters at each layer (3KHK). 3KHK can quickly obtain the anchor set and its optimization is solved fast by the simplex method, which also captures the ambiguity and uncertainty between clusters in large-scale clustering tasks. Second, a similarity matrix learning approach based on possibilistic neighbors is further proposed to get a robust similarity graph, which strengthens the ability of fuzzy clustering to handle large-scale data. Furthermore, the orthogonal projection matrix is integrated into the RSFCAG framework to transform the original high-dimensional space into low-dimensional space. Finally, the$L_{2,1}$-norm loss and regularization are integrated into the joint algorithm RSFCAG, which is solved optimally by block coordinate technique, to enhance the robustness and interpretability of the fuzzy clustering process. The experimental results demonstrate the effectiveness and efficiency of our proposed method in most of benchmark datasets.

Graph clustering typically involves a two-step process: relaxation followed by post-processing. However, it often leads to significant information loss during relaxation and solution deviation in post-processing. Additionally, traditional graph clustering faces computational challenges due to regular graph construction and spectral decomposition, and binary indicators hinder interpretability in uncertain scenarios. We propose a novel method termed Fast Fuzzy Graph Cut (FFGC) to overcome key issues in graph clustering by: preventing information loss by tackling the original graph cut problem; eliminating solution deviation by directly solving for the target variable; alleviating computational burdens by employing anchor graphs in place of regular graphs; and enhancing flexibility by incorporating a regularization term to soften the cluster indicator. The use of a fuzzy cluster indicator within the graph cut framework expands FFGC's applicability to a wider range of real-world data, increasing both its adaptability and interpretability. In addition, we develop two efficient optimization algorithms to solve the resulting objective problem. Extensive experimental evaluations validate the superior efficiency and effectiveness of FFGC in clustering tasks. The code is available athttps://github.com/caccode/FFGC.

Clusteringis an important unsupervised learning technique widely applied in data analysis and pattern recognition. Graph-based clustering methods have gained attention for their ability to effectively model complex data structures. However, traditional methods mainly rely on first-order proximity information, which struggles to capture high-order structural relationships between data points. Such limitation can significantly degrade clustering performance. To address this issue, we propose a novel fuzzy clustering approach leveraging orthogonal tensor decomposition on high-order anchor graphs (OTDHAG). Unlike conventional high-order graph-based methods that rely on self-multiplication of proximity matrices, which are computationally expensive, our method introduces the high-order anchor graph with low computational complexity, and exploits high-order proximity information via orthogonal tensor factorization on high-order anchor graphs. Meanwhile, tensor nuclear norm regularization is adopted to enhance clustering consistency, thereby improving clustering quality. Experiments on 10 benchmark datasets (e.g., MNIST, Wine, PIE) demonstrate that OTDHAG outperforms 14 state-of-the-art methods, achieving 92.3% accuracy on Wine (vs. 86.5% for DenoHOG) and 88.7% NMI on MNIST (vs. 82.1% for AOPL-Root). In conclusion, OTDHAG not only significantly improves clustering accuracy and computational efficiency but also provides a scalable solution for clustering tasks. Code available:https://anonymous.4open.science/r/HOAGTD-CED0.

The surge in remote sensing (RS) data underscores the need for improved data diversity and processing. While integrating hyperspectral (HS) and light detection and ranging (LiDAR) data enhances analysis and addresses spectral variability, the high dimensionality, noise, and outliers inherent in hyperspectral images present significant challenges. In addition, the precise labeling required for HS makes supervised classification labor-intensive, professional-focused, and time-consuming, further motivating the development of advanced HS clustering algorithms to address these issues. Unsupervised clustering addresses the above issues but still struggles due to the underutilization of auxiliary spatial and structural information, high data dimensionality with redundant hyperspectral bands, and information divergence from heterogeneity among multimodal data. These challenges impede the effective extraction of consistent structures, undermining clustering stability and overall model performance. To address these challenges, we propose a superpixel-based bipartite graph clustering (SBGC) enriched with spatial information for hyperspectral and LiDAR data models. Our proposed method fully utilizes spatial information to construct meaningful bipartite graphs for the efficient processing of multimodal RS data. By adopting a projected clustering paradigm, our approach simultaneously clusters and reduces dimensionality, effectively eliminating redundant bands. In addition, it innovatively stacks multimodal data into tensors, thoroughly exploring the consistent structures in the low-rank space among different modalities. This reduces the heterogeneity-induced information divergence and significantly enhances clustering performance. Extensive experiments on real datasets confirm the method’s effectiveness and advanced capabilities.

Label noise in hyperspectral image classification (HIC) has posed a significant challenge, mainly due to the complexity of high-dimensional spectral-spatial data. Existing approaches that rely on sample predictions for verification and correction often amplify confirmation bias, leading to decision boundaries that overfit to noisy labels. To address these issues, we propose a Manifold-Aligned Consistency Contrastive Learning (MACCL) framework that establishes a mutually-guided consistency alignment mechanism between the spectral-spatial representation space and label space through manifold learning theory to combat label noise. Specifically, to mitigate the confirmation bias of noisy labels, we introduce a manifold-aligned consistency learning module. It leverages the manifold assumption in the representation space, modeling local neighborhood graphs and enforcing consistent prediction distributions via KL divergence. This aligns label-space classifications with the representation space’s local geometry, suppressing isolated noise through manifold continuity. Additionally, to combat representation degradation that causes decision boundaries to overfit noisy labels, we integrate a noise-tolerant contrastive representation learning module. By applying confidence-guided criteria, the module focuses on high-confidence sample pairs and regularizes gradients. This emphasizes clean pairs during training, boosting the model’s discriminative ability and preserving true semantic relationships. Through this mutual guidance, the contrastive learning refines the local geometric structure through discriminative representation learning, driving the representation space closer to the intrinsic data manifold, while the manifold alignment propagates geometric constraints to rectify label space corruptions. Finally, experiments on several benchmark datasets with varying levels of noise have validated the superiority of the proposed MACCL framework.

Existing deep clustering methods leverage contrastive or non-contrastive learning to facilitate downstream tasks. Most contrastive-based methods typically learn representations by comparing positive pairs (two views of the same sample) against negative pairs (views of different samples). However, we spot that this hard treatment of samples ignores inter-sample relationships, leading to class collisions and degrade clustering performances. In this paper, we propose a soft neighbor supported contrastive clustering method to address this issue. Specifically, we first introduce a concept called perception radius to quantify similarity confidence between a sample and its neighbors. Based on this insight, we design a two-level soft neighbor loss that captures both local and global neighborhood relationships. Additionally, a cluster-level loss enforces compact and well-separated cluster distributions. Finally, we conduct a pseudo-label refinement strategy to mitigate false negative samples. Extensive experiments on benchmark datasets demonstrate the superiority of our method. The code is available at https://github.com/DuannYu/soft-neighbors-supported-clustering.

As we all known, sparse subspace learning can provide good input for spectral clustering, thereby producing high-quality cluster partitioning. However, it employs complete samples as the dictionary for representation learning, resulting in non-negligible computational costs. Therefore, replacing the complete samples with representative ones (anchors) as the dictionary has become a more popular choice, giving rise to a series of related works. Unfortunately, although these works are linear with respect to the number of samples, they are often quadratic or even cubic with respect to the number of anchors. In this paper, we derive a simpler problem to replace the original scalable subspace clustering, whose properties are utilized. This new problem is linear with respect to both the number of samples and anchors, further enhancing scalability and providing more efficient operations. Furthermore, thanks to the new problem formulation, we can adopt a separate fusion strategy for multi-view extensions. This strategy can better measure the inter-view difference and avoid alternate optimization, so as to achieve more robust and efficient multi-view clustering. Finally, comprehensive experiments demonstrate that our methods not only significantly reduce time overhead but also exhibit superior performance.

Domain adaptation aims to leverage abundant label information from a source domain to an unlabeled target domain with two different distributions. Existing methods usually rely on a classifier to generate high-quality pseudo-labels for the target domain, facilitating the learning of discriminative features. Label propagation (LP), as an effective classifier, propagates labels from the source domain to the target domain by designing a smooth function over a similarity graph, which represents structural relationships among data points in feature space. However, LP has not been thoroughly explored in deep neural network-based domain adaptation approaches. Additionally, the probability labels generated by LP are low-confident and LP is sensitive to class imbalance problem. To address these problems, we propose a novel approach for domain adaptation named deep label propagation with nuclear norm maximization (DLP-NNM). Specifically, we employ the constraint of nuclear norm maximization to enhance both label confidence and class diversity in LP and propose an efficient algorithm to solve the corresponding optimization problem. Subsequently, we utilize the proposed LP to guide the classifier layer in a deep discriminative adaptation network using the cross-entropy loss. As such, the network could produce more reliable predictions for the target domain, thereby facilitating more effective discriminative feature learning. Extensive experimental results on three cross-domain benchmark datasets demonstrate that the proposed DLP-NNM surpasses existing state-of-the-art domain adaptation approaches.

Principal Component Analysis (PCA) is one of the most important unsupervised dimensionality reduction algorithms, which uses squared -norm to make it very sensitive to outliers. Those improved versions based on -norm alleviate this problem, but they have other shortcomings, such as optimization difficulties or lack of rotational invariance, etc. Besides, existing methods only vaguely divide normal samples and outliers to improve robustness, but they ignore the fact that normal samples can be more specifically divided into positive samples and hard samples, which should have different contributions to the model because positive samples are more conducive to learning the projection matrix. In this paper, we propose a novel Data Subdivision Based Dual-Weighted Robust Principal Component Analysis, namely DRPCA, which firstly designs a mark vector to distinguish normal samples and outliers, and directly removes outliers according to mark weights. Moreover, we further divide normal samples into positive samples and hard samples by self-constrained weights, and place them in relative positions, so that the weight of positive samples is larger than hard samples, which makes the projection matrix more accurate. Additionally, the optimal mean is employed to obtain a more accurate data center. To solve this problem, we carefully design an effective iterative algorithm and analyze its convergence. Experiments on real-world and RGB large-scale datasets demonstrate the superiority of our method in dimensionality reduction and anomaly detection.

Label propagation (LP) is a popular semi-supervised learning technique that propagates labels from a training dataset to a test one using a similarity graph, assuming that nearby samples should have similar labels. However, the recent cross-domain problem assumes that training (source domain) and test data sets (target domain) follow different distributions, which may unexpectedly degrade the performance of LP due to small similarity weights connecting the two domains. To address this problem, we propose optimal graph learning-based label propagation (OGL2P), which optimizes one cross-domain graph and two intra-domain graphs to connect the two domains and preserve domain-specific structures, respectively. During label propagation, the cross-domain graph draws two labels close if they are nearby in feature space and from different domains, while the intra-domain graph pulls two labels close if they are nearby in feature space and from the same domain. This makes label propagation more insensitive to cross-domain problems. During graph embedding, we optimize the three graphs using features and labels in the embedded subspace to extract locally discriminative and domain-invariant features and make the graph construction process robust to noise in the original feature space. Notably, as a more relaxed constraint, locally discriminative and domain-invariant can somewhat alleviate the contradiction between discriminability and domain-invariance. Finally, we conduct extensive experiments on five cross-domain image classification datasets to verify that OGL2P outperforms some state-of-the-art cross-domain approaches.

Principal Component Analysis (PCA) is one of the most famous unsupervised dimensionality reduction algorithms and has been widely used in many fields. However, it is very sensitive to outliers, which reduces the robustness of the algorithm. In recent years, many studies have tried to employ \(\ell_{1}\) -norm to improve the robustness of PCA, but they all lack rotation invariance or the solution is expensive. In this article, we propose a novel robust PCA, namely, Fuzzy Weighted Principal Component Analysis (FWPCA), which still uses squared \(\ell_{2}\) -norm to minimize reconstruction error and maintains rotation invariance of PCA. The biggest bright spot is that the contribution of data is restricted by fuzzy weights, so that the contribution of normal samples is much greater than noise or abnormal data, and realizes anomaly detection. Besides, a more reasonable data center can be obtained by solving the optimal mean to make projection matrix more accurate. Subsequently, an effective iterative optimization algorithm is developed to solve this problem, and its convergence is strictly proved. Extensive experimental results on face datasets and RGB anomaly detection datasets show the superiority of our proposed method.

Effectively and efficiently mining valuable clustering patterns is a challenging problem when handling large-scale data from diverse sources. Existing approaches adopt anchor graph learning or binary representation embedding to reduce computational complexity. Normally, anchor graph learning can not directly obtain the clustering assignment except adopt the post-processing stage, such as graph cut or k-means clustering. The binary representation embedding neglects the structure information in Hamming space. In order to overcome these limitations, this paper proposes a novel, effective, and efficient angular reconstructive discrete embedding method with fusion similarity for a multi-view clustering (AFMC) that can jointly learn the global and local structure preserving binary representation and clustering assignment. Specifically, we propose to use angular reconstructive error minimization to maintain the global similarity correlation of binary representations of heterogeneous features in a common Hamming space. Moreover, we design a multi-view discrete ridge regression with fusion similarity term to handle the out-of-sample problem and preserve the local manifold structure. In addition, we propose an efficient optimization algorithm with linear computational complexity to solve the non-convex and non-smooth objective function. The experimental results demonstrate that AFMC outperforms several state-of-the-art large-scale multi-view clustering methods.

Graph clustering has become a crucial technique for uncovering community structures in complex network data. However, existing approaches often introduce cumbersome regularization or constraints (hyperparameter tuning burden) to obtain balanced clustering results, thereby increasing hyperparameter tuning requirements and intermediate variables. These limitations can lead to suboptimal performance, particularly in scenarios involving imbalanced clusters or large-scale datasets. Besides, most graph cut clustering methods solve two separate discrete problems, resulting in information loss and relying on time-consuming eigen-decomposition. To address these challenges, this paper propose an effective graph cut framework, termed Harmonic MaxMin Cut (HMMC), inspired by worst-case objective optimization and the harmonic mean. Unlike traditional spectral clustering, HMMC produces all cluster assignments in a single step, eliminating the need for additional discretization and notably enhancing robustness to “worst-case cluster” boundaries. this paper further devise a fast coordinate descent (CD) solver that scales linearly complexity with the graph size, offering a computationally efficient alternative to eigen decomposition. Extensive experiments on real-world datasets demonstrate that HMMC is comparable to, or even surpasses, state-of-the-art methods, while also finding more favorable local solutions than non-negative matrix factorization techniques.

As a compromise between supervised and unsupervised learning, semi-supervised learning (SSL) harnesses both labeled and unlabeled data to enhance learning performance. Graph-based semi-supervised learning (GSSL) has emerged as a prominent approach owing to its versatility in representing sample interdependencies via graph structures. However, traditional GSSL methods face high time cost when computing matrix inverses, making them inefficient for large datasets. To address this, some researchers have introduced anchors as a bridge to accelerate the process. Nevertheless, most anchor-based models suffer from one or more of the following issues: (1) The anchor graph-based construction of the adjacency matrix has limitations; (2) The objective functions are typically non-convex, leading to local optima and requiring multiple runs to achieve good performance. To tackle these challenges, we develop a probability-driven approach to build the adjacency matrix, defining sample similarity as the probability of sharing the same anchor. Based on this strategy, we design a model (CFSL) with a strictly convex objective function, guaranteeing a globally optimal solution without iterative optimization. Experiments on multiple datasets indicate that our algorithm yields strong performance.

Traditional spectral clustering methods struggle with scalability and robustness in large datasets due to their reliance on similarity matrices and EigenValue Decomposition. We introduce two innovative models: Rcut-based Coordinate Descent Clustering (R-CDC) and Ncut-based Doubly Stochastic Clustering (N-DSC). These models integrate graph construction and segmentation into a unified process optimized through the coordinate descent method, significantly enhancing clustering efficacy. A novel graph structure enhances robustness against noise and outliers, simplifying the clustering process and improving outcomes across diverse datasets. Our extensive experiments show that these models surpass existing spectral clustering techniques in managing large-scale data and complex structures.

This paper proposes a deep pseudo-label method for unsupervised feature selection, which learns non-linear representations to generate pseudo-labels and trains a Neural Network (NN) to select informative features via self-Knowledge Distillation (KD). Specifically, the proposed method divides a standard NN into two sub-components: an encoder and a predictor, and introduces a dependency subnet. It works by self-supervised pretraining the encoder to produce informative representations and then alternating between two steps: (1) learning pseudo-labels by combining the clustering results of the encoder's outputs with the NN's prediction outputs, and (2) updating the NN's parameters by globally selecting a subset of features to predict the pseudo-labels while updating the subnet's parameters through self-KD. Self-KD is achieved by encouraging the subnet to locally capture a subset of the NN features to produce class probabilities that match those produced by the NN. This allows the model to self-absorb the learned inter-class knowledge and evaluate feature diversity, removing redundant features without sacrificing performance. Meanwhile, the potential discriminative capability of a NN can also be self-excavated without the assistance of other NNs. The two alternate steps reinforce each other: in step (2), by predicting the learned pseudo-labels and conducting selfKD, the discrimination of the outputs of both the NN and the encoder is gradually enhanced, while the self-labeling method in step (1) leverages these two improvements to further refine the pseudo-labels for step (2), resulting in the superior performance. Extensive experiments show the proposed method significantly outperforms state-of-the-art methods across various datasets.

Graph-based clustering technique has garnered significant attention due to precise information characterization by pairwise graph similarity. Nevertheless, the post-processing step in traditional methods often limits clustering effects because of crucial information loss. Therefore, the Constrained Laplacian Rank (CLR) theory emerges to directly obtain discrete labels from optimally structural graph, achieving desirable outcomes. However, CLR suffers from substantial time overhead, making it infeasible for large-scale data analysis. To overcome this issue, we propose Anchor-based CLR (ACLR), a simple yet effective method for efficient large-scale clustering. The ACLR method comprises four stages: (1) anchors that roughly cover original data are opted to prepare bipartite graph construction; (2) a novel two-step probability transition (TSPT) strategy initializes a small-scale graph with random walk probability among anchors; (3) the main ACLR model alternately optimizes the graph connected structure and directly produces discrete anchor labels, achieving a time complexity independent of the number of samples due to dramatically reduced graph scale; and (4) labels are propagated from anchors to samples using$K$-NN algorithm. Extensive experiments demonstrate that ACLR yields superior accuracy and efficiency, particularly when applied to large-scale data. The codes are available athttps://github.com/MarathonZhenyuMa/2025-TKDE-ACLR.

Fuzzy c-means based on entropy regularization (FCER) is a commonly used machine learning algorithm that uses maximum entropy as the regularization term to realize fuzzy clustering. However, this model has many constraints and is challenging to optimize directly. During the solution process, the membership matrix and cluster centers are alternately optimized, easily converging to poor local solutions, limiting the clustering performance. In this paper, we start with the optimization model and propose an unconstrained fuzzy clustering model (UFCER) equivalent to FCER, which reduces the size of optimization variables from$(n+d)\times c$to$d\times c$. More importantly, there is no need to calculate the membership matrix during the optimization process iteratively. The time complexity is only linear, and the convergence speed is fast. We conduct extensive experiments on real datasets. The comparison of objective function value and clustering performance fully demonstrates that under the same initialization, our proposed algorithm can converge to smaller local minimums and get better clustering performance.

Most existing multi-view graph clustering models focus on integrating the topological structure of different views directly, which cannot efficiently stimulate the collaboration between multiple views. To alleviate this problem, this paper proposes a Triangle Topology Enhancement (T2E) module, which expands two topological structures based on the raw topology of each view, including the self-triangle enhanced topology that highlights the local view information and the cross-view triangle enhanced topology containing the global-local view information. Afterward, this paper designs a novel multi-view graph clustering model, named MGC-T2E, to integrate both the raw and derived topological structures and directly induce consistent clustering indicators based on a self-supervised clustering module. In the simulation, the experimental results demonstrate that MGC-T2E achieves state-of-the-art performances compared with a mass of current competitors.

Anchor-based clustering methods have attracted increasing attention due to their ability to provide efficient and scalable solutions in clustering tasks, such as subspace, multi-view and ensemble clustering. Nevertheless, the majority of anchor-based methods view anchors merely as tools, concentrating on diminishing computational complexity within original data space. However, in fact, clustering can be directly performed on anchors and then the anchor clustering results could be propagated to original data. Due to the much smaller volume of anchors, this could significantly reduce the computational complexity of clustering algorithms. Building upon this idea, in this paper, we propose a fast anchor graph clustering method (FAGC) via maximizing within-cluster similarity. Inspired by the relaxation and discretization model in spectral clustering, we also propose two corresponding models, namely FAGC-R and FAGC-D. FAGC-R first obtains spectral embedding of anchors and then discretizes the embedding to obtain anchor indicator matrix. While FAGC-D directly solves the discrete anchor membership matrix. Once anchor clustering results are obtained, original data labels can be obtained through anchor label transmission. Extensive experiments conducted on synthetic and real datasets illustrate the effectiveness and efficiency of the proposed methods.

Multi-view spectral clustering has attracted considerable attention since it can explore common geometric structures from diverse views. Nevertheless, existing min-min framework-based models adopt internal minimization to find the view combination with the minimized within-cluster variance, which will lead to effectiveness loss since the real clusters often exhibit high within-cluster variance. To address this issue, we provide a novel scalable min-max multi-view spectral clustering (SMMSC) model to improve clustering performance. Besides, anchor graphs, rather than full sample graphs, are utilized to reduce the computational complexity of graph construction and singular value decomposition, thereby enhancing the applicability of SMMSC to large-scale applications. Then, we rewrite the min-max model as a minimized optimal value function, demonstrate its differentiability, and develop an efficient gradient descent-based algorithm to optimize it with linear computational complexity. Moreover, we demonstrate that the resultant solution of the proposed algorithm is the global optimum. Numerous experiments on different real-world datasets, including some large-scale datasets, demonstrate that SMMSC outperforms existing state-of-the-art multi-view clustering methods regarding clustering performance.

In practical applications, multi-view subspace clustering is hindered by data noise that disrupts the ideal block-diagonal structure of self-representation matrices, thereby degrading performance. Moreover, many existing methods rely solely on sample features, overlooking the valuable structural information in affinity matrices (e.g., pairwise relationships). While conventional contrastive learning strategies often introduce false negative pairs due to noise and unreliable sample selection. To address these challenges, we propose a pseudo-label guided bidirectional discriminative deep multi-view subspace clustering method (PBDMSC). Our approach first employs pseudo-label guided contrastive learning, using previous cluster assignments to select reliable positive and negative samples, which mitigates incorrect pairings and enhances low-dimensional representations. Then, a discriminative self-representation learning method is introduced that leverages pseudo-labels to enforce homogeneous expression constraints and incorporates a bidirectional attention mechanism to preserve the structured information from affinity matrices, thereby enhancing robustness. Experimental results on six real-world datasets demonstrate that our proposed method achieves state-of-the-art clustering performance.

Unsupervised domain adaptation aims to classify unlabeled data points in the target domain using labeled data points from the source domain, while the distributions of data points in two domains are different. To address this issue, we propose a novel method called the anchor guided unsupervised domain adaptation method (AGDA). We minimize distribution divergence in a latent feature subspace using the Maximum Mean Discrepancy (MMD) criterion. Unlike existing unsupervised domain adaptation methods, we introduce anchor points in the original space and impose domains data to the same anchor points rather than center points to further reduce the domain difference. We optimize the anchor-based graph in the subspace to obtain discriminative transformation matrices. This enables our model to perform better on non-Gaussian distribution than methods focusing on global structure. Furthermore, the sparse anchor-based graph reduces time complexity compared to the fully connected graph, enabling exploration of local structure. Experimental results demonstrate that our algorithm outperforms several state-of-the-art methods on various benchmark datasets.

Structured proximity matrix learning, one of the mainstream directions in clustering research, refers to learning a proximity matrix with an explicit clustering structure from the original first-order proximity matrix. Due to the complexity of the data structure, the original first-order proximity matrix always lacks some must-links compared to the groundtruth proximity matrix. It is worth noting that high-order proximity matrices can provide missed must-link information. However, the computation of high-order proximity matrices and clustering based on them are expensive. To solve the above problem, inspired by the anchor bipartite graph, we present a novel high-order bipartite graph proximity matrix and a fast method to compute it. This proposed high-order bipartite graph proximity matrix contains high-order proximity information and can significantly reduce the computational complexity of the whole clustering process. Furthermore, we introduce an efficient and simple high-order bipartite graph fusion framework that can adaptively assign weights to each order of the high-order bipartite graph matrices. Finally, under the Laplace rank constraint, a consensus structured bipartite graph proximity matrix is obtained. At the same time, an efficient solution algorithm is proposed for this model. The model's efficacy is underscored through rigorous experiments, highlighting its superior clustering performance and time efficiency. Code available:https://anonymous.4open.science/r/HBGC-F6C4.

Ensemble clustering can utilize the complementary information among multiple base clusterings, and obtain a clustering model with better performance and more robustness. Despite its great success, there are still two problems in the current ensemble clustering methods. First, most ensemble clustering methods often treat all base clusterings equally. Second, the final ensemble clustering result often relies on$k$-means or other discretization procedures to uncover the clustering indicators, thus obtaining unsatisfactory results. To address these issues, we proposed a novel ensemble clustering method based on structured graph learning, which can directly extract clustering indicators from the obtained similarity matrix. Moreover, our methods take sufficient consideration of correlation among the base clusterings and can effectively reduce the redundancy among them. Extensive experiments on artificial and real-world datasets demonstrate the efficiency and effectiveness of our methods.

Multiview clustering (MVC) aims to integrate multiple related but different views of data to achieve more accurate clustering performance. Contrastive learning has found many applications in MVC due to its successful performance in unsupervised visual representation learning. However, existing MVC methods based on contrastive learning overlook the potential of high similarity nearest neighbors as positive pairs. In addition, these methods do not capture the multilevel (i.e., cluster, instance, and prototype levels) representational structure that naturally exists in multiview datasets. These limitations could further hinder the structural compactness of learned multiview representations. To address these issues, we propose a novel end-to-end deep MVC method called multilevel contrastive MVC (MCMC) with dual self-supervised learning (DSL). Specifically, we first treat the nearest neighbors of an object from the latent subspace as the positive pairs for multiview contrastive loss, which improves the compactness of the representation at the instance level. Second, we perform multilevel contrastive learning (MCL) on clusters, instances, and prototypes to capture the multilevel representational structure underlying the multiview data in the latent space. In addition, we learn consistent cluster assignments for MVC by adopting a DSL method to associate different level structural representations. The evaluation experiment showed that MCMC can achieve intracluster compactness, intercluster separability, and higher accuracy (ACC) in clustering performance. Our code is available at https://github.com/bianjt-morning/MCMC.

Due to the excellent performance of spectral clustering (SC), it has been widely used in many fields of application. However, the high computational complexity and two successive steps have limited SC's development. In addition, the traditional SC is formulated to maximize the arithmetic mean of trace ratios which is dominated by the larger objectives and may reduce the recognition accuracy in practical applications. In this article, we propose a novel graph cut criterion to minimize the trace ratios of harmonic mean with objectives, which can avoid the worst-cluster issue without imposing any regularization or constraints. Furthermore, an efficient and effective coordinate descent (CD) method is exploited to achieve a one-step solution. Therefore, this article can simultaneously solve three main challenges in a unified framework. Extensive experiments verify that the harmonic fast one-step graph cut (HFOC) achieves superior clustering performance with relatively less time-consuming compared to the other state-of-the-art clustering methods.

Feature selection plays an important role in data analysis, yet traditional graph-based methods often produce suboptimal results. These methods typically follow a two-stage process: constructing a graph with data-to-data affinities or a bipartite graph with data-to-anchor affinities and independently selecting features based on their scores. In this article, a large-scale feature selection approach based on structured bipartite graph and row-sparse projection (RS2BLFS) is proposed to overcome this limitation. RS2BLFS integrates the construction of a structured bipartite graph consisting of c connected components into row-sparse projection learning with k nonzero rows. This integration allows for the joint selection of an optimal feature subset in an unsupervised manner. Notably, the c connected components of the structured bipartite graph correspond to c clusters, each with multiple subcluster centers. This feature makes RS2BLFS particularly effective for feature selection and clustering on nonspherical large-scale data. An algorithm with theoretical analysis is developed to solve the optimization problem involved in RS2BLFS. Experimental results on synthetic and real-world datasets confirm its effectiveness in feature selection tasks.

The goal of balanced clustering is partitioning data into distinct groups of equal size. Previous studies have attempted to address this problem by designing balanced regularizers or utilizing conventional clustering methods. However, these methods often rely solely on classic methods, which limits their performance and primarily focuses on low-dimensional data. Although neural networks exhibit effective performance on high-dimensional datasets, they struggle to effectively leverage prior knowledge for clustering with a balanced tendency. To overcome the above limitations, we propose deep semisupervised balanced clustering, which simultaneously learns clustering and generates balance-favorable representations. Our model is based on the autoencoder paradigm incorporating a semisupervised module. Specifically, we introduce a balance-oriented clustering loss and incorporate pairwise constraints into the penalty term as a pluggable module using the Lagrangian multiplier method. Theoretically, we ensure that the proposed model maintains a balanced orientation and provides a comprehensive optimization process. Empirically, we conducted extensive experiments on four datasets to demonstrate significant improvements in clustering performance and balanced measurements. Our code is available at https://github.com/DuannYu/BalancedSemi-TNNLS.

This study addresses the problem of generalized category discovery (GCD), an advanced and challenging semi-supervised learning scenario that deals with unlabeled data from both known and novel categories. Although recent research has effectively engaged with this issue, these studies typically map features into Euclidean space, which fails to maintain the latent semantic hierarchy of the training samples effectively. This limitation restricts the exploration of more detailed and rich information and degrades the performance in discovering new categories. The emerging field of hyperbolic representation learning suggests that hyperbolic geometry could be advantageous for extracting semantic information to tackle this problem. Motivated by this, we proposed hyperbolic hierarchical representation learning for GCD (HypGCD). Specifically, HypGCD enhances representations in hyperbolic space, building upon the Euclidean space representation from two perspectives: instance-class level and instance-instance level. At the instance-class level, HypGCD endeavors to construct well-defined clusters, with each sample forming a robust hierarchical cluster structure. Concurrently, at the instance-instance level, HypGCD anticipates that a subset of samples will display a tree-like structure in local space, which aligns more closely with real-world scenarios. Finally, HypGCD optimizes the Euclidean and hyperbolic space collectively to obtain refined features. Additionally, we show that HypGCD is exceptionally effective, achieving state-of-the-art (SOTA) results on several datasets. The code is available at https://github.com/DuannYu/HypGCD.

Boosting is a well-established ensemble learning approach that aims to enhance overall performance by combining multiple weak learners with a linear combination structure. It operates on the principle of using new learners to compensate for the shortcomings of previous learners and is known for its ability to reduce computational resource requirements while mitigating the risks of overfitting. However, from the perspective of convex optimization, it becomes apparent that classical boosting methods often converge to local optima rather than global optima when minimizing the target loss due to its greedy strategy. In this article, we address the issue and propose a novel optimization framework for the boosting paradigm. Our framework focuses on refining the ensemble model by further minimizing loss function through the reallocation of base learner weights, which results in a more robust and powerful learner. We have conducted experiments on various real-world and synthetic datasets, and our findings confirm that our Reweighted-Boosting model consistently outperforms its counterparts. It also exhibits an increased classification margin for the data, making it a valuable enhancement to original boosting algorithms.

Linear discriminant analysis (LDA) is a classic tool for supervised dimensionality reduction. Because the projected samples can be classified effectively, LDA has been successfully applied in many applications. Among the variants of LDA, trace ratio LDA (TR-LDA) is a classic form due to its explicit meaning. Unfortunately, when the sample size is much smaller than the data dimension, the algorithm for solving TR-LDA does not converge. The so-called small sample size (SSS) problem severely limits the application of TR-LDA. To solve this problem, we propose a revised formation of TR-LDA, which can be applied to datasets with different sizes in a unified form. Then, we present an optimization algorithm to solve the proposed method, explain why it can avoid the SSS problem, and analyze the convergence and computational complexity of the optimization algorithm. Next, based on the introduced theorems, we quantitatively elaborate on when the SSS problem will occur in TR-LDA. Finally, the experimental results on real-world datasets demonstrate the effectiveness of the proposed method.

Many recent research works on unsupervised feature selection (UFS) have focused on how to exploit autoencoders (AEs) to seek informative features. However, existing methods typically employ the squared error to estimate the data reconstruction, which amplifies the negative effect of outliers and can lead to performance degradation. Moreover, traditional AEs aim to extract latent features that capture intrinsic information of the data for accurate data recovery. Without incorporating explicit cluster structure-detecting objectives into the training criterion, AEs fail to capture the latent cluster structure of the data which is essential for identifying discriminative features. Thus, the selected features lack strong discriminative power. To address the issues, we propose to jointly perform robust feature selection and -means clustering in a unified framework. Concretely, we exploit an AE with a -norm as a basic model to seek informative features. To improve robustness against outliers, we introduce an adaptive weight vector for the data reconstruction terms of AE, which assigns smaller weights to the data with larger errors to automatically reduce the influence of the outliers, and larger weights to the data with smaller errors to strengthen the influence of clean data. To enhance the discriminative power of the selected features, we incorporate -means clustering into the representation learning of the AE. This allows the AE to continually explore cluster structure information, which can be used to discover more discriminative features. Then, we also present an efficient approach to solve the objective of the corresponding problem. Extensive experiments on various benchmark datasets are provided, which clearly demonstrate that the proposed method outperforms state-of-the-art methods.

In existing multiview clustering research, the comprehensive learning from multiview graph and feature spaces simultaneously remains insufficient when achieving a consistent clustering structure. In addition, a postprocessing step is often required. In light of these considerations, a cross-view approximation on Grassman manifold (CAGM) model is proposed to address inconsistencies within multiview adjacency matrices, feature matrices, and cross-view combinations from the two sources. The model uses a ratio-formed objective function, enabling parameter-free bidirectional fusion. Furthermore, the CAGM model incorporates a paired encoding mechanism to generate low-dimensional and orthogonal cross-view embeddings. Through the approximation of two measurable subspaces on the Grassmann manifold, the direct acquisition of the indicator matrix is realized. Furthermore, an effective optimization algorithm corresponding to the CAGM model is derived. Comprehensive experiments on four real-world datasets are conducted to substantiate the effectiveness of our proposed method.

Recently, more and more real-world datasets have been composed of heterogeneous but related features from diverse views. Multiview clustering provides a promising attempt at a solution for partitioning such data according to heterogeneous information. However, most existing methods suffer from hyper-parameter tuning trouble and high computational cost. Besides, there is still an opportunity for improvement in clustering performance. To this end, a novel multiview framework, called parameter-free multiview -means clustering with coordinate descent method (PFMVKM), is presented to address the above problems. Specifically, PFMVKM is completely parameter-free and learns the weights via a self-weighted scheme, which can avoid the intractable process of hyper-parameters tuning. Moreover, our model is capable of directly calculating the cluster indicator matrix, with no need to learn the cluster centroid matrix and the indicator matrix simultaneously as previous multiview methods have to do. What's more, we propose an efficient optimization algorithm utilizing the idea of coordinate descent, which can not only reduce the computational complexity but also improve the clustering performance. Extensive experiments on various types of real datasets illustrate that the proposed method outperforms existing state-of-the-art competitors and conforms well with the actual situation.

Graph-based clustering is a hot topic in machine learning, whose effectiveness highly relies on the quality of the learned graph. Recent researches preferred to learn the nearest doubly stochastic approximation of a graph to suppress intercluster connections and enhance intracluster connections and thus improve clustering performance. While current paradigm is limited by three key problems: 1) it is restricted by a predefined graph; 2) the separated stages of spectral decomposition-based way (graph learning, spectral embedding learning, and cluster assignment by k-means) cause mismatched problems and randomness; and 3) the optimization of doubly stochastic conditions is generally achieved by von Neumann successive projection (VNSP) lemma, which separates the conditions to form two subproblems for alternative optimization, converging only to a feasible solution. To solve these problems, in this article, a novel structured doubly stochastic graph-based clustering model termed SDSGC is proposed, which learns a structured doubly stochastic graph from data to directly provide cluster indicators. For optimization, a simple but effective augmented Lagrangian multiplier (ALM)-based method is proposed, which optimizes all the doubly stochastic conditions simultaneously to obtain the optimal solution. Experiments on one toy dataset and eight ad hoc noised face datasets have demonstrated that the proposed SDSGC is more robust to noise. Furthermore, a quantitative comparison of ten benchmarks has verified our SDSGC achieves better clustering performance when compared with SOTA methods. The code is available at https://github.com/NianWang-HJJGCDX/SDSGC.git.

The training process of a domain generalization (DG) model involves utilizing one or more interrelated source domains to attain optimal performance on an unseen target domain. Existing DG methods often use auxiliary networks or require high computational costs to improve the model's generalization ability by incorporating a diverse set of source domains. In contrast, this work proposes a method called Smooth-Guided Implicit Data Augmentation (SGIDA) that operates in the feature space to capture the diversity of source domains. To amplify the model's generalization capacity, a distance metric learning (DML) loss function is incorporated. Additionally, rather than depending on deep features, the suggested approach employs logits produced from cross entropy (CE) losses with infinite augmentations. A theoretical analysis shows that logits are effective in estimating distances defined on original features, and the proposed approach is thoroughly analyzed to provide a better understanding of why logits are beneficial for DG. Moreover, to increase the diversity of the source domain, a sampling-based method called smooth is introduced to obtain semantic directions from interclass relations. The effectiveness of the proposed approach is demonstrated through extensive experiments on widely used DG, object detection, and remote sensing datasets, where it achieves significant improvements over existing state-of-the-art methods across various backbone networks.

As data collection becomes increasingly facile and descriptions of data grow more diverse, exploring heterogeneous multiview data is becoming essential. Extracting valuable insights from vast multiview datasets is profoundly meaningful which can leverage the diversity of multiple features to improve classification accuracy. As is well-known, semi-supervised learning (SSL) utilizes limited set of labeled samples to train models when addressing label scarcity. However, although the existing multiview semi-supervised algorithms can accomplish classification task, they often struggle with high complexity problem and lack interpretability, more transparent, and low-complexity approaches are worth studying. Besides, the interplay between graph structure and multiview consistency makes a deeper understanding of underlying data patterns but challenges persist in optimizing graph and ensuring scalability. In this article, we propose a fast multiview semi-supervised algorithm based on anchor graph (BGFMS), which improves the classification performance. It could significantly reduce the computational complexity by converting the label prediction of the original data into the forecast for few anchor points and avoids the additional processing procedure. Extensive experimental results on synthetic dataset and different real datasets validate the effectiveness and efficiency of our algorithm.

A challenging open problem in deep learning is the representation of tabular data. Unlike the popular domains such as image and text understanding, where the deep convolutional network is fashionable in many applications, there is still no widely used neural architecture that can effectively explore informative structure from tabular data. In addition, existing antoencoder-based nonlinear representation learning approaches that employ reconstruction loss, are incompetent to preserve discriminative information. As a step toward bridging these gaps, we propose a novel adaptive graph convolutional network (AdaGCN) for unsupervised generalizable tabular representation learning in this article. To be specific, we hypothesize that the keys to boosting the efficiency and practicality of learned representations lie in three aspects, i.e., adaptivity, unsupervised, and generalization. As a result, the adaptive graph learning module is first designed to remove the predefined rules in conventional GCN models, which can explore more local patterns on arbitrary tabular data. Moreover, our AdaGCN directly minimizes the difference between distributions of original tabular data and learned embeddings for training without any label information. Last but not least, the parametric property of AdaGCN makes the unseen data to be handled offline, which extremely expends the scope of applications. We present extensive experiments showing that AdaGCN significantly and consistently outperforms several representation learning and clustering methods on several real-world tabular datasets.

Graph-based multiview clustering methods have gained significant attention in recent years. In particular, incorporating graph filtering into these methods allows for the exploration and utilization of both feature and topological information, resulting in a commendable improvement in clustering accuracy. However, these methods still exhibit several limitations: 1) the graph filters are predetermined, which disconnects the link with subsequent clustering tasks and 2) the separability of the filtered features is poor, which may not be suitable for the clustering. To mitigate these aforementioned issues, we propose Multiview Clustering via Block Diagonal Graph Filtering (MvC-BDGF), which can learn cluster-friendly graph filters. Specifically, the block diagonal graph filter with localized characteristics, which could make the filtered features very discriminating, is innovatively designed. The MvC-BDGF model seamlessly integrates the learning of graph filters with the acquisition of consensus graphs, forming a unified framework. This integration allows the model to obtain optimal filters and simultaneously acquire corresponding clustering labels. To solve the optimization problem in the MvC-BDGF model, an iterative solver based on the coordinate descent method is devised. Finally, a large number of experiments on benchmark datasets fully demonstrate the effectiveness and superiority of the proposed model. The code is available at https://github.com/haonanxin/MvC-BDGF_code.

Due to its high computational complexity, graph-based methods have limited applicability in large-scale multiview clustering tasks. To address this issue, many accelerated algorithms, especially anchor graph-based methods and indicator learning-based methods, have been developed and made a great success. Nevertheless, since the restrictions of the optimization strategy, these accelerated methods still need to approximate the discrete graph-cutting problem to a continuous spectral embedding problem and utilize different discretization strategies to obtain discrete sample categories. To avoid the loss of effectiveness and efficiency caused by the approximation and discretization, we establish a discrete fast multiview anchor graph clustering (FMAGC) model that first constructs an anchor graph of each view and then generates a discrete cluster indicator matrix by solving the discrete multiview graph-cutting problem directly. Since the gradient descent-based method makes it hard to solve this discrete model, we propose a fast coordinate descent-based optimization strategy with linear complexity to solve it without approximating it as a continuous one. Extensive experiments on widely used normal and large-scale multiview datasets show that FMAGC can improve clustering effectiveness and efficiency compared to other state-of-the-art baselines.

The affinity graph is regarded as a mathematical representation of the local manifold structure. The performance of locality-preserving projections (LPPs) and its variants is tied to the quality of the affinity graph. However, there are two drawbacks in current approaches. First, the pre-designed graph is inconsistent with the actual distribution of data. Second, the linear projection way would cause damage to the nonlinear manifold structure. In this article, we propose a nonlinear dimensionality reduction model, named deep locality-preserving projections (DLPPs), to solve these problems simultaneously. The model consists of two loss functions, each employing deep autoencoders (AEs) to extract discriminative features. In the first loss function, the affinity relationships among samples in the intermediate layer are determined adaptively according to the distances between samples. Since the features of samples are obtained by nonlinear mapping, the manifold structure can be kept in the low-dimensional space. Additionally, the learned affinity graph is able to avoid the influence of noisy and redundant features. In the second loss function, the affinity relationships among samples in the last layer (also called the reconstruction layer) are learned. This strategy enables denoised samples to have a good manifold structure. By integrating these two functions, our proposed model minimizes the mismatch of the manifold structure between samples in the denoising space and the low-dimensional space, while reducing sensitivity to the initial weights of the graph. Extensive experiments on toy and benchmark datasets have been conducted to verify the effectiveness of our proposed model.

Support Vector Machine (SVM) stands out as a prominent machine learning technique widely applied in practical pattern recognition tasks. It achieves binary classification by maximizing the "margin", which represents the minimum distance between instances and the decision boundary. Although many efforts have been dedicated to expanding SVM for multi-class case through strategies such as one versus one and one versus the rest, satisfactory solutions remain to be developed. In this paper, we propose a novel method for multi-class SVM that incorporates pairwise class loss considerations and maximizes the minimum margin. Adhering to this concept, we embrace a new formulation that imparts heightened flexibility to multi-class SVM. Furthermore, the correlations between the proposed method and multiple forms of multi-class SVM are analyzed. The proposed regularizer, akin to the concept of "margin", can serve as a seamless enhancement over the softmax in deep learning, providing guidance for network parameter learning. Empirical evaluations demonstrate the effectiveness and superiority of our proposed method over existing multi-classification methods. Complete version is available at https://arxiv.org/pdf/2312.06578.pdf. Code is available at https://github.com/zz-haooo/M3SVM.

Most of existing semi-supervised learning methods heavily depend on labeled samples and always indistinguishably regard all unlabeled instances. However, unreliable samples lying around the boundary lines severely disturb training models. To address the problems, we pose a Class-credible Pseudo-label Learning (CPL) framework for semi-supervised data analysis. CPL is a classification model by optimizing the pseudo-label matrix which is probabilistic and exponentially controlled by the coefficient γ. By virtue of it, the model can identify the poor samples who have indistinguishable classes and enhance the solid samples with high class-credibility. That means class-indeterminate samples could be weakened, while class-definite ones could be enhanced. We then present a concise discriminative feature selection model with ℓ2,p-norm (p ∈ (0, 1)) regularization. Extensive experiments on several datasets demonstrate the superior performance of proposed method against representative competitors.

In the realm of high-dimensional data analysis, the existence of outliers presents a substantial hurdle to the efficacy of feature selection methods that rely on the assumption of Gaussian distribution. To tackle this issue, we propose an outlier-robust feature selection method, ORFS, which combines robust ℓ2,1-norm minimization with group row-sparsity induced constrains to achieve both robustness and discriminative prediction capabilities. Moreover, the group row-sparsity constraints subspace learning based on ℓ2,0-norm can directly select features without parameter tuning. Finally, we introduce an iterative optimization strategy to solve NP-hard problem, and extensive experiments demonstrate the efficacy of ORFS in effectively eliminating the impact of outliers and significantly improving classification performance.

A significant challenge in multi-view clustering lies in the comprehensive extraction of consistency and complementary information from heterogeneous multi-view data. Numerous methods employ contrastive learning techniques to explore the information between views. However, the basic contrastive learning strategy does not consider cluster information when constructing sample pairs, potentially leading to the emergence of false negative pairs (FNPs). To tackle this concern, we propose a Multi-view Subspace Clustering with Consensus Graph Contrastive Learning (CGCL) model. Specifically, a self-representation layer is designed to acquire a consensus graph that elucidates the overall data distribution. Furthermore, a contrastive learning layer utilizes the cluster information embedded in the consensus graph to yield reliable sample pairs, resulting in a reduction of the detrimental FNPs and the extraction of complementary information from the various views. Extensive experiments on public datasets demonstrate the effectiveness of CGCL.

Self-supervised heterogeneous graph learning has achieved promising results in various real applications, but it still suffers from the following issues: (i) meta-paths can be employed to capture the homophily in the heterogeneous graph, but meta-paths are human-defined, requiring substantial expert knowledge and computational costs; and (ii) the heterogeneity in the heterogeneous graph is usually underutilized, leading to the loss of task-related information. To solve these issues, this paper proposes to capture both homophily and heterogeneity in the heterogeneous graph without pre-defined meta-paths. Specifically, we propose to learn a self-expressive matrix to capture the homophily from the subspace and nearby neighbors. Meanwhile, we propose to capture the heterogeneity by aggregating the information of nodes from different types. We further design a consistency loss and a specificity loss, respectively, to extract the consistent information between homophily and heterogeneity and to preserve their specific task-related information. We theoretically analyze that the learned homophilous representations exhibit the grouping effect to capture the homophily, and considering both homophily and heterogeneity introduces more task-related information. Extensive experimental results verify the superiority of the proposed method on different downstream tasks.

Graph Neural Networks (GNNs) have emerged as the predominant approach for analyzing graph data on the web and beyond. Contrastive learning (CL), a self-supervised paradigm, not only mitigates reliance on annotations but also has potential in performance. The hard negative sampling strategy that benefits CL in other domains proves ineffective in the context of Graph Contrastive Learning (GCL) due to the message passing mechanism. Embracing the subspace hypothesis in clustering, we propose a method towards expansive and adaptive hard negative mining, referred to as G raph contR astive leA rning via subsP ace prE serving (GRAPE ). Beyond homophily, we argue that false negatives are prevalent over an expansive range and exploring them confers benefits upon GCL. Diverging from existing neighbor-based methods, our method seeks to mine long-range hard negatives throughout subspace, where message passing is conceived as interactions between subspaces. %Empirical investigations back up this strategy. Additionally, our method adaptively scales the hard negatives set through subspace preservation during training. In practice, we develop two schemes to enhance GCL that are pluggable into existing GCL frameworks. The underlying mechanisms are analyzed and the connections to related methods are investigated. Comprehensive experiments demonstrate that our method outperforms across diverse graph datasets and remains competitive across varied application scenarios\footnoteOur code is available at https://github.com/zz-haooo/WWW24-GRAPE. .

Multi-view learning has raised more and more attention in recent years. However, traditional approaches only focus on the difference while ignoring the consistency among views. It may make some views, with the situation of data abnormality or noise, ineffective in the progress of view learning. Besides, the current datasets have become high-dimensional and large-scale gradually. Therefore, this paper proposes a novel multi-view compressed subspace learning method via low-rank tensor constraint, which incorporates the clustering progress and multi-view learning into a unified framework. First, for each view, we take the partial samples to build a small-size dictionary, which can reduce the effect of both redundancy information and computation cost greatly. Then, to find the consistency and difference among views, we impose a low-rank tensor constraint on these representations and further design an auto-weighted mechanism to learn the optimal representation. Last, due to the non-square of the learned representation, the bipartite graph has been introduced, and under the structured constraint, the clustering results can be obtained directly from this graph without any post-processing. Extensive experiments on synthetic and real-world benchmark datasets demonstrate the efficacy and efficiency of our method, especially for the views with noise or outliers.

Principal Component Analysis (PCA) aims to acquire the principal component space containing the essential structure of data, instead of being used for mining and extracting the essential structure of data. In other words, the principal component space contains not only information related to the essential structure of data but also some unrelated information. This frequently occurs when the intrinsic dimensionality of data is unknown or when it has complex distribution characteristics such as multi-modalities, manifolds, etc. Therefore, it is unreasonable to identify noise and useful information based solely on reconstruction error. For this reason, PCA is unsuitable as a preprocessing technique for most applications, especially in noisy environment. To solve this problem, this paper proposes robust PCA based on fuzzy local information reservation (FLIPCA). By analyzing the impact of reconstruction error on sample discriminability, FLIPCA provides a theoretical basis for noise identification and processing. This not only greatly improves its robustness but also extends its applicability and effectiveness as a data preprocessing technique. Meanwhile, FLIPCA maintains consistent mathematical descriptions with traditional PCA while having few adjustable hyperparameters and low algorithmic complexity. Finally, we conducted comprehensive experiments on synthetic and real-world datasets, which substantiated the superiority of our proposed algorithm.

Spectral clustering has been attracting increasing attention due to its well-defined framework and excellent performance. However, most traditional spectral clustering methods consist of two separate steps: 1) Solving a relaxed optimization problem to learn the continuous clustering labels, and 2) Rounding the continuous clustering labels into discrete ones. The clustering results of the relax-and-discretize strategy inevitably result in information loss and unsatisfactory clustering performance. Moreover, the similarity matrix constructed from original data may not be optimal for clustering since data usually have noise and redundancy. To address these problems, we propose a novel and effective algorithm to directly optimize the original spectral clustering model, called Direct Spectral Clustering (DSC). We theoretically prove that the original spectral clustering model can be solved by simultaneously learning a weighted discrete indicator matrix and a structured similarity matrix whose connected components are equal to the number of clusters. Both of them can be used to directly obtain the final clustering results without any post-processing. Further, an effective iterative optimization algorithm is exploited to solve the proposed method. Extensive experiments performed on synthetic and real-world datasets demonstrate the superiority and effectiveness of the proposed method compared to the state-of-the-art algorithms.

Normalized-Cut (N-Cut) is a famous model of spectral clustering. The traditional N-Cut solvers are two-stage: 1) calculating the continuous spectral embedding of normalized Laplacian matrix; 2) discretization via$K$-means or spectral rotation. However, this paradigm brings two vital problems: 1) two-stage methods solve a relaxed version of the original problem, so they cannot obtain good solutions for the original N-Cut problem; 2) solving the relaxed problem requires eigenvalue decomposition, which has${\mathcal {O}}(n^{3})$time complexity ($n$is the number of nodes). To address the problems, we propose a novel N-Cut solver designed based on the famous coordinate descent method. Since the vanilla coordinate descent method also has${\mathcal {O}}(n^{3})$time complexity, we design various accelerating strategies to reduce the time complexity to${\mathcal {O}}(|E|)$($|E|$is the number of edges). To avoid reliance on random initialization which brings uncertainties to clustering, we propose an efficient initialization method that gives deterministic outputs. Extensive experiments on several benchmark datasets demonstrate that the proposed solver can obtain larger objective values of N-Cut, meanwhile achieving better clustering performance compared to traditional solvers.

Clustering aims to partition a set of objects into different groups through the internal nature of these objects. Most existing methods face intractable hyper-parameter problems triggered by various regularization terms, which degenerates the applicability of models. Moreover, traditional graph clustering methods always encounter the expensive time overhead. To this end, we propose a Fast Clustering model with Anchor Guidance (FCAG). The proposed model not only avoids trivial solutions without extra regularization terms, but is also suitable to deal with large-scale problems by utilizing the prior knowledge of the bipartite graph. Moreover, the proposed FCAG can cope with out-of-sample extension problems. Three optimization methods Projected Gradient Descent (PGD) method, Iteratively Re-Weighted (IRW) algorithm and Coordinate Descent (CD) algorithm are proposed to solve FCAG. Extensive experiments verify the superiority of the optimization method CD. Besides, compared with other bipartite graph models, FCAG has the better performance with the less time cost. In addition, we prove through theory and experiment that when the learning rate of PGD tends to infinite, PGD is equivalent to IRW.

Clustering is a fundamental topic in machine learning and various methods are proposed, in which K-Means (KM) and min cut clustering are typical ones. However, they may produce empty or skewed clustering results, which are not as expected. In KM, the constrained clustering methods have been fully studied while in min cut clustering, it still needs to be developed. In this paper, we propose a parameter-insensitive min cut clustering with flexible size constraints. Specifically, we add lower limitations on the number of samples for each cluster, which can perfectly avoid the trivial solution in min cut clustering. As far as we are concerned, this is the first attempt of directly incorporating size constraints into min cut. However, it is a NP-hard problem and difficult to solve. Thus, the upper limits is also added in but it is still difficult to solve. Therefore, an additional variable that is equivalent to label matrix is introduced in and the augmented Lagrangian multiplier (ALM) is used to decouple the constraints. In the experiments, we find that the our algorithm is less sensitive to lower bound and is practical in image segmentation. A large number of experiments demonstrate the effectiveness of our proposed algorithm.

We propose a novel discriminative feature learning method via Max-Min Ratio Analysis (MMRA) for exclusively dealing with the long-standing "worst-case class separation" problem. Existing technologies simply consider maximizing the minimal pairwise distance on all class pairs in the low-dimensional subspace, which is unable to separate overlapped classes entirely especially when the distribution of samples within same class is diverging. We propose a new criterion, i.e., Max-Min Ratio Analysis (MMRA) that focuses on maximizing the minimal ratio value of between-class and within-class scatter to extremely enlarge the separability on the overlapped pairwise classes. Furthermore, we develop two novel discriminative feature learning models for dimensionality reduction and metric learning based on our MMRA criterion. However, solving such a non-smooth non-convex max-min ratio problem is challenging. As an important theoretical contribution in this paper, we systematically derive an alternative iterative algorithm based on a general max-min ratio optimization framework to solve a general max-min ratio problem with rigorous proofs of convergence. More importantly, we also present another solver based on bisection search strategy to solve the SDP problem efficiently. To evaluate the effectiveness of proposed methods, we conduct extensive pattern classification and image retrieval experiments on several artificial datasets and real-world ScRNA-seq datasets, and experimental results demonstrate the effectiveness of proposed methods.

Exploiting consistent structure from multiple graphs is vital for multi-view graph clustering. To achieve this goal, we propose an Efficient Balanced Multi-view Graph Clustering via Good Neighbor Fusion (EBMGC-GNF) model which comprehensively extracts credible consistent neighbor information from multiple views by designing a Cross-view Good Neighbors Voting module. Moreover, a novel balanced regularization term based on p-power function is introduced to adjust the balance property of clusters, which helps the model adapt to data with different distributions. To solve the optimization problem of EBMGC-GNF, we transform EBMGC-GNF into an efficient form with graph coarsening method and optimize it based on accelareted coordinate descent algorithm. In experiments, extensive results demonstrate that, in the majority of scenarios, our proposals outperform state-of-the-art methods in terms of both effectiveness and efficiency.

New Intent Discovery (NID) is the task of categorizing new and unknown intents into distinct clusters. Recent advances in this issue can be roughly grouped into parametric clustering and representation learning. Although both of them have demonstrated exceptional performance, the efficacy of combining these two techniques remains unexplored. To this end, we propose a new framework named INS-NID (INteractiveSupervision forNewIntentDiscovery). This framework is designed to build a bridge between parametric clustering and representation learning. In practice, INS-NID comprises a parametric clustering branch and a representation learning branch, which work collaboratively and provide interactive supervision to boost each other. Specifically, the representation learning branch provides reliable distribution estimation during training, which is used to regularize the cluster assignments of the parametric clustering. On the other hand, the cluster assignments predicted by parametric clustering provide additional supervision information for the representation learning branch. The effectiveness of INS is highlighted with superior performance over several state-of-the-art methods across various benchmarks. For example, INS achieves 90% clustering accuracy on the Banking dataset, surpassing the best competitor by 5.96%.

Classification is a fundamental task in the field of data mining. Unfortunately, high-dimensional data often degrade the performance of classification. To solve this problem, dimensionality reduction is usually adopted as an essential preprocessing technique, which can be divided into feature extraction and feature selection. Due to the ability to obtain category discrimination, linear discriminant analysis (LDA) is recognized as a classic feature extraction method for classification. Compared with feature extraction, feature selection has plenty of advantages in many applications. If we can integrate the discrimination of LDA and the advantages of feature selection, it is bound to play an important role in the classification of high-dimensional data. Motivated by the idea, we propose a supervised feature selection method for classification. It combines trace ratio LDA with$\ell _{2,p}$-norm regularization and imposes the orthogonal constraint on the projection matrix. The learned row-sparse projection matrix can be used to select discriminative features. Then, we present an optimization algorithm to solve the proposed method. Finally, the extensive experiments on both synthetic and real-world datasets indicate the effectiveness of the proposed method.

Dimensionality reduction (DR) targets to learn low-dimensional representations for improving discriminability of data, which is essential for many downstream machine learning tasks, such as image classification, information clustering, etc. Non-Gaussian issue as a long-standing challenge brings many obstacles to the applications of DR methods that established on Gaussian assumption. The mainstream way to address above issue is to explore the local structure of data via graph learning technique, the methods based on which however suffer from a common weakness, that is, exploring locality through pairwise points causes the optimal graph and subspace are difficult to be found, degrades the performance of downstream tasks, and also increases the computation complexity. In this article, we first propose a novel self-evolution bipartite graph (SEBG) that uses anchor points as the landmark of subclasses, and learns anchor-based rather than pairwise relationships for improving the efficiency of locality exploration. In addition, we develop an efficient local coherent structure learning (ELCS) algorithm based on SEBG, which possesses the ability of updating the edges of graph in learned subspace automatically. Finally, we also provide a multivariable iterative optimization algorithm to solve proposed problem with strict theoretical proofs. Extensive experiments have verified the superiorities of the proposed method compared to related SOTA methods in terms of performance and efficiency on several real-world benchmarks and large-scale image datasets with deep features.

Bipartite spectral graph partitioning (BSGP) method as a co-clustering method, has been widely used in document clustering, which simultaneously clusters documents and words by making full use of the duality between documents and words. It consists of two steps: 1) graph construction and 2) singular value decomposition on the bipartite graph to compute a continuous cluster assignment matrix, followed by post-processing to get the discrete solution. However, the generated graph is unstructured and fixed. It heavily relies on the quality of the graph construction. Moreover, the two-stage process may deviate from directly solving the primal problem. In order to tackle these defects, a novel bipartite graph partitioning method is proposed to learn a bipartite graph with exactly c connected components (c is the number of clusters), which can obtain clustering results directly. Furthermore, it is experimentally and theoretically proved that the solution of the proposed model is the discrete solution of the primal BSGP problem for a special situation. Experimental results on synthetic and real-world datasets exhibit the superiority of the proposed method.

Fuzzy theory, which extends precise binary logic to continuous fuzzy logic, provides an effective tool for uncertainty problems in machine learning and thus, evolved fuzzy methods such as fuzzy c-means and fuzzy graph clustering. Among them, graph-based clustering methods have become a hot spot in the field of unsupervised clustering due to their good ability to process nonlinear data. Unfortunately, they usually suffer from high time complexity and cumbersome regularization parameter tuning, so their practical applications are greatly limited. To this end, we propose an efficient graph-cut algorithm called fS2F. Based on the similarity graph between the dataset and landmark subset, fS2F transforms the membership learning of the entire dataset into a clustering problem of representative points, which greatly improves its clustering efficiency. In addition, fS2F softly constrains the cluster size in a way that does not require additional regularization parameters so that it can be widely and conveniently applied. The article also presents the optimization method for this model and demonstrates its effectiveness through experiments.

Feature selection serves as a fundamental technique in machine learning and data analysis, playing a crucial role in extracting valuable features from large-scale and high-dimensional datasets that may contain irrelevant features. To enhance the performance of feature selection, regularizers like${\ell _{1}}$-norm or${\ell _{2,1}}$-norm are commonly utilized to encourage sparsity. Nonetheless, these traditional regularization techniques encounter certain challenges. When correlations exist among features, the sparsity-driven regularization can unfairly diminish weights of correlated features to zero, thus ignoring the feature correlations and lacking group sparsity properties. While a straightforward combination of${\ell _{1}}$-norm and${\ell _{2}}$-norm can uncover feature correlations, it lacks adaptability and effectively balancing sparsity and correlation. To address these challenges, we introduce a novel matrix-based regularization term, called a fuzzy elastic net, in the unsupervised feature selection model. Our model is founded on principal component analysis, a well-established dimensionality reduction technique adept at finding subspaces that retain most information from raw data. The model is enhanced by a fuzzy elastic net, which promotes group or sparsity properties through adaptive parameter tuning. The new regularization term introduces a flexible fuzzy weighted scheme combining the${\ell _{2,2}}$-norm and${\ell _{2,p}}$-norm ($0< p\leq 1$). This approach allows adaptive adjustment based on data characteristics, offering a tunable balance between selecting discriminative features and identifying correlated ones. Consequently, this regularization term equips the model to handle diverse data analysis tasks flexibly, thereby enhancing adaptability and generalization performance. Furthermore, we propose an efficient optimization strategy to solve this model. Extensive experiments conducted on UCI datasets and real-world datasets demonstrate the effectiveness and efficiency of our proposed method.

Recently, anchor-based strategy is getting a lot of attention, which extends spectral clustering to reveal the dual relation between samples and features. However, the acquisition of clustering results follows the relaxed-discrete procedure, which might lead to serious information loss. Given that, local-global fuzzy clustering with anchor graph is proposed in this article, which jointly completes subspace learning and clustering. With the construction of anchor graph, we first impose the fuzzy constraint on the indicator matrix, which is beneficial for revealing the ambiguity and uncertainty in clustering tasks. Thereafter, we introduce the graph divergence regularization term for maximizing the variance of fuzzy indicator matrix, which not only avoids the trivial solutions in local graph learning, but also enhances the separability of data for explicit global structure. In this way, the local graph loss and global divergence regularization term are able to jointly capture the critical clustering structure. Finally, we can obtain the clustering results in accordance with the optimal fuzzy indicator matrix, which is updated alternately by the presented coordinate descent method in optimization process. Therefore, the desirable discrete labels come out automatically under the fuzzy strategy without extra discretization operations, which conforms to reality. The effectiveness of our method will be demonstrated through comprehensive experimental results.

The unsupervised nature of clustering has attracted significant interest. In particular, researchers delve into exploring the superiority of fuzzy clustering in flexibly handling computations involving uncertain data. However, outliers can present considerable challenges by distorting the measurement of similarity between samples, and biases in projection subspace learning may impede accurate partitioning. In this article, we propose a robust discriminant embedding projection fuzzy clustering with optimal mean (RPFCOM) method. First, the weighted loss function term distinguishes outliers and normal samples through boolean weight, thereby inducing row sparsity in the learning of projection subspace. The distribution of boolean weight penalizes outliers with large errors in the projection subspace. Second, we incorporate minimizing projection reconstruction information learning while suppressing redundant features, where the optimal mean dynamically corrects the projection learning bias. And the embedding of discriminative information further strengthens the capability of differentiating normal samples. Finally, the proposed method adaptively updates the boolean weight to identify outliers, which joints fuzzy membership matrix constructed from the maximum entropy graphs, enhancing the stability in distinguishing normal sample clusters. Comprehensive experimental validation on noise contaminated dataset has demonstrated the superiority of RPFCOM.

With the development of information technology, a large number of multiview data has emerged, which makes multiview clustering algorithms considerably attractive. Previous graph-based multiview clustering methods usually contain two steps: obtaining the fusion graph or spectral embedding of all views; and performing clustering algorithms. The two-step process cannot obtain optimal results since the two steps cannot negotiate with each other. To address this drawback, a novel algorithm named as multi-view fuzzy clustering based on anchor graph is presented. The proposed method can simultaneously obtain the membership matrix and minimize the disagreement rates of different views. A novel regularization based on trace norm is also presented in this article, which can not only obtain a clear clustering partition to prevent that all samples belonging to each cluster with the same membership value$\frac{1}{c}$, but also balance the size of each cluster. Moreover, we exploit the reweighted method to optimize the proposed model, which can introduce an adaptive weight to each view to deal with the unreliable views. A series of experiments are conducted on different datasets, and the clustering performance verifies the effectiveness and efficiency of the proposed algorithm.

Deep fuzzy clustering employs neural networks to discover the low-dimensional embedding space of data, providing an effective solution to the clustering problem posed by high-dimensional data. Although some algorithms have achieved good results in application, the field still faces the following problems: the lack of clustering loss function limits the development of deep clustering, and most of them use the self-training strategy-based Kullback-Leibler (KL) divergence; some algorithms directly use conventional constrained clustering objective function as the loss function in deep models, and update network parameters alternately, the optimization process is cumbersome. Focusing on the issues mentioned above, this article first proposed an unconstrained fuzzy$c$-means algorithm that can be solved using gradient descent and then used it as the clustering loss function to obtain a novel deep fuzzy clustering model named unsupervised deep embedding for fuzzy clustering. The proposed model simultaneously learns the low-dimensional representation of data and performs fuzzy clustering. It updates parameters through gradient descent and backpropagation, achieving end-to-end optimization. The proposed algorithm's effectiveness and competitiveness are fully demonstrated through extensive experiments conducted on image and text datasets.

Principal component analysis (PCA) is one of the most versatile techniques for unsupervised dimension reduction, which is implemented as a fundamental preprocessing method in multiple tasks of statistics and machine learning research because of its efficiency. Nevertheless, researchers have concentrated on the identification of outliers that do not conform to the low-dimensional approximation through statistical methods, e.g., outlier rejection, without giving insights on each data point with a dynamic ratio of signal-to-noise components in the high-dimensional regimes. To characterize the dynamic nature of the principal component information, we propose a Normalized Robust PCA with Adaptive Reconstruction Error minimization model, which considers both the adaptive normalization technique and flexible weights learning simultaneously. With this configuration, the principal component information constantly adjusts the degree of sparsity for activated samples. In other words, the signal component's discrimination and noise information restriction could work cooperatively. Empirical studies on one synthetic dataset and several benchmarks demonstrate the effectiveness of our proposed method over existing outlier rejection methods.

Ensemble clustering has emerged as a powerful framework for analyzing heterogeneous and complex data. Despite the abundance of existing schemes, co-association matrix-based methods remain the mainstream approach. However, focusing solely on pairwise correlations falls short of fully capturing the intricate cluster relationships. Moreover, despite its potential, ensemble clustering has yet to effectively leverage the powerful representation capabilities of neural networks. To address these limitations, we propose a deep ensemble clustering method called Ensemble Clustering with Attentional Representation (ECAR). Our method considers the results of base partition as groups with related information to explore higher-order fusion information. ECAR captures the importance of each sample's association with its related group by employing an attentional network, and encodes this information into a low-dimensional representation. The attentional network is trained by jointly optimizing the clustering loss from soft assignments learned from the embeddings and the reconstruction loss from the weighted graph generated from ensemble clustering. During training, the weights of base partitions are adaptively refined to promote diversity and consistency while reducing the impact of low-quality and redundant base partitions. Extensive experimental results on real-world datasets demonstrate the substantial improvement of our method over existing baseline ensemble clustering methods and deep clustering methods.

Fuzzy$k$-Means with Entropy Regularization method (ERFKM) is an extension to Fuzzy$k$-Means (FKM) by introducing a maximum entropy term to FKM, whose purpose is trading off fuzziness and compactness. However, ERFKM often converges to a poor local minimum, which affects its performance. In this paper, we propose an effective optimization method to solve this problem, called IRW-ERFKM. First a new equivalent problem for ERFKM is proposed; then we solve it through Iteratively Re-Weighted (IRW) method. Since IRW-ERFKM optimizes the problem with$k\times 1$instead of$d\times k$intermediate variables, the space complexity of IRW-ERFKM is greatly reduced. Extensive experiments on clustering performance and objective function value show IRW-ERFKM can get a better local minimum than ERFKM with fewer iterations. Through time complexity analysis, it verifies IRW-ERFKM and ERFKM have the same linear time complexity. Moreover, IRW-ERFKM has advantages on evaluation metrics compared with other methods. What's more, there are two interesting findings. One is when we use IRW method to solve the equivalent problem of ERFKM with one factor$\mathbf{U}$, it is equivalent to ERFKM. The other is when the inner loop of IRW-ERFKM is executed only once, IRW-ERFKM and ERFKM are equivalent in this case.

The key challenge of multi-view graph-based clustering is to mine consistent clustering structures from multiple graphs. Existing works seek clustering decisions from either multiple spectral embeddings or multiple affinity matrices, ignoring the interactions among them. To address this problem, we propose a Bidirectional Attentive Multi-view Clustering (BAMC) model to explore a consensus space w.r.t.spectral embedding and affinity matrix simultaneously, where they can promote each other to mine richer structural information from multiple graphs. BAMC is composed of a Spectral Embedding Learning (SEL) module, an Affinity Matrix Learning (AML) module, and a Bidirectional Attentive Clustering (BAC) module. SEL seeks consensus spectral embeddings by aligning the distributions of elements sampled from subspaces spanned by multiple spectral embeddings. AML learns a consensus affinity matrix from input affinity matrices. BAC guarantees consistency between the learned consensus spectral embeddings and the affinity matrix. To balance their effects, it also assigns adaptive weights to SEL and AML's objective functions. To solve the optimization problem involved in BAMC, we propose an efficient algorithm based on the Majority-Minimization framework with an ingenious surrogate problem. Extensive experiments on several synthetic and real-world datasets demonstrate the superb performance of BAMC.

Multimodal aspect-level sentiment analysis has attracted increasing attention in recent years. However, existing methods have two unaddressed limitations: (1) due to the lack of labelled pre-training data of dedicated sentiment analysis, the methods with a pre-training manner produce suboptimal prediction results; (2) most existing methods employ a self-attention encoder to fuse multimodal tokens, which not only ignores the alignment relationship between different modal tokens but also makes the model unable to capture the semantic links between images and texts. In this paper, we propose a momentum contrastive learning network (MOCOLNet) to overcome above limitations. First, we merge the pre-training stage with the training stage to design an end-to-end training manner which uses less labelled data dedicated to sentiment analysis to obtain better prediction results. Second, we propose a multimodal contrastive learning method to align the different modal representations before data fusing, and design a cross-modal matching strategy to provide semantic interactive information between texts and images. Moreover, we introduce an auxiliary momentum strategy to increase the robustness of model. We also analyse the effectiveness of the proposed multimodal contrastive learning method using a mutual information theory. Experiments verify that the proposed MOCOLNet is superior to other strong baselines.

In this paper, we propose a novel algorithm to solve the row-sparse principal component analysis problem without relying on any data structure assumption. Sparse principal component analysis was proposed to improve the interpretability of principal component analysis by restricting the number of non-zero elements in each loading vector, but the varying sparsity patterns among different leading vectors may result in trouble on some occasions, such as feature selection. Then row-sparse principal component analysis was proposed, which demands the same sparse pattern among different loading vectors. However, the optimization of row-sparse principal component analysis problems is NP-hard. Although some algorithms were proposed to solve this problem, but they are only applicable to the specific data structure. In this paper, we transform the original row-sparse principal component analysis problem into a new equivalent problem that can be solved by coordinate descent method without relying on any data structure assumption. Then by carefully eliminating redundant structures to avoid repeating computation, we propose a more efficient coordinate descent method to solve this problem. Furthermore, no parameter needs to be tuned in our algorithm. Finally, extensive experiments are conducted on the real world data sets to demonstrate the superiority of our algorithm.

Feature selection is one of the important topics of machine learning, and it has a wide range of applications in data preprocessing. At present, feature selection based on$\ell _{2,1}$-norm regularization is a relatively mature method, but it is not enough to maximize the sparsity and parameter-tuning leads to increased costs. Later scholars found that the$\ell _{2,0}$-norm constraint is more conductive to feature selection, but it is difficult to solve and lacks convergence guarantees. To address these problems, we creatively propose a novel Outliers Robust Unsupervised Feature Selection for structured sparse subspace (ORUFS), which utilizes$\ell _{2,0}$-norm constraint to learn a structured sparse subspace and avoid tuning the regularization parameter. Moreover, by adding binary weights, outliers are directly eliminated and the robustness of model is improved. More importantly, a Re-Weighted (RW) algorithm is exploited to solve our$\ell _{p}$-norm problem. For the NP-hard problem of$\ell _{2,0}$-norm constraint, we develop an effective iterative optimization algorithm with strict convergence guarantees and closed-form solution. Subsequently, we provide theoretical analysis about convergence and computational complexity. Experimental results on real-world datasets illustrate that our method is superior to the state-of-the-art methods in clustering and anomaly detection tasks.

Most existing multi-view graph clustering models either seek consistent clustering results from similarity matrices and spectral embeddings respectively or follow direct bidirectional integration of them, which ignores the interaction between them. To make up for this flaw, this paper designs a novel multi-view clustering model that performsBidirectionalFusion withCross-viewGraphFilter (BF-CGF). To be specific, BF-CGF first learns a consistent graph embedding via performing the interaction between multi-view graphs and spectral embeddings with the perspective of the graph spectral domain and then considers seeking a consistent indicator matrix via the graph cut model from the consistent graph embedding and the similarity matrices. To solve the optimization problem of BF-CGF, we propose an efficient iterative algorithm and provide the corresponding convergence and complexity analyses. Extensive experimental results demonstrate that the proposed BF-CGF outperforms state-of-the-art competitors in most benchmark datasets.

Feature selection has achieved unprecedented success in obtaining sparse discriminative features. However, the existing methods almost use the$\ell _{2,p}$-norm constraint on transformation matrix to obtain sparse features, which introduces extra parameters and cannot obtain the features directly. In addition, existing algorithms only focused on the global structure and ignored the local structure, leading to poor performance when solving data with non-Gaussian distributions which a single center point cannot describe precisely. Based on above considerations, we propose a supervised feature selection via multi-center and local structure learning. We further introduce trace ratio criterion into our model in favor of improving the discriminant of features selected. In order to address the overlap problem, we use multiple center points to match the distribution of data and construct a$k$-Nearest Neighbor graph to explore the local structure of the data. In addition, we also propose an efficient method to optimize the transformation matrix with the$\ell _{2,0}$-norm constraint and can directly obtain the sparse features. We evaluate our method on Toy datasets and several real-world datasets, show improvement over state-of-the-art feature selection methods, and demonstrate the effectiveness of our model in dealing with non-Gaussian distributed data problems.

In recent years, graph-based multiview clustering methods have become a research hotspot in the clustering field. However, most existing methods lack consideration of cluster balance in their results. In fact, cluster balance is crucial in many real-world scenarios. Additionally, graph-based multiview clustering methods often suffer from high time consumption and cannot handle large-scale datasets. To address these issues, this paper proposes a novel graph-based multiview clustering method. The method is built upon the bipartite graph. Specifically, it employs a label propagation mechanism to update the smaller anchor label matrix rather than the sample label matrix, significantly reducing the computational cost. The introduced balance constraint in the proposed model contributes to achieving balanced clustering results. The entire clustering model combines information from multiple views through graph fusion. The joint graph and view weight parameters in the model are obtained through task-driven self-supervised learning. Moreover, the model can directly obtain clustering results without the need for the two-stage processing typically used in general spectral clustering. Finally, extensive experiments on toy datasets and real-world datasets are conducted to validate the superiority of the proposed method in terms of clustering performance, clustering balance, and time expenditure.

Principal component analysis (PCA) is one of the most successful unsupervised subspace learning methods and has been used in many practical applications. To deal with the outliers in real-world data, robust principal analysis models based on various measure are proposed. However, conventional PCA models can only transform features to unknown subspace for dimensionality reduction and cannot perform features’ selection task. In this article, we propose a novel robust PCA (RPCA) model to mitigate the impact of outliers and conduct feature selection, simultaneously. First, we adopt$\sigma $-norm as reconstruction error (RE), which plays an important role in robust reconstruction. Second, to conduct feature selection task, we apply$\ell _{2,0}$-norm constraint to subspace projection. Furthermore, an efficient iterative optimization algorithm is proposed to solve the objective function with nonconvex and nonsmooth constraint. Extensive experiments conducted on several real-world datasets demonstrate the effectiveness and superiority of the proposed feature selection model.

In the unsupervised feature selection method based on spectral analysis, constructing a similarity matrix is a very important part. In existing methods, the linear low-dimensional projection used in the process of constructing the similarity matrix is too hard, it is very challenging to construct a reliable similarity matrix. To this end, we propose a method to construct a flexible optimal graph. Based on this, we propose an unsupervised feature selection method named unsupervised feature selection with flexible optimal graph and$\ell _{2,1}$-norm regularization (FOG-R). Unlike other methods that use linear projection to approximate the low-dimensional manifold of the original data when constructing a similarity matrix, FOG-R can learn a flexible optimal graph, and by combining flexible optimal graph learning and feature selection into a unified framework to get an adaptive similarity matrix. In addition, an iterative algorithm with a strict convergence proof is proposed to solve FOG-R.$\ell _{2,1}$-norm regularization will introduce an additional regularization parameter, which will cause parameter-tuning trouble. Therefore, we propose another unsupervised feature selection method, that is, unsupervised feature selection with a flexible optimal graph and$\ell _{2,0}$-norm constraint (FOG-C), which can avoid tuning additional parameters and obtain a more sparse projection matrix. Most critically, we propose an effective iterative algorithm that can solve FOG-C globally with strict convergence proof. Comparative experiments conducted on 12 public datasets show that FOG-R and FOG-C perform better than the other nine state-of-the-art unsupervised feature selection algorithms.

Owing to its simplicity, the traditional k - means (Lloyd heuristic) clustering method plays a vital role in a variety of machine-learning applications. Disappointingly, the Lloyd heuristic is prone to local minima. In this article, we propose k - mRSR, which converts the sum-of-squared error (SSE) (Lloyd) into a combinatorial optimization problem and incorporates a relaxed trace maximization term and an improved spectral rotation term. The main advantage of k - mRSR is that it only needs to solve the membership matrix instead of computing the cluster centers in each iteration. Furthermore, we present a nonredundant coordinate descent method that brings the discrete solution infinitely close to the scaled partition matrix. Two novel findings from the experiments are that k - mRSR can further decrease (increase) the objective function values of the k - means obtained by Lloyd (CD), while Lloyd (CD) cannot decrease (increase) the objective function obtained by k - mRSR. In addition, the results of extensive experiments on 15 datasets indicate that k - mRSR outperforms both Lloyd and CD in terms of the objective function value and outperforms other state-of-the-art methods in terms of clustering performance.

In this article, we propose a novel unsupervised feature selection model combined with clustering, named double-structured sparsity guided flexible embedding learning (DSFEL) for unsupervised feature selection. DSFEL includes a module for learning a block-diagonal structural sparse graph that represents the clustering structure and another module for learning a completely row-sparse projection matrix using the$\ell_{2,0}$-norm constraint to select distinctive features. Compared with the commonly used$\ell_{2,1}$-norm regularization term, the$\ell_{2,0}$-norm constraint can avoid the drawbacks of sparsity limitation and parameter tuning. The optimization of the$\ell_{2,0}$-norm constraint problem, which is a nonconvex and nonsmooth problem, is a formidable challenge, and previous optimization algorithms have only been able to provide approximate solutions. In order to address this issue, this article proposes an efficient optimization strategy that yields a closed-form solution. Eventually, through comprehensive experimentation on nine real-world datasets, it is demonstrated that the proposed method outperforms existing state-of-the-art unsupervised feature selection methods.

Multiview clustering (MVC) aims to exploit heterogeneous information from different sources and was extensively investigated in the past decade. However, far less attention has been paid to handling large-scale multiview data. In this brief, we fill this gap and propose a fast multiview clustering by an optimal graph mining model to handle large-scale data. We mine a consistent clustering structure from landmark-based graphs of different views, from which the optimal graph based on the one-hot encoding of cluster labels is recovered. Our model is parameter-free, so intractable hyperparameter tuning is avoided. An efficient algorithm of linear complexity to the number of samples is developed to solve the optimization problems. Extensive experiments on real-world datasets of various scales demonstrate the superiority of our proposal.

Studying the relationship between linear discriminant analysis (LDA) and least squares regression (LSR) is of great theoretical and practical significance. It is well-known that the two-class LDA is equivalent to an LSR problem, and directly casting multiclass LDA as an LSR problem, however, becomes more challenging. Recent study reveals that the equivalence between multiclass LDA and LSR can be established based on a special class indicator matrix, but under a mild condition which may not hold under the scenarios with low-dimensional or oversampled data. In this article, we show that the equivalence between multiclass LDA and LSR can be established based on arbitrary linearly independent class indicator vectors and without any condition. In addition, we show that LDA is also equivalent to a constrained LSR based on the data-dependent indicator vectors. It can be concluded that under exactly the same mild condition, such two regressions are both equivalent to the null space LDA method. Illuminated by the equivalence of LDA and LSR, we propose a direct LDA classifier to replace the conventional framework of LDA plus extra classifier. Extensive experiments well validate the above theoretic analysis.

Graph-based semisupervised learning can explore the graph topology information behind the samples, becoming one of the most attractive research areas in machine learning in recent years. Nevertheless, existing graph-based methods also suffer from two shortcomings. On the one hand, the existing methods generate graphs in the original high-dimensional space, which are easily disturbed by noisy and redundancy features, resulting in low-quality constructed graphs that cannot accurately portray the relationships between data. On the other hand, most of the existing models are based on the Gaussian assumption, which cannot capture the local submanifold structure information of the data, thus reducing the discriminativeness of the learned low-dimensional representations. This article proposes a semisupervised subspace learning with adaptive pairwise graph embedding (APGE), which first builds a -nearest neighbor graph on the labeled data to learn local discriminant embeddings for exploring the intrinsic structure of the non-Gaussian labeled data, i.e., the submanifold structure. Then, a -nearest neighbor graph is constructed on all samples and mapped to GE learning to adaptively explore the global structure of all samples. Clustering unlabeled data and its corresponding labeled neighbors into the same submanifold, sharing the same label information, improves embedded data's discriminative ability. And the adaptive neighborhood learning method is used to learn the graph structure in the continuously optimized subspace to ensure that the optimal graph matrix and projection matrix are finally learned, which has strong robustness. Meanwhile, the rank constraint is added to the Laplacian matrix of the similarity matrix of all samples so that the connected components in the obtained similarity matrix are precisely equal to the number of classes in the sample, which makes the structure of the graph clearer and the relationship between the near-neighbor sample points more explicit. Finally, multiple experiments on several synthetic and real-world datasets show that the method performs well in exploring local structure and classification tasks.

Embedded feature selection approach guides subsequent projection matrix (selection matrix) learning through the acquisition of pseudolabel matrix to conduct feature selection tasks. Yet the continuous pseudolabel matrix learned from relaxed problem based on spectral analysis deviates from reality to some extent. To cope with this issue, we design an efficient feature selection framework inspired by classical least-squares regression (LSR) and discriminative K-means (DisK-means), which is called the fast sparse discriminative K-means (FSDK) for the feature selection method. First, the weighted pseudolabel matrix with discrete trait is introduced to avoid trivial solution from unsupervised LSR. On this condition, any constraint imposed into pseudolabel matrix and selection matrix is dispensable, which is significantly beneficial to simplify the combinational optimization problem. Second, the$\ell_{2,p}$-norm regularizer is introduced to satisfy the row sparsity of selection matrix with flexible$p$. Consequently, the proposed FSDK model can be treated as a novel feature selection framework integrated from the DisK-means algorithm and$\ell_{2,p}$-norm regularizer to optimize the sparse regression problem. Moreover, our model is linearly correlated with the number of samples, which is speedy to handle the large-scale data. Comprehensive tests on various data terminally illuminate the effectiveness and efficiency of FSDK.

Spectral clustering (SC) has been widely used in many applications and shows excellent performance. Its high computational cost limits its applications; many strategies including the anchor technique can partly alleviate the high computational cost problem. However, early methods ignore the fact that SC usually involves two stages: relaxation and postprocessing, i.e., it relaxes the discrete constraints to continuous constraints, and then conducts the postprocessing to get the discrete solution, which is time-consuming and deviates from directly solving the primal problem. In this article, we first adopt the bipartite graph strategy to reduce the time complexity of SC, and then an improved coordinate descent (CD) method is proposed to solve the primal problem directly without singular value decomposition (SVD) and postprocessing, i.e., directly solving the primal problem not approximately solving. Experiments on various real-world benchmark datasets show that the proposed method can get better solutions faster with better clustering performance than traditional optimization methods. Furthermore, it can jump out of local minima of traditional methods and continue to obtain better local solutions. Moreover, compared with other clustering methods, it also shows its superiority.

Spectral clustering (SC) has been applied to analyze varieties of data structures over the past few decades owing to its outstanding breakthrough in graph learning. However, the time-consuming eigenvalue decomposition (EVD) and information loss during relaxation and discretization impact the efficiency and accuracy especially for large-scale data. To address above issues, this brief proposes a simple and fast method named efficient discrete clustering with anchor graph (EDCAG) to circumvent postprocessing by binary label optimization. First of all, sparse anchors are adopted to accelerate graph construction and obtain a parameter-free anchor similarity matrix. Subsequently, inspired by intraclass similarity maximization in SC, we design an intraclass similarity maximization model between anchor-sample layer to cope with anchor graph cut problem and exploit more explicit data structures. Meanwhile, a fast coordinate rising (CR) algorithm is employed to alternatively optimize discrete labels of samples and anchors in designed model. Experimental results show excellent rapidity and competitive clustering effect of EDCAG.

As one of the most popular supervised dimensionality reduction methods, linear discriminant analysis (LDA) has been widely studied in machine learning community and applied to many scientific applications. Traditional LDA minimizes the ratio of squared norms, which is vulnerable to the adversarial examples. In recent studies, many -norm-based robust dimensionality reduction methods are proposed to improve the robustness of model. However, due to the difficulty of -norm ratio optimization and weakness on defending a large number of adversarial examples, so far, scarce works have been proposed to utilize sparsity-inducing norms for LDA objective. In this article, we propose a novel robust discriminative projections learning (rDPL) method based on the -norm trace-ratio minimization optimization algorithm. Minimizing the -norm ratio problem directly is a much more challenging problem than the traditional methods, and there is no existing optimization algorithm to solve such nonsmooth terms ratio problem. We derive a new efficient algorithm to solve this challenging problem and provide a theoretical analysis on the convergence of our algorithm. The proposed algorithm is easy to implement and converges fast in practice. Extensive experiments on both synthetic data and several real benchmark datasets show the effectiveness of the proposed method on defending the adversarial patch attack by comparison with many state-of-the-art robust dimensionality reduction methods.

-norm is used as distance metric. Recently, many scholars have devoted themselves to solving this difficulty. They learn the projection matrix from minimum reconstruction error or maximum projection variance as the starting point, which leads them to ignore a serious problem, that is, the original PCA learns the projection matrix by minimizing the reconstruction error and maximizing the projection variance simultaneously, but they only consider one of them, which imposes various limitations on the performance of model. To solve this problem, we propose a novel robust principal component analysis via joint reconstruction and projection, namely, RPCA-RP, which combines reconstruction error and projection variance to fully mine the potential information of data. Furthermore, we carefully design a discrete weight for model to implicitly distinguish between normal data and outliers, so as to easily remove outliers and improve the robustness of method. In addition, we also unexpectedly discovered that our method has anomaly detection capabilities. Subsequently, an effective iterative algorithm is explored to solve this problem and perform related theoretical analysis. Extensive experimental results on several real-world datasets and RGB large-scale dataset demonstrate the superiority of our method.

The success of existing unsupervised feature selection (UFS) methods heavily relies on the assumption that the intrinsic relationships among original high-dimensional (HD) data samples exist in the discriminative low-dimension (LD) subspace. However, previous UFS methods commonly construct pairwise graphs and employ$\ell_{2,1}$-norm regularization to severally preserve the local structure and calculate the score of features, which is computationally complex and easy to get stuck into local optimum, so that those approaches cannot be applied in dealing with large-scale datasets in practice. To overcome this challenge, we propose a novel UFS method, in which a novel anchor graph embedding paradigm is designed to extract the local neighborhood relationships among data samples by reducing the computational complexity of graph construction to be linear in the number of data. Moreover, to improve the optimality of selected features as well as the performance of downstream tasks, we propose a discrete feature scoring mechanism, which imposes orthogonal$\ell_{2,0}$-norm constraints on learned projections, in order to enhance the distinction of feature scores as well as reduce the probability of falling into local optimum. In addition, solving the proposed nonconvex and nonsmooth NP-hard problem is challenging, and we present an efficient optimization algorithm to address it and acquire a closed-form solution of the transformation matrix. Extensive experiments demonstrate the effectiveness and efficiency of the proposed UFS by comparison with several state-of-the-art approaches to clustering and image segmentation tasks.

Recently, graph-based multi-view clustering (GMC) has attracted extensive attention from researchers, in which multi-view clustering based on structured graph learning (SGL) can be considered as one of the most interesting branches, achieving promising performance. However, most of the existing SGL methods suffer from sparse graphs lacking useful information, which normally appears in practice. To alleviate this problem, we propose a novel multi-view and multi-order SGL ( [Formula: see text]SGL) model which introduces multiple different orders (multi-order) graphs into the SGL procedure reasonably. To be more specific, [Formula: see text]SGL designs a two-layer weighted-learning mechanism, in which the first layer truncatedly selects part of views in different orders to retain the most useful information, and the second layer assigns smooth weights into retained multi-order graphs to fuse them attentively. Moreover, an iterative optimization algorithm is derived to solve the optimization problem involved in [Formula: see text]SGL, and the corresponding theoretical analyses are provided. In experiments, extensive empirical results demonstrate that the proposed [Formula: see text]SGL model achieves the state-of-the-art performance in several benchmarks.

As a simple yet effective method, least squares regression (LSR) is extensively applied for data regression and classification. Combined with sparse representation, LSR can be extended to feature selection (FS) as well, in which$\ell_{1}$regularization is often applied in embedded FS algorithms. However, because the loss function is in the form of squared error, LSR and its variants are sensitive to noises, which significantly degrades the effectiveness and performance of classification and FS. To cope with the problem, we propose a generalized and robust LSR (GRLSR) for classification and FS, which is made up of arbitrary concave loss function and the$\ell_{2,p}$-norm regularization term. Meanwhile, an iterative algorithm is applied to efficiently deal with the nonconvex minimization problem, in which an additional weight to suppress the effect of noises is added to each data point. The weights can be automatically assigned according to the error of the samples. When the error is large, the value of the corresponding weight is small. It is this mechanism that allows GRLSR to reduce the impact of noises and outliers. According to the different formulations of the concave loss function, four specific methods are proposed to clarify the essence of the framework. Comprehensive experiments on corrupted datasets have proven the advantage of the proposed method.

In this article, we propose a new unsupervised feature selection method named pseudo-label guided structural discriminative subspace learning (PSDSL). Unlike the previous methods that perform the two stages independently, it introduces the construction of probability graph into the feature selection learning process as a unified general framework, and therefore the probability graph can be learned adaptively. Moreover, we design a pseudo-label guided learning mechanism, and combine the graph-based method and the idea of maximizing the between-class scatter matrix with the trace ratio to construct an objective function that can improve the discrimination of the selected features. Besides, the main existing strategies of selecting features are to employ -norm for feature selection, but this faces the challenges of sparsity limitations and parameter tuning. For addressing this issue, we employ the -norm constraint on the learned subspace to ensure the row sparsity of the model and make the selected feature more stable. Effective optimization strategy is given to solve such NP-hard problem with the determination of parameters and complexity analysis in theory. Ultimately, extensive experiments conducted on nine real-world datasets and three biological ScRNA-seq genes datasets verify the effectiveness of the proposed method on the data clustering downstream task.

The existing multiview clustering models learn a consistent low-dimensional embedding either from multiple feature matrices or multiple similarity matrices, which ignores the interaction between the two procedures and limits the improvement of clustering performance on multiview data. To address this issue, a bidirectional probabilistic subspaces approximation (BPSA) model is developed in this article to learn a consistently orthogonal embedding from multiple feature matrices and multiple similarity matrices simultaneously via the disturbed probabilistic subspace modeling and approximation. A skillful bidirectional fusion strategy is designed to guarantee the parameter-free property of the BPSA model. Two adaptively weighted learning mechanisms are introduced to ensure the inconsistencies among multiple views and the inconsistencies between bidirectional learning processes. To solve the optimization problem involved in the BPSA model, an iterative solver is derived, and a rigorous convergence guarantee is provided. Extensive experimental results on both toy and real-world datasets demonstrate that our BPSA model achieves state-of-the-art performance even if it is parameter-free.

Clustering aims to make data points in the same group have higher similarity or make data points in different groups have lower similarity. Therefore, we propose three novel fast clustering models motivated by maximizing within-class similarity, which can obtain more instinct clustering structure of data. Different from traditional clustering methods, we divide all n samples into m classes by the pseudo label propagation algorithm first, and then m classes are merged to c classes ( ) by the proposed three co-clustering models, where c is the real number of categories. On the one hand, dividing all samples into more subclasses first can preserve more local information. On the other hand, proposed three co-clustering models are motivated by the thought of maximizing the sum of within-class similarity, which can utilize the dual information between rows and columns. Besides, the proposed pseudo label propagation algorithm can be a new method to construct anchor graphs with linear time complexity. A series of experiments are conducted on both synthetic and real-world datasets and the experimental results show the superior performance of three models. It is worth noting that for the proposed models, FMAWS2 is the generalization of FMAWS1 and FMAWS3 is the generalization of other two.

Neighborhood reconstruction methods have been widely applied to feature engineering. Existing reconstruction-based discriminant analysis methods normally project high-dimensional data into a low-dimensional space while preserving the reconstruction relationships among samples. However, there are three limitations: 1) the reconstruction coefficients are learned based on the collaborative representation of all sample pairs, which requires the training time to be the cube of the number of samples; 2) these coefficients are learned in the original space, ignoring the interference of the noise and redundant features; and 3) there is a reconstruction relationship between heterogeneous samples; this will enlarge the similarity of heterogeneous samples in the subspace. In this article, we propose a fast and adaptive discriminant neighborhood projection model to tackle the above drawbacks. First, the local manifold structure is captured by bipartite graphs in which each sample is reconstructed by anchor points derived from the same class as that sample; this can avoid the reconstruction between heterogeneous samples. Second, the number of anchor points is far less than the number of samples; this strategy can reduce the time complexity substantially. Third, anchor points and reconstruction coefficients of bipartite graphs are updated adaptively in the process of dimensionality reduction, which can enhance the quality of bipartite graphs and extract discriminative features simultaneously. An iterative algorithm is designed to solve this model. Extensive results on toy data and benchmark datasets show the effectiveness and superiority of our model.

Graph clustering based on graph contrastive learning (GCL) is one of the dominant paradigms in the current graph clustering research field. However, those GCL-based methods often yield false negative samples, which can distort the learned representations and limit clustering performance. In order to alleviate this issue, we propose the idea of maintaining mutual information (MI) between the representations and the inputs to mitigate the loss of semantic information of false negative samples. We demonstrate the validity of this proposal through relevant experiments. Since maximizing MI can be approximately replaced by minimizing reconstruction error, we further propose a graph clustering method based on GCL penalized by reconstruction error, in which our carefully designed reconstruction decoder, as well as reconstruction error term, improve the clustering performance. In addition, we use a pseudo-label-guided strategy to improve the GCL process and further alleviate the problem of false negative samples. Our experiment results demonstrate the superiority and great potential of our proposed graph clustering method compared with state-of-the-art algorithms.

Sparse learning based feature selection has been widely investigated in recent years. In this study, we focus on the l2,0-norm based feature selection, which is effective for exact top-k feature selection but challenging to optimize. To solve the general l2,0-norm constrained problems, we novelly develop a parameter-free optimization framework based on the coordinate descend (CD) method, termed CD-LSR. Specifically, we devise a skillful conversion from the original problem to solving one continuous matrix and one discrete selection matrix. Then the nontrivial l2,0-norm constraint can be solved efficiently by solving the selection matrix with CD method. We impose the l2,0-norm on a vanilla least square regression (LSR) model for feature selection and optimize it with CD-LSR. Extensive experiments exhibit the efficiency of CD-LSR, as well as the discrimination ability of l2,0-norm to identify informative features. More importantly, the versatility of CD-LSR facilitates the applications of the l2,0-norm in more sophisticated models. Based on the competitive performance of l2,0-norm on the baseline LSR model, the satisfactory performance of its applications is reasonably expected. The source MATLAB code are available at: https://github.com/solerxl/Code_For_AAAI_2023.

Most of the existing multi-view clustering algorithms are performed in the original feature space, and their performance in heavily reliant on the quality of the raw data. Besides, some two-stage strategies cannot achieve ideal results due to the absence of capturing the correlation between views. In view of this, we propose Multi-View K-means with Laplacian Embedding (MVKLE), which is capable of clustering multi-view data in the learned embedding space. Specifically, we employ local structure-preserving dimensionality reduction to obtain the underlying representation of each view, and obtain the clustering results directly through an effective optimization formulation. Experiments on several common multi-view datasets demonstrate the superiority of the proposed method.

At data mining field, it is a fundamental problem to dispose of high-dimensional data. Many existing unsupervised methods select features by manifold learning or exploring spectral analysis, thus preserving the intrinsic structure of raw data. But most of them follow an assumption that all features are equally importance. To settle this problem, we draw a novel feature selection module that simultaneously performs learning of feature weights matrix, similarity graph structure and projection matrix, so that the local structure after feature weighting and subspace sparse projection is received. Finally, we solve the model based on ℓ2,0-norm directly by an iterative optimization algorithm and demonstrate the feasibility and effectiveness of our approach via extensive experiments.

Two-dimensional (2D) feature extraction techniques are specifically designed for reducing the dimension of data in matrix representation. Existing methods mostly rely on bilateral projections of matrices. This rasterized manner critically limits the freedom of feature combinations, and thus degrades the fitting ability of models. The robustness of previous techniques is also unsatisfactory due to the insufficiency of discerning outliers with severe damages. In this work, we propose a novel bilinear subspace learning model to achieve flexible and robust two-dimensional feature extraction. The features are exploited in a multilayer bilinear low-rank space under a collaborative orthogonality constraint, and a self-sparse robust strategy is imposed. The model greatly extends the projection space and relaxes the restriction in conventional orthogonal space. We accordingly design a tactful and efficient optimization method based on the coordinate descent method, optimally addressing the proposed model. Experimental results demonstrate the excited improvements of our extracted features in image classification task.

Feature selection is widely used in multimedia applications to determine informative features from high-dimensional data. Due to the explosive growth of the data size and the expensive cost of obtaining labeled data, it is increasingly demanded to utilize both labeled and unlabeled data for feature selection. In this paper, we introduce the l2,0-norm in semi-supervised feature selection, which is able to select exact k informative features. Due to the non-convexity of l2,0-norm, we further devise an efficient coordinate-descent-based algorithm to solve the l2,0-norm constraint, which facilitates the application of l2,0-norm to more complex applications, including but not limited to the proposed model in this study. We experimentally verify the effectiveness of the proposed l2,0-norm-based semi-supervised method and the efficiency of the proposed optimization algorithm.

With the proliferation of multimedia applications, data is frequently derived from multiple sources, leading to the accelerated advancement of multi-view clustering (MVC) methods. In this paper, we propose a novel MVC method, termed GLSEF, to handle the inconsistency existing in multiple spectral embeddings. To this end, GLSEF contains a two-level learning mechanism. Specifically, on the global level, GLSEF considers the diversity of features and selectively assigns smooth weights to partial more discriminative features that are conducive to clustering. On the local level, GLSEF resorts to the Grassmann manifold to maintain spatial and topological information and local structure in each view, thereby enhancing its suitability and accuracy for clustering. Moreover, unlike most previous methods that learn a low-dimension embedding and perform the k-means algorithm to obtain the final cluster labels, GLSEF directly acquires the discrete indicator matrix to prevent potential information loss during post-processing. To address the optimization involved in GLSEF, we present an efficient alternating optimization algorithm accompanied by convergence and time complexity analyses. Extensive empirical results on nine real-world datasets demonstrate the effectiveness and efficiency of GLSEF compared to existing state-of-the-art MVC methods.

Feature selection (FS) plays an important role in machine learning, which extracts important features and accelerates the learning process. In this paper, we propose a deep FS method that simultaneously conducts feature selection and differentiable $ k $-NN graph learning based on the Dirichlet Energy. The Dirichlet Energy identifies important features by measuring their smoothness on the graph structure, and facilitates the learning of a new graph that reflects the inherent structure in new feature subspace. We employ Optimal Transport theory to address the non-differentiability issue of learning $ k $-NN graphs in neural networks, which theoretically makes our method applicable to other graph neural networks for dynamic graph learning. Furthermore, the proposed framework is interpretable, since all modules are designed algorithmically. We validate the effectiveness of our model with extensive experiments on both synthetic and real-world datasets.

Deep clustering seeks to group data into distinct clusters using deep learning techniques . Existing approaches of deep clustering can be broadly categorized into two groups: offline clustering based on unsupervised representation learning and online clustering based on unsupervised classification . While both groups have demonstrated impressive performance in deep clustering, no study has explored the integration of their respective strengths. To this end, we propose Mutual-Taught Deep Clustering (MTDC), which unifies unsupervised representation learning and unsupervised classification into a framework while realizing mutual promotion using a novel mutual-taught mechanism . Specifically, MTDC alternates between predicting pseudolabels in label space and estimating semantic similarity in feature space during training. Moreover, pseudolabels provide weakly-supervised information to enhance unsupervised representation learning, while semantic similarities function as structural priors that regularize unsupervised classification. Consequently, unsupervised classification and unsupervised representation learning can mutually benefit from one another. MTDC is decoupled from prevailing deep clustering methods . For the sake of clarity, we build upon a straightforward baseline in this paper. Despite its simplicity, we demonstrate that MTDC is exceedingly efficacious and consistently enhances the baseline results by substantial margins. For example, MTDC achieves 2.5 % ∼ 7.9 % (NMI), 3.0 % ∼ 13.9 % (ACC), and 3.1 % ∼ 16.7 % (ARI) gains over the baseline on six widely used image datasets. Source code is available at:https://github.com/yichenwang231/MTDC.

Currently, unsupervised hyperspectral image (HSI) segmentation methods are mainly implemented by clustering. Nevertheless, hyperspectral data contains a large amount of noise during the acquisition process, resulting in an abnormal distribution of many pixel points. Traditional clustering algorithms suffer from inaccurate segmentation when dealing with these data. For example, FCM is sensitive to anomalies in the clustering problem of HSI, that makes the clustering accuracy degraded. To address these problems, this paper proposes a method called Fuzzy C-Multiple-Means (FCMM). The method divides data points with multiple subclusters into definedcclusters. Different from the bottom-up coalescent strategy, the proposed FCMM transforms the problem of merging multiple subclusters into an optimisation problem for the fuzzy affiliation matrix, and updates the partitioning of theqsubclusters andcclasses by an alternating iterative update method. This enhances the robustness of the algorithm and reduces the effect of outliers in the HSI datasets on the FCMM, which provides superior clustering results. Experiments on several HSI datasets validate the effectiveness of FCMM.

In the field of data mining, how to deal with high-dimensional data is an inevitable topic. Since it does not rely on labels, unsupervised feature selection has attracted a lot of attention. The performance of spectral-based unsupervised methods depends on the quality of the constructed similarity matrix, which is used to depict the intrinsic structure of data. However, real-world data often contain plenty of noise features, making the similarity matrix constructed by original data cannot be completely reliable. Worse still, the size of a similarity matrix expands rapidly as the number of samples rises, making the computational cost increase significantly. To solve this problem, a simple and efficient unsupervised model is proposed to perform feature selection. We formulate PCA as a reconstruction error minimization problem, and incorporate a L2,p-norm regularization term to make the projection matrix sparse. The learned row-sparse and orthogonal projection matrix is used to select discriminative features. Then, we present an efficient optimization algorithm to solve the proposed unsupervised model, and analyse the convergence and computational complexity of the algorithm theoretically. Finally, experiments on both synthetic and real-world data sets demonstrate the effectiveness of our proposed method.

Multi-view clustering aims to discover common patterns from multi-source data, whose generality is remarkable. Compared with traditional methods, deep learning methods are data-driven and have a larger search space for solutions, which may find a better solution to the problem. In addition, more considerations can be introduced by loss functions, so deep models are highly reusable. However, compared with deep learning methods, traditional methods have better interpretability, whose optimization is relatively stable. In this paper, we propose a multi-view spectral clustering model, combining the advantages of traditional methods and deep learning methods. Specifically, we start with the objective function of traditional spectral clustering, perform multi-view extension, and then obtain the traditional optimization process. By partially parameterizing this process, we further design corresponding differentiable modules, and finally construct a complete network structure. The model is interpretable and extensible to a certain extent. Experiments show that the model performs better than other multi-view clustering algorithms, and its semi-supervised classification extension also has excellent performance compared to other algorithms. Further experiments also show the stability and fewer iterations of the model training.

The principal subspace estimation is directly connected to dimension reduction and is important when there is more than one principal component of interest. In this article, we introduce two new algorithms to solve the feature-sparsity constrained PCA problem (FSPCA) for the principal subspace estimation task, which performs feature selection and PCA simultaneously. Existing optimization methods for FSPCA require data distribution assumptions and are lack of global convergence guarantee. Though the general FSPCA problem is NP-hard, we show that, for a low-rank covariance, FSPCA can be solved globally (Algorithm 1). Then, we propose another strategy (Algorithm 2) to solve FSPCA for the general covariance by iteratively building a carefully designed proxy. We prove (data-dependent) approximation bound and regular stationary convergence guarantees for the new algorithms. For the spectrum of covariance with exponential/Zipf's distribution, we provide exponential/posynomial approximation bounds. Constructive examples and numerical results are provided to demonstrate the tightness of our results. Experimental results show the promising performance and efficiency of the new algorithms compared with the state-of-the-arts on both synthetic and real-world datasets.

Although lots of clustering models have been proposed recently, k-means and the family of spectral clustering methods are both still drawing a lot of attention due to their simplicity and efficacy. We first reviewed the unified framework of k-means and graph cut models, and then proposed a clustering method called k-sums where a k-nearest neighbor ( k-NN) graph is adopted. The main idea of k-sums is to minimize directly the sum of the distances between points in the same cluster. To deal with the situation where the graph is unavailable, we proposed k-sums-x that takes features as input. The computational and memory overhead of k-sums are both O(nk), indicating that it can scale linearly w.r.t. the number of objects to group. Moreover, the costs of computational and memory are Irrelevant to the product of the number of points and clusters. The computational and memory complexity of k-sums-x are both linear w.r.t. the number of points. To validate the advantage of k-sums and k-sums-x on facial datasets, extensive experiments have been conducted on 10 synthetic datasets and 17 benchmark datasets. While having a low time complexity, the performance of k-sums is comparable with several state-of-the-art clustering methods.

In most existing graph-based multi-view clustering methods, the eigen-decomposition of the graph Laplacian matrix followed by a post-processing step is a standard configuration to obtain the target discrete cluster indicator matrix. However, we can naturally realize that the results obtained by the two-stage process will deviate from that obtained by directly solving the primal clustering problem. In addition, it is essential to properly integrate the information from different views for the enhancement of the performance of multi-view clustering. To this end, we propose a concise model referred to as Multi-view Discrete Clustering (MDC), aiming at directly solving the primal problem of multi-view graph clustering. We automatically weigh the view-specific similarity matrix, and the discrete indicator matrix is directly obtained by performing clustering on the aggregated similarity matrix without any post-processing to best serve graph clustering. More importantly, our model does not introduce an additive, nor does it has any hyper-parameters to be tuned. An efficient optimization algorithm is designed to solve the resultant objective problem. Extensive experimental results on both synthetic and real benchmark datasets verify the superiority of the proposed model.

Spectral Clustering (SC) has been the main subject of intensive research due to its remarkable clustering performance. Despite its successes, most existing SC methods suffer from several critical issues. First, they typically involve two independent stages, i.e., learning the continuous relaxation matrix followed by the discretization of the cluster indicator matrix. This two-stage approach can result in suboptimal solutions that negatively impact the clustering performance. Second, these methods are hard to maintain the balance property of clusters inherent in many real-world data, which restricts their practical applicability. Finally, these methods are computationally expensive and hence unable to handle large-scale datasets. In light of these limitations, we present a novel Discrete and Balanced Spectral Clustering with Scalability (DBSC) model that integrates the learning the continuous relaxation matrix and the discrete cluster indicator matrix into a single step. Moreover, the proposed model also maintains the size of each cluster approximately equal, thereby achieving soft-balanced clustering. What's more, the DBSC model incorporates an anchor-based strategy to improve its scalability to large-scale datasets. The experimental results demonstrate that our proposed model outperforms existing methods in terms of both clustering performance and balance performance. Specifically, the clustering accuracy of DBSC on CMUPIE data achieved a 17.93% improvement compared with that of the SOTA methods (LABIN, EBSC, etc.).

The widespread arising of data size gives rise to the necessity of undertaking large-scale data clustering tasks. To do so, the bipartite graph theory is frequently applied to design a scalable algorithm, which depicts the relations between samples and a few anchors, instead of binding pairwise samples. However, the bipartite graphs and existing spectral embedding methods ignore the explicit cluster structure learning. They have to obtain cluster labels by using post-processing like K-Means. More than that, existing anchor-based approaches always acquire anchors by using centroids of K-Means or a few random samples, both of which are time-saving but performance-unstable. In this paper, we investigate the scalability, stableness and integration in large-scale graph clustering. We propose a cluster-structured graph learning model, thus obtaining a c-connected ( c is the cluster number) bipartite graph and also getting discrete labels straightforward. Taking data feature or pairwise relation as a start point, we further design an initialization-independent anchor selection strategy. Experimental results reported for synthetic and real-world datasets demonstrate the proposed method outperforms its peers.

Clustering is used widely in various kinds of signal processing tasks, in which$k$-means is warmly welcomed by the researchers due to its efficiency and simplicity. Nevertheless, it fails to process non-spherical clusters which are common data distribution. As a variant of$k$-means, kernel$k$-means uses a kernel trick to map the raw data into a feature space to better describe the data with improved clustering performance. But the algorithms still have a lot of shortcomings in the application of signal processing: 1) Not all features contain a wealth of useful information, so all dimensions of features cannot be treated equally; 2) The use of high dimensional features for clustering exceedingly increases computational complexity with negligible improvement of clustering performance. To solve the problems, we propose a self-weighted Euler$k$-means (SWEKM) model, which can adaptively identify the importance of different features, perfectly integrating clustering and feature selection into a joint framework. Moreover, Euler kernel is adopted in SWEKM, which is capable of suppressing the interference of noise points and outliers with comparable computational complexity. Extensive experiments on datasets from the UCI database show that the SWEKM outperforms the state-of-the-art kernel$k$-means for clustering-based signal processing tasks.

This work focuses on the task of Video-based Visible-Infrared Person Re-Identification, a promising technique for achieving 24-hour surveillance systems. Two main issues in this field are modality discrepancy mitigating and spatial–temporal information mining. In this work, we propose a novel method, named Intermediary-guided Bidirectional spatial–temporal Aggregation Network (IBAN), to address both issues at once. Specifically, IBAN is designed to learn modality-irrelevant features by leveraging the anaglyph data of pedestrian images to serve as the intermediary. Furthermore, a bidirectional spatial–temporal aggregation module is introduced to exploit the spatial–temporal information of video data, while mitigating the impact of noisy image frames. Finally, we design an Easy-sample-based loss to guide the final embedding space and further improve the model’s generalization performance. Extensive experiments on Video-based Visible-Infrared benchmarks show that IBAN achieves promising results and outperforms the state-of-the-art ReID methods by a large margin, improving the rank-1/mAP by$1.29\%/3.46\%$at the Infrared to Visible situation, and by$5.04\%/3.27\%$at the Visible to Infrared situation. The source code of the proposed method will be released athttps://github.com/lhf12278/IBAN.

In the field of data mining, how to deal with high-dimensional data is a fundamental problem. If they are used directly, it is not only computationally expensive but also difficult to obtain satisfactory results. Unsupervised feature selection is designed to reduce the dimension of data by finding a subset of features in the absence of labels. Many unsupervised methods perform feature selection by exploring spectral analysis and manifold learning, such that the intrinsic structure of data can be preserved. However, most of these methods ignore a fact: due to the existence of noise features, the intrinsic structure directly built from original data may be unreliable. To solve this problem, a new unsupervised feature selection model is proposed. The graph structure, feature weights, and projection matrix are learned simultaneously, such that the intrinsic structure is constructed by the data that have been feature weighted and projected. For each data point, its nearest neighbors are acquired in the process of graph construction. Therefore, we call them adaptive neighbors. Besides, an additional constraint is added to the proposed model. It requires that a graph, corresponding to a similarity matrix, should contain exactly c connected components. Then, we present an optimization algorithm to solve the proposed model. Next, we discuss the method of determining the regularization parameter γ in our proposed method and analyze the computational complexity of the optimization algorithm. Finally, experiments are implemented on both synthetic and real-world datasets to demonstrate the effectiveness of the proposed method.

In this article, we focus on utilizing the idea of co-clustering algorithms to address the subspace clustering problem. In recent years, co-clustering methods have been developed greatly with many important applications, such as document clustering and gene expression analysis. Different from the traditional graph-based methods, co-clustering can utilize the bipartite graph to extract the duality relationship between samples and features. It means that the bipartite graph can obtain more information than other traditional graph methods. Therefore, we proposed a novel method to handle the subspace clustering problem by combining dictionary learning with a bipartite graph under the constraint of the (normalized) Laplacian rank. Besides, to avoid the effect of redundant information hiding in the data, the original data matrix is not used as the static dictionary in our model. By updating the dictionary matrix under the sparse constraint, we can obtain a better coefficient matrix to construct the bipartite graph. Based on Theorem 2 and Lemma 1, we further speed up our algorithm. Experimental results on both synthetic and benchmark datasets demonstrate the superior effectiveness and stability of our model.

Linear discriminant analysis (LDA) aims to find a low-dimensional space in which data points in the same class are to be close to each other while keeping data points from different classes apart. To improve the robustness of LDA to non-Gaussian distribution data, most existing discriminant analysis methods extend LDA by approximating the underlying manifold of data. However, these methods suffer from the following problems: 1) local affinity or reconstruction coefficients are learned on the basis of the relationships of all data pairs, which would lead to a sharp increase in the amount of computation and 2) they learn the manifold information in the original space, ignoring the interference of the noise and redundant features. Motivated by these challenges, this article represents a novel discriminant analysis model, called fast and adaptive locality discriminant analysis (FALDA), to improve the efficiency and robustness. First, with the anchor-based strategy, a bipartite graph of each class is constructed to characterize the local structure of data. Since the number of anchor points is far less than that of data points, learning of fuzzy membership relationships between data points and anchor points within each class can save training time. Second, a maximum entropy regularization is introduced to control the uniformity of the weights of graphs and avoid the trivial solution. Third, the above relationships are updated adaptively in the process of dimensionality reduction, which can suppress the interference of the noise and redundant features. Fourth, the whitening constraint is imposed on the projection matrix to remove the relevance between features and restrict the total scatter of data in the subspace. Last but not the least, data with complex distribution can be explicitly divided into sub-blocks according to the learned anchor points (or subclass center points). We test our proposed method on synthetic data, benchmark datasets, and imbalanced datasets. Promising experimental results demonstrate the success of this novel model.

Fuzzy C-means (FCM) is a kind of classic cluster method, which has been widely used in various fields, such as image segmentation and data mining. Euclidean distance is a frequently used distance metric in FCM, but it is only suitable for data with spherical structure. As an extension of Euclidean distance, Mahalanobis distance has been used in Gustafson–Kessel FCM and its variants to tackle ellipsoidal data. For the convenience of optimizing, most existing Mahalanobis distance-based FCM algorithms only focus on squared Mahalanobis distance. However, squared Mahalanobis distance may not be the best distance metric for FCM because it is easy to enlarge the influence of outliers. In this article, we propose a novel$\ell _{2,p}$-norm and Mahalanobis distance-based FCM model, abbreviated as LM-FCM, which can help FCM improve the ability of tackling ellipsoidal clusters and outliers. Then, in order to reduce computational complexity, we propose a more simplified model, abbreviated as SLM-FCM. Furthermore, we develop an iteratively reweighted optimization algorithm to optimize the proposed models and provide a rigorous monotonous convergence proof. Finally, compared with the existing state-of-the-art FCM algorithms, we conduct extensive experiments on both synthetic and real-world datasets to manifest the superior clustering performance and robustness of the proposed algorithms.

The clustering method has been widely used in data mining, pattern recognition, and image identification. Fuzzy c-means (FCM) is a soft clustering method that introduces the concept of membership. In this method, the fuzzy membership matrix is obtained by calculating the distance between data points in the original space. However, these methods may yield suboptimal results owing to the influence of redundant features. Moreover, FCM is always sensitive to noise points and heavily subject to outliers. In this article, we propose a method called sparsity FCM clustering with principal component analysis embedding (P_SFCM). We simultaneously conduct principal component analysis and membership learning, and then add an additional weighting factor for each data point. The goal of this operation is to identify the noise or outliers. Overall, the benefit of our framework is that it retains most of the information in the subspace while improving the robustness of the noise. In this article, we employ an iterative optimization algorithm to efficiently solve our model. To verify the reliability of the proposed method, we conduct a convergence analysis, noise robustness analysis, and multicluster experiments. Furthermore, comparative experiments are conducted on both synthetic and real benchmark datasets. The experimental results show that the P_SFCM is competitive with comparable methods.

Spectral clustering have attracted more and more attention due to their well-defined mathematical frameworks and superior performance. However, there still exist two limitations to be solved: 1) most spectral clustering methods consist of two independent stages, which may cause unpredictable deviation of obtained clustering results from the genuine ones and lead to severe information loss and performance degradation; 2) spectral clustering methods employ the hard clustering mode, which lacks interpretability for data points in the boundary area belonging to multiple clusters. To simultaneously address these challenging issues for spectral clustering, we propose a graph-based soft-balanced fuzzy clustering (GBFC) model. Specifically, we explicitly preserve the nonnegative property of the clustering indicator matrix to enhance the interpretability of clustering results. Moreover, row normalization is imposed on the cluster indicator matrix to show the membership of each data point to different clusters. Additionally, a novel balanced constraint is designed to regularize the clustering results and constrain the size of clusters. We can directly obtain the clustering assignments without any post-processing, and the limitations of the previous two-stage clustering framework can be effectively addressed. Extensive experiments performed on both synthetic datasets and real world datasets demonstrate the superiority and effectiveness of the proposed algorithm compared with several state-of-the-art methods.

As one of the classical clustering techniques, spectral embedding boasts extensive applicability across numerous domains. Traditional spectral embedding techniques entail the mapping of graph models to low-dimensional vector spaces (indicator vectors) to facilitate hard partitioning. However, data boundaries occasionally exhibit ambiguity, thereby constraining the utility of hard partitioning. In this article, we introduce an innovative spectral embedding method, namely, joint learning of anchor graph-based fuzzy spectral embedding model and fuzzy K-means (AFSEFK). Drawing inspiration from fuzzy logic, our method employs a membership vector in lieu of the conventional indicator vector for spectral embedding, amalgamating it with fuzzy K-means to concurrently optimize membership, thereby simultaneously learning the local and global structures inherent in the data. Moreover, to enhance the quality of similarity graphs and augment clustering performance, we implement the balanced K-means-based hierarchical K-means technique to generate representative anchors. Subsequently, an anchor-based similarity graph is devised through a parameter-free neighbor assignment strategy. Comprehensive extensive experimentation with synthetic and real-world datasets substantiates the efficacy of the AFSEFK algorithm.

Due to the weak and nonstationary properties, electroencephalogram (EEG) data present significant individual differences. To align data distributions of different subjects, transfer learning showed promising performance in cross-subject EEG emotion recognition. However, most of the existing models sequentially learned the domain-invariant features and estimated the target domain label information. Such a two-stage strategy breaks the inner connections of both processes, inevitably causing the suboptimality. In this article, we propose a joint EEG feature transfer and semisupervised cross-subject emotion recognition model in which the shared subspace projection matrix and target label are jointly optimized toward the optimum. Extensive experiments are conducted on SEED-IV and SEED, and the results show that the emotion recognition performance is significantly enhanced by the joint learning mode and the spatial-frequency activation patterns of critical EEG frequency bands and brain regions in cross-subject emotion expression are quantitatively identified by analyzing the learned shared subspace.

Since obtaining data labels is a time-consuming and laborious task, unsupervised feature selection has become a popular feature selection technique. However, the current unsupervised feature selection methods are facing three challenges: (1) they rely on a fixed similarity matrix derived from the original data, which will affect their performance; (2) due to the limitation of sparsity, they can only obtain sub-optimal solutions; (3) they have high computational complexity and cannot handle large-scale data. To solve this dilemma, we propose a fast unsupervised feature selection algorithm with bipartite graph and 2;0-norm constraint (BGCFS). We use the original data and the selected anchors to construct an adaptive bipartite graph in the subspace, and apply the l2,0-norm constraint to the projection matrix for feature selection. In this way, we can update the adaptive bipartite graph and the projection matrix simultaneously, and we can get the feature subset directly, without sorting the features. In addition, we propose an iterative algorithm that can solve the proposed problem globally to obtain a closed-form solution, and we provide a strict proof of convergence for it. Experiments on eight real data sets with different scales show that our method can select more valuable feature subsets more quickly

Recently, several robust principal component analysis (RPCA) models were presented to enhance the robustness of PCA by exploiting the robust norms as their loss functions. But an important problem is that they have no ability to discriminate outliers from correct samples. To solve this problem, we propose a RPCA method based on discriminant information (RPCA-DI). RPCA-DI disentangles the robust PCA with a two-step fashion: the identification and the processing of outliers. To identity outliers, a sample representation model based on entropy regularization is constructed to analyze the membership of data belonging to the principal component space(PC) and its orthogonal complement(OC), the discriminative information of data will be extracted based on measuring the differences of retained information on PC(or OC) of data. By this way, we can discriminate correct samples when we deal with outliers, which is more reasonable for robustness learning respective to previous works. In the noise processing step, in addition to considering the levels of noise, the resistance of the sample points to noise is also considered to prevent overfitting, thereby improving the generalization performance of RPCA-DI. Finally, an iterative algorithm is designed to solve the corresponding model. Compared with some state-of-art RPCA methods on artificial datasets, UCI datasets and face databases that verifies the effectiveness of RPCA-DI.

Fuzzy clustering algorithms have been widely used to reveal the possible hidden structure of data. However, with the increasing of data amount, large scale data has brought genuine challenges for fuzzy clustering. Most fuzzy clustering algorithms suffer from the long time-consumption problem since a large amount of distance calculations are involved to update the solution per iteration. To address this problem, we introduce the popular anchor graph technique into fuzzy clustering and propose a scalable fuzzy clustering algorithm referred to as Scalable Fuzzy Clustering with Anchor Graph (SFCAG). The main characteristic of SFCAG is that it addresses the scalability issue plaguing fuzzy clustering from two perspectives: anchor graph construction and membership matrix learning. Specifically, we select a small number of anchors and construct a sparse anchor graph, which is beneficial to reduce the computational complexity. We then formulate a trace ratio model, which is parameter-free, to learn the membership matrix of anchors to speed up the clustering procedure. In addition, the proposed method enjoys linear time complexity with the data size. Extensive experiments performed on both synthetic and real world datasets demonstrate the superiority (both effectiveness and scalability) of the proposed method over some representative large scale clustering methods.

Among a big body of recently developed algorithms for machine learning and data mining, a class of models using non-convex/non-smooth sparsity-inducing norms achieves promising results on many challenging tasks. An important problem faced with such models is to find an effective solution for the objective function with one or multiple intractable terms. Although a large number of optimization approaches have been developed, most of them are tailored to a specific model. Besides, these approaches generally introduce some additional parameters and no longer guarantee convergence. In this work, we first revisit some representative non-convex/non-smooth machine learning models, and then unity them into a generic formulation. Theoretically, we develop a simple yet efficient optimization framework, namely Iteratively Re-Weighted method (IRW), to solve such a class of models and provide the corresponding convergence analysis. Particularly, we validate our proposed method on two challenging machine learning tasks: multi-task regression and feature selection. Source codes are available at:https://github.com/KDD-Code/Sparse.git.

K-means is one of the most simple and popular clustering algorithms, which implemented as a standard clustering method in most of machine learning researches. The goal of K-means clustering is finding a set of cluster centers and minimizing the sum of squared distances between each sample and its nearest clustering center. In this paper, we proposed a novel K-means clustering algorithm, which reformulate the classical K-Means objective function as a trace maximization problem and then replace it with a new formulation. The proposed algorithm does not need to calculate the cluster centers in each iteration and requires fewer additional intermediate variables during the optimization process. In addition, we proposed an efficient iterative re-weighted algorithm to solve the involved optimization problem and provided the corresponding convergence analysis. The proposed algorithm keeps a consistent computational complexity as Lloyd's algorithm,$\mathcal {O}(ndk)$, but shows a faster convergence rate in experiments. Extensive experimental results on real world benchmark datasets show the effectiveness and efficiency of the proposed algorithm.

Multi-view clustering attracts considerable attention due to its effectiveness in unsupervised learning. However, previous multi-view spectral clustering methods include two separated steps: 1) Obtaining a spectral embedding; 2) Performing classical clustering methods. Although these methods have achieved promising performance, there is still some limitations. First, in computing spectral embedding, multi-view spectral clustering approaches exist high computational complexity since they usually need eigenvalue decomposition on laplacian matrix L. Its computational complexity is O(n3) where n is the number of samples; Second, in constructing similarity matrices, previous methods need to compute similarity between any two samples; Third, the two-stage approach only can obtain sub-optimal solution; Fourth, treating equally all views is unreasonable. To address these issues, we propose a Fast Multi-view Clustering via Prototype Graph (FMVPG) method. Specifically, the prototype graph is firstly constructed, and then simultaneously perform spectral embedding to obtain the real matrix and spectral rotation to get the indicator matrix. In addition, the Alternating Direction Method of Multipliers (ADMM) is used to solve the joint optimization problem. Further, we conduct extensive experiments to evaluate the proposed FMVPG approach. These experimental results show the comparable or even better clustering performance than the state-of-the-art approaches.

Spectral clustering is a vital clustering method and has been widely applied for data analysis and pattern reorganization. A routine of solving spectral clustering problem consists of two successive stages: (1) solving a relaxed continuous optimization problem to obtain a real-valued indicator solution (2) transform the real-valued indicator into a 0-1 discrete one as the final clustering result. However, we may lose the optimal solution with such a two-stage process. Besides, the spectral clustering has a high time complexity which limits the analysis of large-scale data. To alleviate these problems, this paper proposes an efficient spectral clustering framework that computes spectral embedding and improved spectral rotation simultaneously (SE-ISR). In addition, we also provide a parameter-free method (SE-ISR-PF) to automatically choose the trade-off parameter. Furthermore, with an anchor-based similarity matrix construction, it is scalable to large-scale data. An effective algorithm with a strict convergence proof is provided to solve the corresponding optimization problem. Experimental results on several benchmark datasets demonstrate that the proposed algorithm outperforms the state-of-art methods.

The recent work Unsupervised Linear Discriminant Analysis (Un-LDA) completes its clustering process during the alternating optimization by converting equivalently the objective and finally using the K-means algorithm. However, the K-means algorithm has its inherent drawbacks. It is hard for the K-means algorithm to deal well with some complex clustering cases where there are too many real clusters or non-convex clusters. In this paper, a novel clustering optimization method is presented to accomplish the clustering process in Un-LDA and the resulting method can be named Un-LDA(CD). Specifically, instead of the K-means algorithm, an elaborately designed coordinate descent algorithm is adopted to obtain the clusters after the objective function goes through a series of simple but deft equivalent conversions. Extensive experiments have demonstrated that the coordinate descent clustering solution for Un-LDA can outperform the original K-means based solution on the tested data sets especially those complex data sets with a pretty large number of real clusters.

In recent decades, unsupervised feature selection methods have become increasingly popular. Nevertheless, most of the existing unsupervised feature selection methods suffer from two major problems that lead to suboptimal solutions. Many methods impose a hard linear projection constraint on original data, which is overly strict in nature and not suitable for dealing with data sampled from nonlinear manifolds. Second, most existing methods usel2,p-norm (02S and SF2SOG, which can simultaneously learn optimal flexible projections and obtain an orthogonal sparse projection to directly select discriminative features by applyingl2,0-norm constraint. Moreover, we propose to explore the local structure of flexible embedding through preserving the manifold structure of original data and adaptively constructing an optimal graph in subspace. Thirdly, the novel iterative optimization algorithms are presented to solve objective functions guaranteeing convergence theoretically. Various evaluation experiments on synthetic and real-world datasets demonstrate the effectiveness and superiority of our proposed methods.

Graph-based semi-supervised learning, which further utilizes graph structure behind samples for boosting semi-supervised learning, gains convincing results in several machine learning tasks. Nevertheless, existing graph-based methods have shortcomings from two aspects. On the one hand, many of them concentrate on improving label propagation over the constructed graph through time-saving methods, e.g. path searching, without giving insights on constructing a proper graph accommodated to samples. On the other hand, some models are only devoted to constructing the appropriate graph resulting in a two-stage procedure, which may incur a suboptimal scenario. In this paper, we develop a joint learning method that considers both bipartite graph construction and label propagation simultaneously. With this configuration, the constructed graph is constantly adjusted by the smoothness term in the objective as the algorithm proceeds. The time complexity of our method gets significant improvement compared with traditional graph-based methods, and the experimental results on one synthetic dataset and several real-world benchmarks demonstrate the effectiveness and scalability of our proposed method.

Given an affinity graph of data samples, graph-based clustering aims to partition these samples into disjoint groups based on the affinities, and most previous works are based on spectral clustering. However, two problems among spectral-based methods heavily affect the clustering performance. Firstly, the randomness of post-processing procedures, such as$K$-means, affects the stability of clustering. Secondly, the separated stages of spectral-based methods, including graph construction, spectral embedding learning, and clustering decision, lead to mismatched problems. In this paper, we explore a structured graph learning (SGL) framework that aims to fuse these stages to improve clustering stability. Specifically, SGL adaptively learns a structured affinity graph that contains exact$k$connected components. Each connected component corresponds to a cluster so clustering assignments can be directly obtained according to the connectivity of the learned graph. In this way, SGL avoids the randomness brought by reliance on traditional post-processing procedures. Meanwhile, the graph construction and structured graph learning procedures happen simultaneously, which alleviates the mismatched problem effectively. Moreover, we propose an efficient algorithm to solve the involved optimization problems and discuss the connections between this work and previous works. Numerical experiments on several synthetic and real datasets demonstrate the effectiveness of our methods.

Considering many graph clustering methods are with quadratic or cubic time complexity and need post-processing to obtain the discrete solution. Combining with the anchor graph, we study a novel graph clustering model called Rank-$r$Discrete Matrix Factorization (DMF-RR), which is linear time complexity, and motivated by nonnegative matrix factorization (NMF). Instead of constraining the factor matrices of NMF to be nonnegative as many existed methods, we constrain them to indicator matrices. Thus, DMF-RR can obtain the discrete solution by directly solving the original problem without post-processing. Furthermore, considering the greater similarity between samples of the same category, an anchor graph is constructed as an input to capture essential clustering structure by utilizing the duality information between samples and anchors. Subsequently, an efficient and simple algorithm is proposed due to the nature of indicator matrices. Extensive experiments performed on synthetic and real-world datasets demonstrate the superiority of DMF-RR.

Multi-view clustering has attracted a lot of attention due to its ability to integrate information from distinct views, but how to improve efficiency is still a hot research topic. Anchor graph-based methods and k-means-based methods are two current popular efficient methods, however, both have limitations. Clustering on the derived anchor graph takes a while for anchor graph-based methods, and the efficiency of k-means-based methods drops significantly when the data dimension is large. To emphasize these issues, we developed an efficient multi-view k-means clustering method with multiple anchor graphs (EMKMC). It first constructs anchor graphs for each view and then integrates these anchor graphs using an improved k-means strategy to obtain sample categories without any extra post-processing. Since EMKMC combines the high-efficiency portions of anchor graph-based methods and k-means-based methods, its efficiency is substantially higher than current fast methods, especially when dealing with large-scale high-dimensional multi-view data. Extensive experiments demonstrate that, compared to other state-of-the-art methods, EMKMC can boost clustering efficiency by several to thousands of times while maintaining comparable or even exceeding clustering effectiveness.

Fuzzy K-Means (FKM) clustering regards each cluster as a fuzzy set and assigns each sample to multiple clusters with a certain degree of membership. However, conventional FKM methods perform clustering on original data directly where the intrinsic structure of data may be corrupted by the noise. According, the performance of these methods would be challenged. In this paper, we present a novel fuzzy K-Means clustering model to conduct clustering tasks on the flexible manifold. Technically, we perform fuzzy clustering based on the shrunk patterns which have desired manifold structure. The shrunk patterns can be viewed as an approximation to the original data; and a penalty term is employed to measure the mismatch between them. Moreover, we integrate the learning of shrunk patterns and the learning of membership degree between shrunk patterns and clusters into a unified framework. Furthermore, we extend the proposed model for projected FKM clustering to find a suitable subspace to fit the non-linear manifold structure of data, reduce the interference of the noise and redundant features and gather homogeneous samples together simultaneously. Two alternating iterative algorithms are derived to solve these two models, respectively. Extensive experimental results demonstrate the feasibility and effectiveness of our proposed clustering algorithms.

In traditional graph embedding methods, graph construction is sensitive to high-dimensional data with noise and outliers, making an effective exploration of the neighborhood structure of the data difficult. Besides, with these methods, constructing graphs and reducing dimensions are disconnected and cannot be mutually optimized. To address these problems, we propose an unsupervised dimensionality reduction method based on bipartite graph, named unsupervised adaptive bipartite graph embedding (UABGE). First, the anchors are generated from the raw data by K-means or random sampling. Second, the bipartite graph, which is constructed between the samples and the anchors in the low-dimensional subspace, utilizes the adaptive allocation method to assign neighbors for each sample, so that the local structure of high-dimensional data can be captured effectively. Third, we present an objective function that combines bipartite graph construction and projection matrix learning to achieve mutual optimization between them, which can be solved with an alternating optimization algorithm. Finally, the computational complexity and the convergence of the algorithm are analyzed. Experimental results on synthetic data and publicly available datasets illustrate the effectiveness of the proposed method.

Graph embedding is a widely used method for dimensionality reduction due to its computational effectiveness. The quality of the graph and the efficiency of graph construction will directly affect the performance and the efficiency of the graph embedding methods. However, in the unsupervised graph embedding methods, the graph is not considered as an optimized graph since there is no label that can be used to construct this graph. In addition, the running of traditional graph embedding methods become very time-consuming on large-scale datasets due to the high computational cost in the step of graph construction. Aiming to solve these problems, we propose an unsupervised dimensionality reduction method based on bipartite graph, called Unsupervised Optimized Bipartite Graph Embedding (UOBGE). Representative anchors are firstly identified in the data. Then, we construct the bipartite graph between the projected samples and the projected anchors and the intrinsic graph connecting all the projected sample pairs with equal weights, which keep the local and global geometric structures of the data, respectively. Finally, the bipartite graph and the projection matrix are optimized simultaneously by introducing an alternating optimization procedure. Extensive experiments on several datasets demonstrate that the effectiveness and efficiency of the proposed method.

Due to the difficulty of annotating large amounts of training data, directly learning the association of sound and its makers in natural videos is a challenging task for machines. In this paper, we present a novel audiovisual model that introduces a soft-clustering module as the audio and visual content detector, and regards the pervasive property of audiovisual concurrency as the latent supervision for inferring the correlation among detected contents. Furthermore, we discover for the first time that the complexity of data has an impact on the training efficiency and subsequent performance of audiovisual model, i.e., more complex data brings more obstacles to the model training, and degrades the performance of downstream audiovisual tasks. To address the issue of audiovisual learning, we propose a novel heterogeneous audiovisual scene analysis module that trains the model from simple to complex scene. We show that such ordered learning procedure rewards the model the merits of easy training and fast convergence. Meanwhile, our audiovisual model can also provide effective unimodal representation and cross-modal alignment performance. We further deploy the well-trained model into practical audiovisual sound localization and separation tasks. We show that our localization model significantly outperforms existing methods, based on which we show comparable performance in sound separation task by comparison to several related SOTA audiovisual learning methods without referring external visualsupervision.

Multitask learning (MTL) is a joint learning paradigm, which fuses multiple related tasks together to achieve the better performance than single-task learning methods. It has been observed by many researchers that different tasks with certain similarities share a low-dimensional common yet latent subspace. In order to get the low-rank structure shared across tasks, trace norm has been used as a convex relaxation of the rank minimization problem. However, trace norm is not a tight approximation for the rank function. To address this important issue, we propose two novel regularization-based models to approximate the rank minimization problem by minimizing the k minimal singular values. For our new models, if the minimal singular values are suppressed to zeros, the rank would also be reduced. Compared with the standard trace norm, our new regularization-based models are the tighter approximations, which can help our models capture the low-dimensional subspace among multiple tasks better. Besides, it is an NP-hard problem to directly solve the exact rank minimization problem for our models. In this article, we proposed two simple but effective strategies to optimize our models, which tactically solves the exact rank minimization problem by setting a large penalizing parameter. Experimental results performed on synthetic and real-world benchmark datasets demonstrate that the proposed models have the ability of learning the low-rank structure shared across tasks and the better performance than other classical MTL methods.

This work focuses on the projected clustering problem. Specifically, an efficient and parameter-free clustering model, named discriminative projected clustering (DPC), is proposed for simultaneously low-dimensional and discriminative projection learning and clustering, from the perspective of least squares regression. The proposed DPC, a constrained regression model, aims at finding both a transformation matrix and a binary indicator matrix to minimize the sum-of-squares error. Theoretically, a significant conclusion is drawn and used to reveal the connection between DPC and linear discriminant analysis (LDA). Experimentally, experiments are conducted on both toy and real-world data to validate the effectiveness and efficiency of DPC; experiments are also conducted on hyperspectral images to further verify its practicability in real-world applications. Experimental results demonstrate that DPC achieves comparable or superior results to some state-of-the-art clustering methods.

Exploiting different representations, or views, of the same object for better clustering has become very popular these days, which is conventionally called multi-view clustering. In general, it is essential to measure the importance of each individual view, due to some noises, or inherent capacities in the description. Many previous works model the view importance as weight, which is simple but effective empirically. In this article, instead of following the traditional thoughts, we propose a new weight learning paradigm in the context of multi-view clustering in virtue of the idea of the reweighted approach, and we theoretically analyze its working mechanism. Meanwhile, as a carefully achieved example, all of the views are connected by exploring a unified Laplacian rank constrained graph, which will be a representative method to compare with other weight learning approaches in experiments. Furthermore, the proposed weight learning strategy is much suitable for multi-view data, and it can be naturally integrated with many existing clustering learners. According to the numerical experiments, the proposed implicit weight learning approach is proven effective and practical to use in multi-view clustering.

Principal component analysis (PCA) is a typical unsupervised dimensionality reduction algorithm, and one of its important weaknesses is that the squared$\ell _{2}$-norm cannot overcome the influence of outliers. Existing robust PCA methods based on paradigm have the following two drawbacks. First, the objective function of PCA based on the$\ell _{1}$-norm has no rotational invariance and limited robustness to outliers, and its solution mostly uses a greedy search strategy, which is expensive. Second, the robust PCA based on the$\ell _{2,1}$-norm and the$\ell _{2,p}$-norm is essential to learn probability weights for data, which only weakens the influence of outliers on the learning projection matrix and cannot be completely eliminated. Moreover, the ability to detect anomalies is also very poor. To solve these problems, we propose a novel discrete robust principal component analysis (DRPCA). Through self-learning binary weights, the influence of outliers on the projection matrix and data center estimation can be completely eliminated, and anomaly detection can be directly performed. In addition, an alternating iterative optimization algorithm is designed to solve the proposed problem and realize the automatic update of binary weights. Finally, our proposed model is successfully applied to anomaly detection applications, and experimental results demonstrate that the superiority of our proposed method compared with the state-of-the-art methods.

Linear discriminant analysis (LDA) is one of the most effective and popular methods to reduce the dimensionality of data with Gaussian assumption. However, LDA cannot handle non-Gaussian data because the center point is incompetent to represent the distribution of data. Some existing methods based on graph embedding focus on exploring local structures via pairwise relationships of data for addressing the non-Gaussian issue. Due to massive pairwise relationships, the computational complexity is high as well as the locally optimal solution is hard to find. To address these issues, we propose a novel and efficient local embedding learning via landmark-based dynamic connections (LDC) in which we leverage several landmarks to represent different subclusters in the same class and establish the connections between each point and landmark. Furthermore, in order to explore the relationship of landmarks pairwise more precisely, the relationship between each point and their corresponding neighbor landmarks are found in the optimal subspace, rather than the original space, which can avoid the negative influence of the noises. We also propose an efficient iterative algorithm to deal with the proposed ratio minimization problem. Extensive experiments conducted on several real-world datasets have demonstrated the advantages of the proposed method.

The goal of multi-view clustering is to partition samples into different subsets according to their diverse features. Previous multi-view clustering methods mainly exist two forms: multi-view spectral clustering and multi-view matrix factorization. Although they have shown excellent performance in many occasions, there are still many disadvantages. For example, multi-view spectral clustering usually needs to perform postprocessing. Multi-view matrix factorization directly decomposes the original data features. When the size of features is large, it encounters the expensive time consumption to decompose these data features thoroughly. Therefore, we proposed a novel multi-view clustering approach. The main advantages include the following three aspects: 1) it searches for a common joint graph across multiple views, which fully explores the hidden structure information by utilizing the compatibility among views; 2) the introduced nonnegative constraint manipulates that the final clustering results can be directly obtained; and 3) straightforwardly decomposing the similarity matrix can transform the eigenvalue factorization in spectral clustering with computational complexity$O(n^{3})$into the singular value decomposition (SVD) with$O({n}{c}^{2})$time cost, where${n}$and$c$, respectively, denote the numbers of samples and classes. Thus, the computational efficiency can be improved. Moreover, in order to learn a better clustering model, we set that the constructed similarity graph approximates each view affinity graph as close as possible by adding the constraint as the initial affinity matrices own. Furthermore, substantial experiments are conducted, which verifies the superiority of the proposed two clustering methods comparing with single-view clustering approaches and state-of-the-art multi-view clustering methods.

Feature selection is an important and effective data preprocessing method, which can remove the noise and redundant features while retaining the relevant and discriminative features in high-dimensional data. In real-world applications, the relationships between data samples and their labels are usually nonlinear. However, most of the existing feature selection models focus on learning a linear transformation matrix, which cannot capture such a nonlinear structure in practice and will degrade the performance of downstream tasks. To address the issue, we propose a novel nonlinear feature selection method to select those most relevant and discriminative features in high-dimensional dataset. Specifically, our method learns the nonlinear structure of high-dimensional data by a neural network with cross entropy loss function, and then using the structured sparsity norm such as$\ell _{2,p}$-norm to regularize the weights matrix connecting the input layer and the first hidden layer of the neural network model to learn weight of each feature. Therefore, a structural sparse weights matrix is obtained by conducting nonlinear learning based on a neural network with structured sparsity regularization. Then, we use the gradient descent method to achieve the optimal solution of the proposed model. Evaluating the experimental results on several synthetic datasets and real-world datasets shows the effectiveness and superiority of the proposed nonlinear feature selection model.

Existing domain adaptation approaches often try to reduce distribution difference between source and target domains and respect domain-specific discriminative structures by some distribution [e.g., maximum mean discrepancy (MMD)] and discriminative distances (e.g., intra-class and inter-class distances). However, they usually consider these losses together and trade off their relative importance by estimating parameters empirically. It is still under insufficient exploration so far to deeply study their relationships to each other so that we cannot manipulate them correctly and the model's performance degrades. To this end, this article theoretically proves two essential facts: 1) minimizing MMD equals to jointly minimizing their data variance with some implicit weights but, respectively, maximizing the source and target intra-class distances so that feature discriminability degrades and 2) the relationship between intra-class and inter-class distances is as one falls and another rises. Based on this, we propose a novel discriminative MMD with two parallel strategies to correctly restrain the degradation of feature discriminability or the expansion of intra-class distance; specifically: 1) we directly impose a tradeoff parameter on the intra-class distance that is implicit in the MMD according to 1) and 2) we reformulate the inter-class distance with special weights that are analogical to those implicit ones in the MMD and maximizing it can also lead to the intra-class distance falling according to 2). Notably, we do not consider the two strategies in one model due to 2). The experiments on several benchmark datasets not only prove the validity of our revealed theoretical results but also demonstrate that the proposed approach could perform better than some compared state-of-art methods substantially. Our preliminary MATLAB code will be available at https://github.com/WWLoveTransfer/.

Feature selection and feature extraction, in the field of data dimensionality reduction, are the two main strategies. Nevertheless, each of these two strategies has its own advantages and disadvantages. The features chosen by feature selection method have complete physical meaning. However, feature selection cannot reveal the implicit structural information of the samples. In this article, the methods proposed by us combine both feature selection and feature extraction, called joint feature selection and extraction with sparse unsupervised projection (SUP) and graph optimization SUP (GOSUP). A constraint on the number of nonzero rows of the projection matrix is added, which ensures the sparsity of the projection matrix, and only the features corresponding to the nonzero rows of the projection matrix are selected for the feature extraction procedure. We invoke a newly proposed algorithm to tackle this constrained optimization problem. A new concept of "purification matrix" is invented, the use of which could better eliminate meaningless information of samples in subspace. The performance on several datasets verifies the effectiveness of the proposed method for data dimensionality reduction.

Least squares regression (LSR) is widely applied in statistics theory due to its theoretical solution, which can be used in supervised, semisupervised, and multiclass learning. However, LSR begins to fail and its discriminative ability cannot be guaranteed when the original data have been corrupted and noised. In reality, the noises are unavoidable and could greatly affect the error construction in LSR. To cope with this problem, a robust supervised LSR (RSLSR) is proposed to eliminate the effect of noises and outliers. The loss function adopts$\ell _{2,p}$-norm ($0< p\leq 2$) instead of square loss. In addition, the probability weight is added to each sample to determine whether the sample is a normal point or not. Its physical meaning is very clear, in which if the point is normal, the probability value is 1; otherwise, the weight is 0. To effectively solve the concave problem, an iterative algorithm is introduced, in which additional weights are added to penalize normal samples with large errors. We also extend RSLSR to robust semisupervised LSR (RSSLSR) to fully utilize the limited labeled samples. A large number of classification performances on corrupted data illustrate the robustness of the proposed methods.

One of the hottest topics in unsupervised learning is how to efficiently and effectively cluster large amounts of unlabeled data. To address this issue, we propose an orthogonal conceptual factorization (OCF) model to increase clustering effectiveness by restricting the degree of freedom of matrix factorization. In addition, for the OCF model, a fast optimization algorithm containing only a few low-dimensional matrix operations is given to improve clustering efficiency, as opposed to the traditional CF optimization algorithm, which involves dense matrix multiplications. To further improve the clustering efficiency while suppressing the influence of the noises and outliers distributed in real-world data, an efficient correntropy-based clustering algorithm (ECCA) is proposed in this article. Compared with OCF, an anchor graph is constructed and then OCF is performed on the anchor graph instead of directly performing OCF on the original data, which can not only further improve the clustering efficiency but also inherit the advantages of the high performance of spectral clustering. In particular, the introduction of the anchor graph makes ECCA less sensitive to changes in data dimensions and still maintains high efficiency at higher data dimensions. Meanwhile, for various complex noises and outliers in real-world data, correntropy is introduced into ECCA to measure the similarity between the matrix before and after decomposition, which can greatly improve the clustering effectiveness and robustness. Subsequently, a novel and efficient half-quadratic optimization algorithm was proposed to quickly optimize the ECCA model. Finally, extensive experiments on different real-world datasets and noisy datasets show that ECCA can archive promising effectiveness and robustness while achieving tens to thousands of times the efficiency compared with other state-of-the-art baselines.

Graph-based subspace learning has been widely used in various applications as the rapid growth of data dimension, while the graph is constructed by affinity matrix of input data. However, it is difficult for these subspace learning methods to preserve the intrinsic local structure of data with the high-dimensional noise. To address this problem, we proposed a novel unsupervised dimensionality reduction approach named unsupervised subspace learning with flexible neighboring (USFN). We learn a similarity graph by adaptive probabilistic neighborhood learning process to preserve the manifold structure of high-dimensional data. In addition, we utilize the flexible neighboring to learn projection and latent representation of manifold structure of high-dimensional data to remove the impact of noise. The adaptive similarity graph and latent representation are jointly learned by integrating adaptive probabilistic neighborhood learning and manifold residue term into a unified objection function. The experimental results on synthetic and real-world datasets demonstrate the performance of the proposed unsupervised subspace learning USFN method.

Neighborhood reconstruction is a good recipe to learn the local manifold structure. Representation-based discriminant analysis methods normally learn the reconstruction relationship between each sample and all the other samples. However, reconstruction graphs constructed in these methods have three limitations: 1) they cannot guarantee the local sparsity of reconstruction coefficients; 2) heterogeneous samples may own nonzero coefficients; and 3) they learn the manifold information prior to the process of dimensionality reduction. Due to the existence of noise and redundant features in the original space, the prelearned manifold structure may be inaccurate. Accordingly, the performance of dimensionality reduction would be affected. In this article, we propose a joint model to simultaneously learn the affinity relationship, reconstruction relationship, and projection matrix. In this model, we actively assign neighbors for each sample and learn the inter-reconstruction coefficients between each sample and their neighbors with the same label information in the process of dimensionality reduction. Specifically, a sparse constraint is employed to ensure the sparsity of neighbors and reconstruction coefficients. The whitening constraint is imposed on the projection matrix to remove the relevance between features. An iterative algorithm is proposed to solve this method. Extensive experiments on toy data and public datasets show the superiority of the proposed method.

With its simplicity and effectiveness, k-means is immensely popular, but it cannot perform well on complex nonlinear datasets. Multiple kernel k-means (MKKM) demonstrates the ability to describe highly complex nonlinear separable data structures. However, its speed requirement cannot scale as well as the data size grows beyond tens of thousands. Nowadays, digital data explosion mandates more scalable clustering methods to assist the machine learning tasks in easy-to-access form. To address the issue, we propose to employ the Nystrom scheme for MKKM clustering, termed scalable multiple kernel k-means clustering. It significantly reduces the computational complexity by replacing the original kernel matrix with a low-rank approximation. Analytically and empirically, we demonstrate that our method performs as well as existing state-of-the-art methods, but at a significantly lower compute cost, allowing us to scale the method more effectively for clustering tasks.

Multiple Kernel K-means (MKKM) uses various kernels from different sources to improve clustering performance. However, most of the existing models are non-convex, which is prone to be stuck into bad local optimum, especially with noise and outliers. To address the issue, we propose a novel Self-Paced and Discrete Multiple Kernel K-Means (SPD-MKKM). It learns the MKKM model in a meaningful order by progressing both samples and kernels from easy to complex, which is beneficial to avoid bad local optimum. In addition, whereas existing methods optimize in two stages: learning the relaxation matrix and then finding the discrete one by extra discretization, our work can directly gain the discrete cluster indicator matrix without extra process. What's more, a well-designed alternative optimization is employed to reduce the overall computational complexity via using the coordinate descent technique. Finally, thorough experiments performed on real-world datasets illustrated the excellence and efficacy of our method.

Multiple Kernel k-means and its variants integrate a group of kernels to improve clustering performance, but it still has some drawbacks: 1) linearly combining base kernels to get the optimal one limits the kernel representability and cuts off the negotiation of kernel learning and clustering; 2) ignoring the correlation among kernels leads to kernel redundancy; 3) solving NP-hard cluster assignment problem by a two-stage strategy leads to information loss. In this paper, we propose the Discrete Multi-kernel k-means with Diverse and Optimal Kernel Learning (DMK-DOK) model, which adaptively seeks for a better kernel by residing in the base kernel neighborhood and negotiates the kernel learning and clustering. Moreover, it implicitly penalizes the highly correlated kernels to enhance the kernel fusion with less redundancy and more diversity. What’s more, it jointly learns discrete and relaxed labels in the same optimization objective, which can avoid information loss. Lastly, extensive experiments conducted on real-world datasets illustrated the superiority of our model.

Multiple kernel k-means clustering (MKKM) and its variants have been thoroughly studied over the past decades. However, most existing models utilize a spectrum-based two-step approach to solve the clustering objective, which may deviate from the final cluster labels and lead to suboptimal performance. To address this issue, we elaborate a novel MKKMSR framework that simultaneously optimizes the discrete and continuous cluster labels by incorporating spectral rotation into MKKM. In addition, the proposed model can be easily integrated with other MKKM models to boost their performance. What’s more, an efficient alternative algorithm is proposed to solve the joint optimization problem. Extensive experiments on real-world datasets demonstrate the superiorities of the proposed framework.

Multiple kernel clustering seeks to combine several kernels for boosting the clustering performance. However, most existing MKKM methods fail to evaluate kernel correlation adequately, which may inevitably select highly correlated kernels resulting in kernel redundancy. Besides, most existing methods solve the NP-hard cluster labels assignment task in two stages: first learning the relaxed labels with continuous values and then obtaining the discrete labels via other discretization methods like k-means. This two-stage strategy may result in the loss of information owing to the deviation between the genuine solution and the approximated one. In this work, we present a unified framework for Discrete Multi-kernel k-means with Kernel Diversity Regularization (DMK-KDR). It is capable of penalizing highly correlated kernels through a well-designed matrix-induced regularization, thus allowing for improved diversity and reduced redundancy in kernel fusion. Additionally, it learns both discrete and continuous clustering indicator matrices simultaneously, thereby ensuring the integrity of the discrete solution without over-reliance on k-means or the loss of information. The efficacy of our model has been evaluated in a number of experiments using real-world datasets.

This paper proposes an Efficient Multi-view Graph Clustering with Comprehensive Fusion (EMGC²F) model and a corresponding efficient optimization algorithm to address multi-view graph clustering tasks effectively and efficiently. Compared to existing works, our proposals have the following highlights: 1) EMGC²F directly finds a consistent cluster indicator matrix with a Super Nodes Similarity Minimization module from multiple views, which avoids time-consuming spectral decomposition in previous works. 2) EMGC²F comprehensively mines information from multiple views. More formally, it captures the consistency of multiple views via a Cross-view Nearest Neighbors Voting (CN²V) mechanism, meanwhile capturing the importance of multiple views via an adaptive weighted-learning mechanism. 3) EMGC²F is a parameter-free model and the time complexity of the proposed algorithm is far less than existing works, demonstrating the practicability. Empirical results on several benchmark datasets demonstrate that our proposals outperform SOTA competitors both in effectiveness and efficiency.

Hyperspectral image (HSI) contains rich spectral information and spatial features, but the huge amount of data often leads to problems of low clustering accuracy and large computational complexity. In this letter, a new clustering method for HSI is proposed, which is named fast spectral embedded clustering based on structured graph learning (FSECSGL). First, the low-dimensional representation of data can be obtained to reduce the scale by the fast spectral embedded method. Then, we use the embedded data to learn an optimal similarity matrix by structured graph learning. Furthermore, the learning structure graph gives feedback to the original bipartite graph to generate better spectral embedded data. As a result, we can obtain a better similarity matrix and clustering result by iteration, which can overcome the limitation of$K$-means initialization. Experiments show that this method can obtain good clustering performance compared with other methods.

Multiview clustering partitions data into different groups according to their heterogeneous features. Most existing methods degenerate the applicability of models due to their intractable hyper-parameters triggered by various regularization terms. Moreover, traditional spectral based methods always encounter the expensive time overheads and fail in exploring the explicit clusters from graphs. In this paper, we present a scalable and parameter-free graph fusion framework for multiview clustering, seeking for a joint graph compatible across multiple views in a self-supervised weighting manner. Our formulation coalesces multiple view-wise graphs straightforward and learns the weights as well as the joint graph interactively, which could actively release the model from any weight-related hyper-parameters. Meanwhile, we manipulate the joint graph by a connectivity constraint such that the connected components indicate clusters directly. The designed algorithm is initialization-independent and time-economical which obtains the stable performance and scales well with the data size. Substantial experiments on toy data as well as real datasets are conducted that verify the superiority of the proposed method compared to the state-of-the-arts over the clustering performance and time expenditure.

Recently, several robust principle component analysis (RPCA) models have been proposed to improve the robustness of principle component analysis (PCA). But an important problem that the robustness to outliers affects the discrimination of correct samples has not been solved yet. To solve this problem, we propose a truncated robust principle component analysis (T-RPCA) model which treats correct samples and outliers separately. In fact, the proposed model performs an implicitly truncated weighted learning scheme which is more reasonable for robustness learning respective to previous works. Moreover, we propose a re-weighted (RW) optimization framework to solve a general problem and generalize two sub-frameworks upon it. To be specific, the first sub-framework orients a general truncated loss optimization problem which contains the objective problem of T-RPCA, and the second one focuses on a general singular-value based optimization problem. Besides, we provide rigorously theoretical guarantees for the proposed model, RW framework and sub-frameworks. Empirical studies demonstrate that the proposed T-RPCA model outperforms previous RPCA models on reconstruction and classification tasks.

k-means method using Lloyd heuristic is a traditional clustering method which has played a key role in multiple downstream tasks of machine learning because of its simplicity. However, Lloyd heuristic always finds a bad local minimum, i.e., the bad local minimum makes objective function value not small enough, which limits the performance of k-means. In this paper, we use coordinate descent (CD) method to solve the problem. First, we show that the k-means minimization problem can be reformulated as a trace maximization problem, then a simple and efficient coordinate descent scheme is proposed to solve the maximization problem. Two interesting findings through theory are that Lloyd cannot decrease the objective function value of k-means produced by our CD further, and our proposed method CD to solve k-means problem can avoid produce empty clusters. In addition, according to the computational complexity analysis, it is verified CD has the same time complexity with original k-means method. Extensive experiments including statistical hypothesis testing, on several real-world datasets with varying number of clusters, varying number of samples and varying number of dimensions show that CD performs better compared to Lloyd, i.e., lower objective value, better local minimum and fewer iterations. And CD is more robust to initialization than Lloyd whether the initialization strategy is random or initialization of k-means++.

Linear discriminant analysis (LDA) has been proven to be effective in dimensionality reduction. However, the performance of LDA depends on the consistency assumption of the global structure and the local structure. Some work extended LDA along this line of research and proposed local formulations of LDA. Unfortunately, the learning scheme of these algorithms is suboptimal in that the intrinsic relationship between data points is pre-learned in the original space, which is usually affected by the noise and redundant features. Besides, the time cost is relatively high. To alleviate these drawbacks, we propose a Fast Locality Discriminant Analysis framework (FLDA), which has three advantages: (1) It can divide a non-Gaussian distribution class into many sub-blocks that obey Gaussian distributions by using the anchor-based strategy. (2) It captures the manifold structure of data by learning the fuzzy membership relationship between data points and the corresponding anchor points, which can reduce computation time. (3) The weights between data points and anchor points are adaptively updated in the subspace where the irrelevant information and the noise in high-dimensional space have been effectively suppressed. Extensive experiments on toy data sets, UCI benchmark data sets and imbalanced data sets demonstrate the efficiency and effectiveness of the proposed method.

In this paper, we reveal that metric learning would suffer from serious inseparable problem if without informative sample mining. Since the inseparable samples are often mixed with hard samples, current informative sample mining strategies used to deal with inseparable problem may bring up some side-effects, such as instability of objective function, etc. To alleviate this problem, we propose a novel distance metric learning algorithm, named adaptive neighborhood metric learning (ANML). In ANML, we design two thresholds to adaptively identify the inseparable similar and dissimilar samples in the training procedure, thus inseparable sample removing and metric parameter learning are implemented in the same procedure. Due to the non-continuity of the proposed ANML, we develop an ingenious function, named log-exp mean function to construct a continuous formulation to surrogate it, which can be efficiently solved by the gradient descent method. Similar to Triplet loss, ANML can be used to learn both the linear and deep embeddings. By analyzing the proposed method, we find it has some interesting properties. For example, when ANML is used to learn the linear embedding, current famous metric learning algorithms such as the large margin nearest neighbor (LMNN) and neighbourhood components analysis (NCA) are the special cases of the proposed ANML by setting the parameters different values. When it is used to learn deep features, the state-of-the-art deep metric learning algorithms such as Triplet loss, Lifted structure loss, and Multi-similarity loss become the special cases of ANML. Furthermore, the log-exp mean function proposed in our method gives a new perspective to review the deep metric learning methods such as Prox-NCA and N-pairs loss. At last, promising experimental results demonstrate the effectiveness of the proposed method.

Dimension reduction is a critical technology for high-dimensional data processing, where Linear Discriminant Analysis (LDA) and its variants are effective supervised methods. However, LDA prefers to feature with smaller variance, which causes feature with weak discriminative ability retained. In this paper, we propose a novel Ratio Sum for Linear Discriminant Analysis (RSLDA), which aims at maximizing discriminative ability of each feature in subspace. To be specific, it maximizes the sum of ratio of the between-class distance to the within-class distance in each dimension of subspace. Since the original RSLDA problem is difficult to obtain the closed solution, an equivalent problem is developed which can be solved by an alternative optimization algorithm. For solving the equivalent problem, it is transformed into two sub-problems, one of which can be solved directly, the other is changed into a convex optimization problem, where singular value decomposition is employed instead of matrix inversion. Consequently, performance of algorithm cannot be affected by the non-singularity of covariance matrix. Furthermore, Kernel RSLDA (KRSLDA) is presented to improve the robustness of RSLDA. Additionally, time complexity of RSLDA and KRSLDA are analyzed. Extensive experiments show that RSLDA and KRSLDA outperforms other comparison methods on toy datasets and multiple public datasets.

L,ineaTDiscriminant Analysis (LDA) has been widely used in supervised dimensionality reduction fields. How- ever, LDA is usually weak in tackling data with Non-Gaussian distribution due to its incapability of extracting the intrinsic structure of data. In order to learn the intrinsic information more effectively, some dimensionality reduction methods incorporate the adaptive full-connected graph into the algorithm frame, but the defect is that the calculation of each pairwise distance is very time-consuming. In this paper, we propose a novel fast adaptive local subspace learning with regressive regularization model to solve the supervised dimensional reduction problem. Firstly, the adaptive anchor point graph is used to capture local structure information, which can greatly reduce computation complexity. Secondly, by using regressive regularization, the samples from different classes can be better separated in the projected space and the workload of selecting the optimal reduced dimension is easier. Moreover, entropy regularization is used to derive more appropriate weights. Finally, extensive experiments are conducted on real world data sets to verify the superiority of our model.

Though Electroencephalogram (EEG) could objectively reflect emotional states of our human beings, its weak, non-stationary, and low signal-to-noise properties easily cause the individual differences. To enhance the universality of affective brain-computer interface systems, transfer learning has been widely used to alleviate the data distribution discrepancies among subjects. However, most of existing approaches focused mainly on the domain-invariant feature learning, which is not unified together with the recognition process. In this paper, we propose a joint feature adaptation and graph adaptive label propagation model (JAGP) for cross-subject emotion recognition from EEG signals, which seamlessly unifies the three components of domain-invariant feature learning, emotional state estimation and optimal graph learning together into a single objective. We conduct extensive experiments on two benchmark SEED_IV and SEED_V data sets and the results reveal that 1) the recognition performance is greatly improved, indicating the effectiveness of the triple unification mode; 2) the emotion metric of EEG samples are gradually optimized during model training, showing the necessity of optimal graph learning, and 3) the projection matrix-induced feature importance is obtained based on which the critical frequency bands and brain regions corresponding to subject-invariant features can be automatically identified, demonstrating the superiority of the learned shared subspace.

In domain adaptation (DA), label-induced losses generally occupy a dominant position and most previous models regard hard or soft labels as their inputs. However, these two types of labels may mislead the modeling process of label-induced losses since hard label is sensitive to a wrongly-predicted sample while soft label may introduce label noise, thus they may cause negative transfer. To relieve this problem, we propose a novel label learning approach namely confidence regularized label propagation (CRLP) that regularizes the confidence of predicted soft labels with constraints of F-norm or L21-norm. It is validated that maximizing either one of these two constraints equals to minimizing entropy loss. Specially, we illustrate that L21-norm is more suitable for DA than F-norm when the dataset contain a large number of categories. Then, we leverage the regularized soft labels produced by CRLP to reformulate some popular label-induced losses that consider feature transferability and discriminability such as class-wise maximum mean discrepancy, intra-class compactness and inter-class dispersion in a probability manner to present a novel DA method (i.e., CRLP-DA). Comprehensive analysis and experiments on four cross-domain object recognition datasets verify that the proposed CRLP-DA outperforms some state-of-the-art methods, especially 59.5% for Office10+Caltech10 dataset with SURF features. For others to better reproduce, our preliminary Matlab code will be available athttps://github.com/WWLoveTransfer/CRLP-DA/.

Multi-view clustering has received widespread attention owing to its effectiveness by integrating multi-view data appropriately, but traditional algorithms have limited applicability to large-scale real-world data due to their high computational complexity and low robustness. Focusing on the aforementioned issues, we propose an efficient and robust multi-view clustering algorithm with anchor graph regularization (ERMC-AGR). In this work, a novel anchor graph regularization (ARG) is designed to improve the quality of the learned embedded anchor graph (EAG), and the obtained EAG is decomposed by nonnegative matrix factorization (NMF) under correntropy criterion to acquire clustering results directly. Different from the traditional graph regularization that needs to construct a large-scale Laplacian matrix pertaining to the all-sample graph, our lightweight AGR, constructed from the perspective of anchors, can reduce the computational complexity significantly while improving the EAG quality. Moreover, a factor matrix of NMF is constrained to be the cluster indicator matrix to omit additional k-means after optimization. Subsequently, correntropy is utilized to improve the effectiveness and robustness of ERMC-AGR owing to its promising performance to complex noises and outliers. Extensive experiments on real-world datasets and noisy datasets show that ERMC-ARG can improve the clustering efficiency and robustness while ensuring comparable or even better effectiveness.

Recently, semisupervised feature selection has gained more attention in many real applications due to the high cost of obtaining labeled data. However, existing methods cannot solve the "multimodality" problem that samples in some classes lie in several separate clusters. To solve the multimodality problem, this article proposes a new feature selection method for semisupervised task, namely, semisupervised structured manifold learning (SSML). The new method learns a new structured graph which consists of more clusters than the known classes. Meanwhile, we propose to exploit the submanifold in both labeled data and unlabeled data by consuming the nearest neighbors of each object in both labeled and unlabeled objects. An iterative optimization algorithm is proposed to solve the new model. A series of experiments was conducted on both synthetic and real-world datasets and the experimental results verify the ability of the new method to solve the multimodality problem and its superior performance compared with the state-of-the-art methods.

Linear discriminant analysis (LDA) is a well-known technique for supervised dimensionality reduction and has been extensively applied in many real-world applications. LDA assumes that the samples are Gaussian distributed, and the local data distribution is consistent with the global distribution. However, real-world data seldom satisfy this assumption. To handle the data with complex distributions, some methods emphasize the local geometrical structure and perform discriminant analysis between neighbors. But the neighboring relationship tends to be affected by the noise in the input space. In this research, we propose a new supervised dimensionality reduction method, namely, locality adaptive discriminant analysis (LADA). In order to directly process the data with matrix representation, such as images, the 2-D LADA (2DLADA) is also developed. The proposed methods have the following salient properties: 1) they find the principle projection directions without imposing any assumption on the data distribution; 2) they explore the data relationship in the desired subspace, which contains less noise; and 3) they find the local data relationship automatically without the efforts for tuning parameters. The performance of dimensionality reduction shows the superiorities of the proposed methods over the state of the art.

Matrix completion, in essence, involves recovering a low-rank matrix from a subset of its entries. Most existing methods for matrix completion neglect two significant issues. First, in several practical applications, such as collaborative filtering, some samples may be corrupted completely. However, most of the robust algorithms consider only the condition that a few components of each column have been arbitrarily contaminated. Second, many real data are not static in nature. Nevertheless, the conventional batch-based matrix completion methods cannot efficiently deal with the out-of-sample, that is, the vector completion problem. In this article, we first provide a novel robust matrix completion model and then develop an efficient optimization method that only requires conducting one time singular value decomposition for a thin matrix per iteration. Furthermore, by exploiting the essence of online matrix completion algorithms, we develop a vector completion model which can help users predict the missing values of out of sample. Numerical comparisons with traditional batch-based and online matrix completion algorithms demonstrate the benefits of the proposed method on streaming data corrupted by column outliers. Moreover, we show that our model can be used to detect outliers from incomplete information. This advantage is validated via numerous experimental results on synthetic and real-world data.

In this article, we propose a novel feature selection approach via explicitly addressing the long-standing subspace sparsity issue. Leveraging$\ell _{2,1}$-norm regularization for feature selection is the major strategy in existing methods, which, however, confronts sparsity limitation and parameter-tuning trouble. To circumvent this problem, employing the$\ell _{2,0}$-norm constraint to improve the sparsity of the model has gained more attention recently whereas, optimizing the subspace sparsity constraint is still an unsolved problem, which only can acquire an approximate solution and without convergence proof. To address the above challenges, we innovatively propose a novel subspace sparsity discriminative feature selection (S2DFS) method which leverages a subspace sparsity constraint to avoid tuning parameters. In addition, the trace ratio formulated objective function extremely ensures the discriminability of selected features. Most important, an efficient iterative optimization algorithm is presented to explicitly solve the proposed problem with a closed-form solution and strict convergence proof. To the best of our knowledge, such an optimization algorithm of solving the subspace sparsity issue is first proposed in this article, and a general formulation of the optimization algorithm is provided for improving the extensibility and portability of our method. Extensive experiments conducted on several high-dimensional text and image datasets demonstrate that the proposed method outperforms related state-of-the-art methods in pattern classification and image retrieval tasks.

In big data time, selecting informative features has become an urgent need. However, due to the huge cost of obtaining enough labeled data for supervised tasks, researchers have turned their attention to semisupervised learning, which exploits both labeled and unlabeled data. In this article, we propose a sparse discriminative semisupervised feature selection (SDSSFS) method. In this method, the$\epsilon $-dragging technique for the supervised task is extended to the semisupervised task, which is used to enlarge the distance between classes in order to obtain a discriminative solution. The flexible$\ell _{2,p}$norm is implicitly used as regularization in the new model. Therefore, we can obtain a more sparse solution by setting smaller$p$. An iterative method is proposed to simultaneously learn the regression coefficients and$\epsilon $-dragging matrix and predicting the unknown class labels. Experimental results on ten real-world datasets show the superiority of our proposed method.

These days, clustering is one of the most classical themes to analyze data structures in machine learning and pattern recognition. Recently, the anchor-based graph has been widely adopted to promote the clustering accuracy of plentiful graph-based clustering techniques. In order to achieve more satisfying clustering performance, we propose a novel clustering approach referred to as the progressive self-supervised clustering method with novel category discovery (PSSCNCD), which consists of three separate procedures specifically. First, we propose a new semisupervised framework with novel category discovery to guide label propagation processing, which is reinforced by the parameter-insensitive anchor-based graph obtained from balanced K -means and hierarchical K -means (BKHK). Second, we design a novel representative point selected strategy based on our semisupervised framework to discover each representative point and endow pseudolabel progressively, where every pseudolabel hypothetically corresponds to a real category in each self-supervised label propagation. Third, when sufficient representative points have been found, the labels of all samples will be finally predicted to obtain terminal clustering results. In addition, the experimental results on several toy examples and benchmark data sets comprehensively demonstrate that our method outperforms other clustering approaches.

Fuzzy clustering is one of the most popular clustering approaches and has attracted considerable attention in many fields. However, high computational cost has become a bottleneck which limits its applications in large-scale problems. Moreover, most fuzzy clustering algorithms are sensitive to noise. To address these issues, a novel fuzzy clustering algorithm, called fast fuzzy clustering based on anchor graph (FFCAG), is proposed. The FFCAG algorithm integrates anchor-based similarity graph construction and membership matrix learning into a unified framework, such that the prior knowledge of anchors can be further utilized to improve clustering performance. Specifically, FFCAG first constructs an anchor-based similarity graph with a parameter-free neighbor assignment strategy. Then, it designs a quadratic programming model to learn the membership matrix of anchors, which is very different from traditional fuzzy clustering algorithms. More importantly, a novel balanced regularization term is introduced into the objective function to produce more accurate clustering results. Finally, we adopt an alternating optimization algorithm with guaranteed convergence to solve the proposed method. Experimental results performed on synthetic and real-world datasets demonstrate the proposed FFCAG can significantly reduce the computational time with comparable, even superior, clustering performance, compared with state-of-the-art algorithms.

Fuzzy$c$-means method (FCM) is a popular clustering method, which uses alternating iteration algorithm to update membership matrix$\mathbf {F}$and center matrix$\mathbf {M}$of$d \times c$size. However, original FCM suffers from finding a suboptimal local minimum, which limits the performance of FCM. In this article, we propose a new optimization method for fuzzy$c$-means problem. We first propose an equivalent minimization problem of FCM, then, a simple alternating iteration algorithm is proposed to solve the new minimization problem, which involves an effective and theoretically guaranteed Iteratively Reweighted (IRW) method, so we call the new optimization method IRW-FCM. Our IRW-FCM utilizes$c$not$dc$intermediate variables to update$\mathbf {F}$, which can decrease space complexity. Extensive experiments including objective-function value comparison and clustering-performance comparison show that IRW-FCM can obtain better local minima than FCM with fewer iterations. And according to the time-complexity analysis, it is verified IRW-FCM has the same linear time complexity with FCM. What's more, compared with other clustering methods, IRW-FCM also shows its superiority.

Robust principal component analysis (RPCA) is a technique that aims to make principal component analysis (PCA) robust to noise samples. The current modeling approaches of RPCA were proposed by analyzing the prior distribution of the reconstruction error terms. However, these methods ignore the influence of samples with large reconstruction errors, as well as the valid information of these samples in principal component space, which will degrade the ability of PCA to extract the principal component of data. In order to solve this problem, Fuzzy sparse deviation regularized robust principal component Analysis (FSD-PCA) is proposed in this paper. First, FSD-PCA learns the principal components by minimizing the square of ℓ2-norm-based reconstruction error. Then, FSD-PCA introduces sparse deviation on reconstruction error term to relax the samples with large bias, thus FSD-PCA can process noise and principal components of samples separately as well as improve the ability of FSD-PCA for retaining the principal component information. Finally, FSD-PCA estimates the prior probability of each sample by fuzzy weighting based on the relaxed reconstruction error, which can improve the robustness of the model. The experimental results indicate that the proposed model performs excellent robustness against different types of noise than the state-of-art algorithms, and the sparse deviation term enables FSD-PCA to process noise information and principal component information separately, so FSD-PCA can filter the noise information of an image and restore the corrupted image.

The multiple kernel k -means (MKKM) and its variants utilize complementary information from different sources, achieving better performance than kernel k -means (KKM). However, the optimization procedures of most previous works comprise two stages, learning the continuous relaxation matrix and obtaining the discrete one by extra discretization procedures. Such a two-stage strategy gives rise to a mismatched problem and severe information loss. Even worse, most existing MKKM methods overlook the correlation among prespecified kernels, which leads to the fusion of mutually redundant kernels and bad effects on the diversity of information sources, finally resulting in unsatisfying results. To address these issues, we elaborate a novel Discrete and Parameter-free Multiple Kernel k -means (DPMKKM) model solved by an alternative optimization method, which can directly obtain the cluster assignment results without subsequent discretization procedure. Moreover, DPMKKM can measure the correlation among kernels by implicitly introducing a regularization term, which is able to enhance kernel fusion by reducing redundancy and improving diversity. Noteworthily, the time complexity of optimization algorithm is successfully reduced, through masterly utilizing of coordinate descent technique, which contributes to higher algorithm efficiency and broader applications. What's more, our proposed model is parameter-free avoiding intractable hyperparameter tuning, which makes it feasible in practical applications. Lastly, extensive experiments conducted on a number of real-world datasets illustrated the effectiveness and superiority of the proposed DPMKKM model.

Spectral clustering has been a hot topic in unsupervised learning owing to its remarkable clustering effectiveness and well-defined framework. Despite this, due to its high computation complexity, it is unable of handling large-scale or high-dimensional data, particularly multi-view large-scale data. To address this issue, in this paper, we propose a fast multi-view clustering algorithm with spectral embedding (FMCSE), which speeds up both the spectral embedding and spectral analysis stages of multi-view spectral clustering. Furthermore, unlike conventional spectral clustering, FMCSE can acquire all sample categories directly after optimization without extra k-means, which can significantly enhance efficiency. Moreover, we also provide a fast optimization strategy for solving the FMCSE model, which divides the optimization problem into three decoupled small-scale sub-problems that can be solved in a few iteration steps. Finally, extensive experiments on a variety of real-world datasets (including large-scale and high-dimensional datasets) show that, when compared to other state-of-the-art fast multi-view clustering baselines, FMCSE can maintain comparable or even better clustering effectiveness while significantly improving clustering efficiency.

Recently, many real-world applications exploit multi-view data, which is collected from diverse domains or obtained from various feature extractors and reflect different properties or distributions of the data. In this work, a novel unsupervised multi-view framework is proposed to cluster such data. The proposed method, called Multi-View clustering with Adaptive Sparse Memberships and Weight Allocation (MVASM), pays more attention to constructing a common membership matrix with proper sparseness over different views and learns the centroid matrix and its corresponding weight of each view. Concretely, MVASM method attempts to learn a common and flexible sparse membership matrix to indicate the clustering, which explores the underlying consensus information of multiple views, and solves the multiple centroid matrices and weights to utilize the view-specific information and further modifies the above-mentioned membership matrix. In addition, the theoretical analysis, including the determination of the power exponent parameter, convergence analysis, and complexity analysis are also presented. Compared to the state-of-the-art methods, the proposed method improves the performance of clustering on different public datasets and demonstrates its reasonability and superiority.

In this paper, a novel low-rank structural model is proposed for segmenting data drawn from a high-dimensional space. Our method is based on the fact that all groups clustered from a high-dimensional dataset are distributed in multiple low-rank subspaces. In general, it’s a very difficult task to find the low-rank structures hidden in data. Different from the classical sparse subspace clustering (SSC) and low-rank representation (LRR) which all take two steps including building the affinity matrix and spectral clustering, we introduce a new rank constraint into our model. This constraint allows our model to learn a subspace indicator which can capture different clusters directly from the data without any postprocessing. To further approximate the rank constraint, a piecewise function is utilized as the relaxing item for the proposed model. Besides, under the subspace indicator constraints, the integer programming problem is avoided, which makes our algorithm more efficient and scalable. In addition, we prove the convergence of the proposed algorithm in theory and further discuss the general case in which subspaces don’t pass through the origin. Experiment results on both synthetic and real-world datasets demonstrate that our algorithm significantly outperforms the state-of-the-art methods.

Semi-supervised learning as one of most attractive problems in machine learning research field has aroused broad attentions in recent years. In this paper, we propose a novel locality preserved dimensionality reduction framework, named Semi-supervised Adaptive Local Embedding learning (SALE), which learns a local discriminative embedding by constructing a$k_1$Nearest Neighbors ($k_1$NN) graph on labeled data, so as to explore the intrinsic structure, i.e., sub-manifolds from non-Gaussian labeled data. Then, mapping all samples into learned embedding and constructing another$k_2$NN graph on all embedded data to explore the global structure of all samples. Therefore, the unlabeled data and their corresponding labeled neighbors can be clustered into same sub-manifold, so as to improve the discriminative power of embedded data. Furthermore, we propose two semi-supervised dimensionality reduction methods with orthogonal and whitening constraints based on proposed SALE framework. An efficient alternatively iterative optimization algorithm is developed to solve the NP-hard problem in our models. Extensive experiments conducted on several synthetic and real-world data sets demonstrate the superiorities of our methods on local structure exploration and classification task.

There are many successful spectral based unsupervised dimensionality reduction methods, including Laplacian Eigenmap (LE), Locality Preserving Projection (LPP), Spectral Regression (SR), etc. We find that LPP and SR are equivalent if the symmetric similarity matrix is doubly stochastic, Positive Semi-Definite (PSD) and with rank$p$, where$p$is the reduced dimension. Since solving SR is believed faster than solving LPP based on some related literature, the discovery promotes us to seek to construct such specific similarity matrix to speed up LPP solving procedures. We then propose an unsupervised linear method called Unsupervised Large Graph Embedding (ULGE). ULGE starts with a similar idea as LPP but adopts an efficient approach to construct anchor-based similarity matrix and then performs spectral analysis on it. Moreover, since conventional anchor generation strategies suffer kinds of problems, we propose an efficient and effective anchor generation strategy, called Balanced$K$-means based Hierarchical$K$-means (BHKH). The computational complexity of ULGE can reduce to$O(ndm)$, which is a significant improvement compared to conventional methods need$O(n^2d)$at least, where$n$,$d$and$m$are the number of samples, dimensions, and anchors, respectively. Extensive experiments on several publicly available datasets demonstrate the efficiency and effectiveness of the proposed method.

Fuzzy K-Means (FKM) clustering is of great importance for analyzing unlabeled data. FKM algorithms assign each data point to multiple clusters with some degree of certainty measured by the membership function. In these methods, the fuzzy membership degree matrix is obtained based on the calculation of the distance between data points in the original space. However, this operation may lead to suboptimal results because of the influence of noises and redundant features. Besides, some FKM clustering methods ignore the importance of the weighting exponent. In this paper, we propose a novel FKM method called Fuzzy K-Means Clustering With Discriminative Embedding. Within this method, we simultaneously conduct dimensionality reduction along with fuzzy membership degree learning. To retain most information in the embedding subspace and improve the robustness of this method, principal component analysis is incorporated into our framework. An iterative optimization algorithm is proposed to solve the model. To validate the efficacy of the proposed method, we perform comprehensive analyses, including convergence behavior, parameter determination and computational complexity. Moreover, we also match a appropriate weighting exponent for each data set. Experimental results on benchmark data sets show that the proposed method is more discriminative and effective for clustering tasks.

Feature selection is one of the most important techniques to deal with the high-dimensional data for a variety of machine learning and data mining tasks, such clustering, classification, and retrieval, etc. Fuzziness is a widespread nature of data in nature human society. However, most existing feature selection methods ignore the existence of fuzziness in the data, resulting in sub-optimal feature subsets. To address the problem, we propose a novel unsupervised feature selection method, called Unsupervised Discriminative Projection for Feature Selection (UDPFS) to select discriminative features by conducting fuzziness learning and sparse learning, simultaneously. Specifically, we use projection matrix transform data as its low-dimensional representation, which are partitioned into clusters by using membership matrix with sparse constraint. In addition,$\ell _{2, 1}$-norm regularization is applied to the projection matrix. Then, a discriminative projection matrix with row sparse is obtained by perform fuzziness learning and sparse learning, simultaneously. An effective alternative optimization algorithm is proposed to solve the objective function. Evaluate experimental results on several real-world datasets show the effectiveness and superiority of the proposed unsupervised feature selection method.

Feature selection has attracted a lot of attention in obtaining discriminative and non-redundant features from high-dimension data. Compared with traditional filter and wrapper methods, embedded methods can obtain a more informative feature subset by fully considering the importance of features in the classification tasks. However, the existing embedded methods emphasize the above importance of features and mostly ignore the correlation between the features, which leads to retain the correlated and redundant features with similar scores in the feature subset. To solve the problem, we propose a novel supervised embedded feature selection framework, called feature selection under global redundancy minimization in orthogonal regression (GRMOR). The proposed framework can effectively recognize redundant features from a global view of redundancy among the features. We also incorporate the large margin constraint into GRMOR for robust multi-class classification. Compared with the traditional embedded methods based on least square regression, the proposed framework utilizes orthogonal regression to preserve more discriminative information in the subspace, which can help accurately rank the importance of features in the classification tasks. Experimental results on twelve public datasets demonstrate that the proposed framework can obtain superior classification performance and redundancy removal performance than twelve other feature selection methods.

Inspired by the mean calculation of RPCA_OM and inductiveness of IRPCA, we first propose an inductive robust principal component analysis method with removing the optimal mean automatically, which is shorted as IRPCA_OM. Furthermore, IRPCA_OM is extended to Schatten-$p$norm and a more general framework (i.e., EIRPCA_OM) is presented. The objective function of EIRPCA_OM includes two terms, the first term is a robust reconstruction error term constrained by an$\ell _{2,1}$-norm and the second term is a regularization term constrained by a Schatten-$p$norm. The proposed EIRPCA_OM method is robust, inductive and accurate. However, on the high-dimensional data, it would spend a large computation cost in training stage. To this end, a fast version of EIRPCA_OM called as FEIRPCA_OM is proposed, and its basic idea is to eliminate the zero eigenvalues of data matrix. More importantly, an effective theoretical proof is presented to ensure that FEIRPCA_OM has faster processing speed than EIRPCA_OM when processing high-dimensional data, but without any performance loss. Based on it, we also can exchange the less performance loss for the higher computation efficiency by removing the small eigenvalues of data matrix. Experimental results on the public datasets demonstrate that FEIRPCA_OM works efficiently on the high-dimensional data.

Multi-view semi-supervised learning (SSL) has attracted great attention due to its effectiveness in information utilization of multiple views and labeled and unlabeled data to solve practical problems. However, most existing methods exhibit high computational complexity. Effective integration of the information on different views to achieve enhanced performance remains a challenging task. In this study, we combine an anchor-based approach with multi-view semi-supervised learning to address these problems. A novel multi-view SSL method called fast multi-view SSL (FMSSL) based on learned graph is proposed. Starting from the affinity graphs constructed by using an anchor-based strategy, FMSSL learns an optimal multi-view consensus graph by using feature and label information. The learned graph can jointly consider the relation of multiple views to approximate the manifold structure. The learned graph is then introduced into the SSL model as the weight matrix of a bipartite graph to simultaneously perform separate classification on the original samples and anchors. Accordingly, multi-view SSL can be efficiently performed, and the computational complexity can be significantly reduced. We propose an effective algorithm to optimize the objective function. Extensive experimental results on different real-world datasets demonstrate the effectiveness and efficiency of the proposed algorithm.