Time series analysis plays a vital role in fields such as finance, healthcare, industry, and meteorology, underpinning key tasks including classification, forecasting, and anomaly detection. Although deep learning models have achieved remarkable progress in these areas in recent years, constructing an efficient, multi-task compatible, and generalizable unified framework for time series analysis remains a significant challenge. Existing approaches are often tailored to single tasks or specific data types, making it difficult to simultaneously handle multi-task modeling and effectively integrate information across diverse time series types. Moreover, real-world data are often affected by noise, complex frequency components, and multi-scale dynamic patterns, which further complicate robust feature extraction and analysis. To ameliorate these challenges, we propose FusAD, a unified analysis framework designed for diverse time series tasks. FusAD features an adaptive time-frequency fusion mechanism, integrating both Fourier and Wavelet transforms to efficiently capture global-local and multi-scale dynamic features. With an adaptive denoising mechanism, FusAD automatically senses and filters various types of noise, highlighting crucial sequence variations and enabling robust feature extraction in complex environments. In addition, the framework integrates a general information fusion and decoding structure, combined with masked pre-training, to promote efficient learning and transfer of multi-granularity representations. Extensive experiments demonstrate that FusAD consistently outperforms state-of-the-art models on mainstream time series benchmarks for classification, forecasting, and anomaly detection tasks, while maintaining high efficiency and scalability. Code is available at https://github.com/zhangda1018/FusAD.

Spectral clustering has received widespread attention for its effectiveness in handling nonconvex geometries. The classic methods of spectral clustering include Ratio Cut (Rcut), Normalized Cut (Ncut), and Min-Max Cut (MMcut). Among them, the objective function of MMcut is more reasonable. Unfortunately, existing methods cannot solve MMcut problem without relaxing the discrete or nonnegative constraints. To this end, based on coordinate descent, we propose a basic optimization algorithm to solve MMcut problem without relaxing any constraint. And then, a fast version of the solver is proposed to improve the computational efficiency. Besides, we prove the convergence of the proposed solver, evaluate its computational complexity, and discuss the connection between MMcut and NCut. Finally, extensive experiments are performed to evaluate the effectiveness of our proposed method.

Multi-view clustering (MVC) with bipartite graph has been extensively studied to rapidly handle multi-source heterogeneous information via sparse anchors. However, most existing methods follow a two-stage learning paradigm that first learns continuous label matrix and then discretizes it, not only bringing extra trade-off parameters but yielding suboptimal solutions. Also, numerous methods still exhibit limited scalability for large-scale problems. Thus, this paper proposes two novel models for discrete, trade-off parameter-free and rapid MVC. First, the Bipartite Graph Discrete Reconstruction (BGDR) model uniquely leverages the discrete label matrices of both samples and anchors to dynamically reconstruct a consensus bipartite graph across views. This concise reconstruction style eliminates redundant computations, and anchor labels enable to enrich cluster partition information during reconstruction, enhancing both accuracy and efficiency. The final clustering outcomes are directly acquired via discrete sample labels. Second, to free the optimization time overheads from the limitation of sample size, we further devise the Compact Graph Discrete Reconstruction (CGDR) model, which reconstructs a smaller compact affinity graph among anchors for significant acceleration. Original sample labels are then gained by label propagation. Systematic experiments illuminate that both models reach superior outcomes in term of efficacy and efficiency.

We propose the DPSM method, a density-based node clustering approach that automatically determines the number of clusters and can be applied in both data space and graph space. Unlike traditional density-based clustering methods, which necessitate calculating the distance between any two nodes, our proposed technique determines density through a propagation process, thereby making it suitable for a graph space. In DPSM, nodes are partitioned into small clusters based on propagated density. The partitioning technique has been proved to be sound and complete. We then extend the concept of spectral clustering from individual nodes to these small clusters, while introducing the CluCut measure to guide cluster merging. This measure is modified in various ways to account for cluster properties, thus provides guidance on when to terminate the merging process. Various experiments have validated the effectiveness of DPSM and the accuracy of these conclusions.

K-means algorithm divides samples into c classes based on their structural characteristics. However, due to the non convex nature of the clustering problem, algorithms are prone to converge to poor local minima. To address the aforementioned issues, we propose the Local Structure Preserving Clustering with Learnable Anchors (LSPC-LA) method. We assume that with a well-designed anchor selection strategy, samples near the same anchor tend to belong to the same cluster, which reveal high confidence Must-Link local structural information for clustering. Based on this observation, we first construct an anchor-based bipartite graph, transforming the sample clustering problem into anchor clustering problem by local structural information, thus reducing the solution space and minimizing the risk of poor local minima. Then we create an anchor guiding matrix to allow anchors to learn the sample structure, improving clustering performance. Subsequently, an alternating iterative algorithm is proposed to optimize the LSPC-LA model. Finally, extensive experiments demonstrate the accuracy of the local structural information and the effectiveness of LSPC-LA.

Masked autoencoders have been extensively utilized in 3D point cloud self-supervised learning, where the fundamental approach involves masking a portion of the point cloud and subsequently reconstructing it. This process is hypothesized to enhance model learning by leveraging the inherent structure of the point cloud data. However, the information density within point clouds is inherently uneven, contrasting with the more uniform distributions found in language and 2D image data. This uneven distribution suggests that the application of random masking strategies, commonly adopted from NLP and 2D vision, may not be optimal for point cloud data, potentially leading to suboptimal learning outcomes. Based on this observation, we propose a simple yet effective Density-directed Masked Autoencoders for Point Cloud Self-supervised Learning (Point-DMAE), which learns latent semantic point cloud features using a density-directed masking strategy. Specifically, our method employs a dual-branch Transformer architecture to extract both high-level and fine-grained point features through global and local block density-directed masking, respectively. Point-DMAE demonstrates high pre-training efficiency and significantly outperforms our baseline (Point-MAE) on 3D object classification tasks within the ScanObjectNN dataset by 4.13% on OBJ-BG, 5.17% on OBJ-ONLY, and 4.17% on PB-T50-RS. Codes are available at https://github.com/jinxianglong10/Point-DMAE.

Generalized Category Discovery (GCD) is a recently proposed open-world problem that aims to automatically classify and discover new categories based on partially labeled data. For unlabeled data, previous research commonly considers using pseudo-labels to assist in model learning. These pseudo-labels, together with the true labels of labeled data, form the learning targets for the final classifier, leading to better predictive outcomes. However, low-quality labels can inevitably hinder the learning process of the model. To address this issue, inspired by previous methods, we propose the Calibrated Generalized Category Discovery (CGCD) framework, which incorporates a projection head, a classifier head, and a calibration head. The projection head is used for representation learning. The calibration head learns high-quality labels from the robust predictions of the classifier head, and the classifier head utilizes these high-quality labels for more efficient learning. Both heads mutually enhance each other during training, ultimately leading to a superior solution. In addition, leveraging the characteristics of both the classifier head and calibration head, we designed a classifier representation distribution regularization term to further ensure consistency in their learning processes. Extensive experimental results demonstrate that the proposed CGCD framework achieves state-of-the-art performance across five general and fine-grained visual recognition datasets by leveraging high-quality label learning.

Principal Component Analysis (PCA) is one of the most famous unsupervised dimensionality reduction algorithms and has been widely used in many fields. However, it is very sensitive to outliers, which reduces the robustness of the algorithm. In recent years, many studies have tried to employ \(\ell_{1}\) -norm to improve the robustness of PCA, but they all lack rotation invariance or the solution is expensive. In this article, we propose a novel robust PCA, namely, Fuzzy Weighted Principal Component Analysis (FWPCA), which still uses squared \(\ell_{2}\) -norm to minimize reconstruction error and maintains rotation invariance of PCA. The biggest bright spot is that the contribution of data is restricted by fuzzy weights, so that the contribution of normal samples is much greater than noise or abnormal data, and realizes anomaly detection. Besides, a more reasonable data center can be obtained by solving the optimal mean to make projection matrix more accurate. Subsequently, an effective iterative optimization algorithm is developed to solve this problem, and its convergence is strictly proved. Extensive experimental results on face datasets and RGB anomaly detection datasets show the superiority of our proposed method.

Effectively and efficiently mining valuable clustering patterns is a challenging problem when handling large-scale data from diverse sources. Existing approaches adopt anchor graph learning or binary representation embedding to reduce computational complexity. Normally, anchor graph learning can not directly obtain the clustering assignment except adopt the post-processing stage, such as graph cut or k-means clustering. The binary representation embedding neglects the structure information in Hamming space. In order to overcome these limitations, this paper proposes a novel, effective, and efficient angular reconstructive discrete embedding method with fusion similarity for a multi-view clustering (AFMC) that can jointly learn the global and local structure preserving binary representation and clustering assignment. Specifically, we propose to use angular reconstructive error minimization to maintain the global similarity correlation of binary representations of heterogeneous features in a common Hamming space. Moreover, we design a multi-view discrete ridge regression with fusion similarity term to handle the out-of-sample problem and preserve the local manifold structure. In addition, we propose an efficient optimization algorithm with linear computational complexity to solve the non-convex and non-smooth objective function. The experimental results demonstrate that AFMC outperforms several state-of-the-art large-scale multi-view clustering methods.

Graph clustering has become a crucial technique for uncovering community structures in complex network data. However, existing approaches often introduce cumbersome regularization or constraints (hyperparameter tuning burden) to obtain balanced clustering results, thereby increasing hyperparameter tuning requirements and intermediate variables. These limitations can lead to suboptimal performance, particularly in scenarios involving imbalanced clusters or large-scale datasets. Besides, most graph cut clustering methods solve two separate discrete problems, resulting in information loss and relying on time-consuming eigen-decomposition. To address these challenges, this paper propose an effective graph cut framework, termed Harmonic MaxMin Cut (HMMC), inspired by worst-case objective optimization and the harmonic mean. Unlike traditional spectral clustering, HMMC produces all cluster assignments in a single step, eliminating the need for additional discretization and notably enhancing robustness to “worst-case cluster” boundaries. this paper further devise a fast coordinate descent (CD) solver that scales linearly complexity with the graph size, offering a computationally efficient alternative to eigen decomposition. Extensive experiments on real-world datasets demonstrate that HMMC is comparable to, or even surpasses, state-of-the-art methods, while also finding more favorable local solutions than non-negative matrix factorization techniques.

As a compromise between supervised and unsupervised learning, semi-supervised learning (SSL) harnesses both labeled and unlabeled data to enhance learning performance. Graph-based semi-supervised learning (GSSL) has emerged as a prominent approach owing to its versatility in representing sample interdependencies via graph structures. However, traditional GSSL methods face high time cost when computing matrix inverses, making them inefficient for large datasets. To address this, some researchers have introduced anchors as a bridge to accelerate the process. Nevertheless, most anchor-based models suffer from one or more of the following issues: (1) The anchor graph-based construction of the adjacency matrix has limitations; (2) The objective functions are typically non-convex, leading to local optima and requiring multiple runs to achieve good performance. To tackle these challenges, we develop a probability-driven approach to build the adjacency matrix, defining sample similarity as the probability of sharing the same anchor. Based on this strategy, we design a model (CFSL) with a strictly convex objective function, guaranteeing a globally optimal solution without iterative optimization. Experiments on multiple datasets indicate that our algorithm yields strong performance.

Traditional spectral clustering methods struggle with scalability and robustness in large datasets due to their reliance on similarity matrices and EigenValue Decomposition. We introduce two innovative models: Rcut-based Coordinate Descent Clustering (R-CDC) and Ncut-based Doubly Stochastic Clustering (N-DSC). These models integrate graph construction and segmentation into a unified process optimized through the coordinate descent method, significantly enhancing clustering efficacy. A novel graph structure enhances robustness against noise and outliers, simplifying the clustering process and improving outcomes across diverse datasets. Our extensive experiments show that these models surpass existing spectral clustering techniques in managing large-scale data and complex structures.

This paper proposes a deep pseudo-label method for unsupervised feature selection, which learns non-linear representations to generate pseudo-labels and trains a Neural Network (NN) to select informative features via self-Knowledge Distillation (KD). Specifically, the proposed method divides a standard NN into two sub-components: an encoder and a predictor, and introduces a dependency subnet. It works by self-supervised pretraining the encoder to produce informative representations and then alternating between two steps: (1) learning pseudo-labels by combining the clustering results of the encoder's outputs with the NN's prediction outputs, and (2) updating the NN's parameters by globally selecting a subset of features to predict the pseudo-labels while updating the subnet's parameters through self-KD. Self-KD is achieved by encouraging the subnet to locally capture a subset of the NN features to produce class probabilities that match those produced by the NN. This allows the model to self-absorb the learned inter-class knowledge and evaluate feature diversity, removing redundant features without sacrificing performance. Meanwhile, the potential discriminative capability of a NN can also be self-excavated without the assistance of other NNs. The two alternate steps reinforce each other: in step (2), by predicting the learned pseudo-labels and conducting selfKD, the discrimination of the outputs of both the NN and the encoder is gradually enhanced, while the self-labeling method in step (1) leverages these two improvements to further refine the pseudo-labels for step (2), resulting in the superior performance. Extensive experiments show the proposed method significantly outperforms state-of-the-art methods across various datasets.

Graph-based clustering technique has garnered significant attention due to precise information characterization by pairwise graph similarity. Nevertheless, the post-processing step in traditional methods often limits clustering effects because of crucial information loss. Therefore, the Constrained Laplacian Rank (CLR) theory emerges to directly obtain discrete labels from optimally structural graph, achieving desirable outcomes. However, CLR suffers from substantial time overhead, making it infeasible for large-scale data analysis. To overcome this issue, we propose Anchor-based CLR (ACLR), a simple yet effective method for efficient large-scale clustering. The ACLR method comprises four stages: (1) anchors that roughly cover original data are opted to prepare bipartite graph construction; (2) a novel two-step probability transition (TSPT) strategy initializes a small-scale graph with random walk probability among anchors; (3) the main ACLR model alternately optimizes the graph connected structure and directly produces discrete anchor labels, achieving a time complexity independent of the number of samples due to dramatically reduced graph scale; and (4) labels are propagated from anchors to samples using$K$-NN algorithm. Extensive experiments demonstrate that ACLR yields superior accuracy and efficiency, particularly when applied to large-scale data. The codes are available athttps://github.com/MarathonZhenyuMa/2025-TKDE-ACLR.

Fuzzy c-means based on entropy regularization (FCER) is a commonly used machine learning algorithm that uses maximum entropy as the regularization term to realize fuzzy clustering. However, this model has many constraints and is challenging to optimize directly. During the solution process, the membership matrix and cluster centers are alternately optimized, easily converging to poor local solutions, limiting the clustering performance. In this paper, we start with the optimization model and propose an unconstrained fuzzy clustering model (UFCER) equivalent to FCER, which reduces the size of optimization variables from$(n+d)\times c$to$d\times c$. More importantly, there is no need to calculate the membership matrix during the optimization process iteratively. The time complexity is only linear, and the convergence speed is fast. We conduct extensive experiments on real datasets. The comparison of objective function value and clustering performance fully demonstrates that under the same initialization, our proposed algorithm can converge to smaller local minimums and get better clustering performance.

Most existing multi-view graph clustering models focus on integrating the topological structure of different views directly, which cannot efficiently stimulate the collaboration between multiple views. To alleviate this problem, this paper proposes a Triangle Topology Enhancement (T2E) module, which expands two topological structures based on the raw topology of each view, including the self-triangle enhanced topology that highlights the local view information and the cross-view triangle enhanced topology containing the global-local view information. Afterward, this paper designs a novel multi-view graph clustering model, named MGC-T2E, to integrate both the raw and derived topological structures and directly induce consistent clustering indicators based on a self-supervised clustering module. In the simulation, the experimental results demonstrate that MGC-T2E achieves state-of-the-art performances compared with a mass of current competitors.

Anchor-based clustering methods have attracted increasing attention due to their ability to provide efficient and scalable solutions in clustering tasks, such as subspace, multi-view and ensemble clustering. Nevertheless, the majority of anchor-based methods view anchors merely as tools, concentrating on diminishing computational complexity within original data space. However, in fact, clustering can be directly performed on anchors and then the anchor clustering results could be propagated to original data. Due to the much smaller volume of anchors, this could significantly reduce the computational complexity of clustering algorithms. Building upon this idea, in this paper, we propose a fast anchor graph clustering method (FAGC) via maximizing within-cluster similarity. Inspired by the relaxation and discretization model in spectral clustering, we also propose two corresponding models, namely FAGC-R and FAGC-D. FAGC-R first obtains spectral embedding of anchors and then discretizes the embedding to obtain anchor indicator matrix. While FAGC-D directly solves the discrete anchor membership matrix. Once anchor clustering results are obtained, original data labels can be obtained through anchor label transmission. Extensive experiments conducted on synthetic and real datasets illustrate the effectiveness and efficiency of the proposed methods.

Multi-view spectral clustering has attracted considerable attention since it can explore common geometric structures from diverse views. Nevertheless, existing min-min framework-based models adopt internal minimization to find the view combination with the minimized within-cluster variance, which will lead to effectiveness loss since the real clusters often exhibit high within-cluster variance. To address this issue, we provide a novel scalable min-max multi-view spectral clustering (SMMSC) model to improve clustering performance. Besides, anchor graphs, rather than full sample graphs, are utilized to reduce the computational complexity of graph construction and singular value decomposition, thereby enhancing the applicability of SMMSC to large-scale applications. Then, we rewrite the min-max model as a minimized optimal value function, demonstrate its differentiability, and develop an efficient gradient descent-based algorithm to optimize it with linear computational complexity. Moreover, we demonstrate that the resultant solution of the proposed algorithm is the global optimum. Numerous experiments on different real-world datasets, including some large-scale datasets, demonstrate that SMMSC outperforms existing state-of-the-art multi-view clustering methods regarding clustering performance.

In practical applications, multi-view subspace clustering is hindered by data noise that disrupts the ideal block-diagonal structure of self-representation matrices, thereby degrading performance. Moreover, many existing methods rely solely on sample features, overlooking the valuable structural information in affinity matrices (e.g., pairwise relationships). While conventional contrastive learning strategies often introduce false negative pairs due to noise and unreliable sample selection. To address these challenges, we propose a pseudo-label guided bidirectional discriminative deep multi-view subspace clustering method (PBDMSC). Our approach first employs pseudo-label guided contrastive learning, using previous cluster assignments to select reliable positive and negative samples, which mitigates incorrect pairings and enhances low-dimensional representations. Then, a discriminative self-representation learning method is introduced that leverages pseudo-labels to enforce homogeneous expression constraints and incorporates a bidirectional attention mechanism to preserve the structured information from affinity matrices, thereby enhancing robustness. Experimental results on six real-world datasets demonstrate that our proposed method achieves state-of-the-art clustering performance.

Unsupervised domain adaptation aims to classify unlabeled data points in the target domain using labeled data points from the source domain, while the distributions of data points in two domains are different. To address this issue, we propose a novel method called the anchor guided unsupervised domain adaptation method (AGDA). We minimize distribution divergence in a latent feature subspace using the Maximum Mean Discrepancy (MMD) criterion. Unlike existing unsupervised domain adaptation methods, we introduce anchor points in the original space and impose domains data to the same anchor points rather than center points to further reduce the domain difference. We optimize the anchor-based graph in the subspace to obtain discriminative transformation matrices. This enables our model to perform better on non-Gaussian distribution than methods focusing on global structure. Furthermore, the sparse anchor-based graph reduces time complexity compared to the fully connected graph, enabling exploration of local structure. Experimental results demonstrate that our algorithm outperforms several state-of-the-art methods on various benchmark datasets.

Structured proximity matrix learning, one of the mainstream directions in clustering research, refers to learning a proximity matrix with an explicit clustering structure from the original first-order proximity matrix. Due to the complexity of the data structure, the original first-order proximity matrix always lacks some must-links compared to the groundtruth proximity matrix. It is worth noting that high-order proximity matrices can provide missed must-link information. However, the computation of high-order proximity matrices and clustering based on them are expensive. To solve the above problem, inspired by the anchor bipartite graph, we present a novel high-order bipartite graph proximity matrix and a fast method to compute it. This proposed high-order bipartite graph proximity matrix contains high-order proximity information and can significantly reduce the computational complexity of the whole clustering process. Furthermore, we introduce an efficient and simple high-order bipartite graph fusion framework that can adaptively assign weights to each order of the high-order bipartite graph matrices. Finally, under the Laplace rank constraint, a consensus structured bipartite graph proximity matrix is obtained. At the same time, an efficient solution algorithm is proposed for this model. The model's efficacy is underscored through rigorous experiments, highlighting its superior clustering performance and time efficiency. Code available:https://anonymous.4open.science/r/HBGC-F6C4.

Ensemble clustering can utilize the complementary information among multiple base clusterings, and obtain a clustering model with better performance and more robustness. Despite its great success, there are still two problems in the current ensemble clustering methods. First, most ensemble clustering methods often treat all base clusterings equally. Second, the final ensemble clustering result often relies on$k$-means or other discretization procedures to uncover the clustering indicators, thus obtaining unsatisfactory results. To address these issues, we proposed a novel ensemble clustering method based on structured graph learning, which can directly extract clustering indicators from the obtained similarity matrix. Moreover, our methods take sufficient consideration of correlation among the base clusterings and can effectively reduce the redundancy among them. Extensive experiments on artificial and real-world datasets demonstrate the efficiency and effectiveness of our methods.

Graph Neural Networks (GNNs) have emerged as the predominant approach for analyzing graph data on the web and beyond. Contrastive learning (CL), a self-supervised paradigm, not only mitigates reliance on annotations but also has potential in performance. The hard negative sampling strategy that benefits CL in other domains proves ineffective in the context of Graph Contrastive Learning (GCL) due to the message passing mechanism. Embracing the subspace hypothesis in clustering, we propose a method towards expansive and adaptive hard negative mining, referred to as G raph contR astive leA rning via subsP ace prE serving (GRAPE ). Beyond homophily, we argue that false negatives are prevalent over an expansive range and exploring them confers benefits upon GCL. Diverging from existing neighbor-based methods, our method seeks to mine long-range hard negatives throughout subspace, where message passing is conceived as interactions between subspaces. %Empirical investigations back up this strategy. Additionally, our method adaptively scales the hard negatives set through subspace preservation during training. In practice, we develop two schemes to enhance GCL that are pluggable into existing GCL frameworks. The underlying mechanisms are analyzed and the connections to related methods are investigated. Comprehensive experiments demonstrate that our method outperforms across diverse graph datasets and remains competitive across varied application scenarios\footnoteOur code is available at https://github.com/zz-haooo/WWW24-GRAPE. .

Principal component analysis (PCA) is one of the most versatile techniques for unsupervised dimension reduction, which is implemented as a fundamental preprocessing method in multiple tasks of statistics and machine learning research because of its efficiency. Nevertheless, researchers have concentrated on the identification of outliers that do not conform to the low-dimensional approximation through statistical methods, e.g., outlier rejection, without giving insights on each data point with a dynamic ratio of signal-to-noise components in the high-dimensional regimes. To characterize the dynamic nature of the principal component information, we propose a Normalized Robust PCA with Adaptive Reconstruction Error minimization model, which considers both the adaptive normalization technique and flexible weights learning simultaneously. With this configuration, the principal component information constantly adjusts the degree of sparsity for activated samples. In other words, the signal component's discrimination and noise information restriction could work cooperatively. Empirical studies on one synthetic dataset and several benchmarks demonstrate the effectiveness of our proposed method over existing outlier rejection methods.

Ensemble clustering has emerged as a powerful framework for analyzing heterogeneous and complex data. Despite the abundance of existing schemes, co-association matrix-based methods remain the mainstream approach. However, focusing solely on pairwise correlations falls short of fully capturing the intricate cluster relationships. Moreover, despite its potential, ensemble clustering has yet to effectively leverage the powerful representation capabilities of neural networks. To address these limitations, we propose a deep ensemble clustering method called Ensemble Clustering with Attentional Representation (ECAR). Our method considers the results of base partition as groups with related information to explore higher-order fusion information. ECAR captures the importance of each sample's association with its related group by employing an attentional network, and encodes this information into a low-dimensional representation. The attentional network is trained by jointly optimizing the clustering loss from soft assignments learned from the embeddings and the reconstruction loss from the weighted graph generated from ensemble clustering. During training, the weights of base partitions are adaptively refined to promote diversity and consistency while reducing the impact of low-quality and redundant base partitions. Extensive experimental results on real-world datasets demonstrate the substantial improvement of our method over existing baseline ensemble clustering methods and deep clustering methods.

Fuzzy$k$-Means with Entropy Regularization method (ERFKM) is an extension to Fuzzy$k$-Means (FKM) by introducing a maximum entropy term to FKM, whose purpose is trading off fuzziness and compactness. However, ERFKM often converges to a poor local minimum, which affects its performance. In this paper, we propose an effective optimization method to solve this problem, called IRW-ERFKM. First a new equivalent problem for ERFKM is proposed; then we solve it through Iteratively Re-Weighted (IRW) method. Since IRW-ERFKM optimizes the problem with$k\times 1$instead of$d\times k$intermediate variables, the space complexity of IRW-ERFKM is greatly reduced. Extensive experiments on clustering performance and objective function value show IRW-ERFKM can get a better local minimum than ERFKM with fewer iterations. Through time complexity analysis, it verifies IRW-ERFKM and ERFKM have the same linear time complexity. Moreover, IRW-ERFKM has advantages on evaluation metrics compared with other methods. What's more, there are two interesting findings. One is when we use IRW method to solve the equivalent problem of ERFKM with one factor$\mathbf{U}$, it is equivalent to ERFKM. The other is when the inner loop of IRW-ERFKM is executed only once, IRW-ERFKM and ERFKM are equivalent in this case.

The key challenge of multi-view graph-based clustering is to mine consistent clustering structures from multiple graphs. Existing works seek clustering decisions from either multiple spectral embeddings or multiple affinity matrices, ignoring the interactions among them. To address this problem, we propose a Bidirectional Attentive Multi-view Clustering (BAMC) model to explore a consensus space w.r.t.spectral embedding and affinity matrix simultaneously, where they can promote each other to mine richer structural information from multiple graphs. BAMC is composed of a Spectral Embedding Learning (SEL) module, an Affinity Matrix Learning (AML) module, and a Bidirectional Attentive Clustering (BAC) module. SEL seeks consensus spectral embeddings by aligning the distributions of elements sampled from subspaces spanned by multiple spectral embeddings. AML learns a consensus affinity matrix from input affinity matrices. BAC guarantees consistency between the learned consensus spectral embeddings and the affinity matrix. To balance their effects, it also assigns adaptive weights to SEL and AML's objective functions. To solve the optimization problem involved in BAMC, we propose an efficient algorithm based on the Majority-Minimization framework with an ingenious surrogate problem. Extensive experiments on several synthetic and real-world datasets demonstrate the superb performance of BAMC.

Multimodal aspect-level sentiment analysis has attracted increasing attention in recent years. However, existing methods have two unaddressed limitations: (1) due to the lack of labelled pre-training data of dedicated sentiment analysis, the methods with a pre-training manner produce suboptimal prediction results; (2) most existing methods employ a self-attention encoder to fuse multimodal tokens, which not only ignores the alignment relationship between different modal tokens but also makes the model unable to capture the semantic links between images and texts. In this paper, we propose a momentum contrastive learning network (MOCOLNet) to overcome above limitations. First, we merge the pre-training stage with the training stage to design an end-to-end training manner which uses less labelled data dedicated to sentiment analysis to obtain better prediction results. Second, we propose a multimodal contrastive learning method to align the different modal representations before data fusing, and design a cross-modal matching strategy to provide semantic interactive information between texts and images. Moreover, we introduce an auxiliary momentum strategy to increase the robustness of model. We also analyse the effectiveness of the proposed multimodal contrastive learning method using a mutual information theory. Experiments verify that the proposed MOCOLNet is superior to other strong baselines.

In this paper, we propose a novel algorithm to solve the row-sparse principal component analysis problem without relying on any data structure assumption. Sparse principal component analysis was proposed to improve the interpretability of principal component analysis by restricting the number of non-zero elements in each loading vector, but the varying sparsity patterns among different leading vectors may result in trouble on some occasions, such as feature selection. Then row-sparse principal component analysis was proposed, which demands the same sparse pattern among different loading vectors. However, the optimization of row-sparse principal component analysis problems is NP-hard. Although some algorithms were proposed to solve this problem, but they are only applicable to the specific data structure. In this paper, we transform the original row-sparse principal component analysis problem into a new equivalent problem that can be solved by coordinate descent method without relying on any data structure assumption. Then by carefully eliminating redundant structures to avoid repeating computation, we propose a more efficient coordinate descent method to solve this problem. Furthermore, no parameter needs to be tuned in our algorithm. Finally, extensive experiments are conducted on the real world data sets to demonstrate the superiority of our algorithm.

Feature selection is one of the important topics of machine learning, and it has a wide range of applications in data preprocessing. At present, feature selection based on$\ell _{2,1}$-norm regularization is a relatively mature method, but it is not enough to maximize the sparsity and parameter-tuning leads to increased costs. Later scholars found that the$\ell _{2,0}$-norm constraint is more conductive to feature selection, but it is difficult to solve and lacks convergence guarantees. To address these problems, we creatively propose a novel Outliers Robust Unsupervised Feature Selection for structured sparse subspace (ORUFS), which utilizes$\ell _{2,0}$-norm constraint to learn a structured sparse subspace and avoid tuning the regularization parameter. Moreover, by adding binary weights, outliers are directly eliminated and the robustness of model is improved. More importantly, a Re-Weighted (RW) algorithm is exploited to solve our$\ell _{p}$-norm problem. For the NP-hard problem of$\ell _{2,0}$-norm constraint, we develop an effective iterative optimization algorithm with strict convergence guarantees and closed-form solution. Subsequently, we provide theoretical analysis about convergence and computational complexity. Experimental results on real-world datasets illustrate that our method is superior to the state-of-the-art methods in clustering and anomaly detection tasks.

Most existing multi-view graph clustering models either seek consistent clustering results from similarity matrices and spectral embeddings respectively or follow direct bidirectional integration of them, which ignores the interaction between them. To make up for this flaw, this paper designs a novel multi-view clustering model that performsBidirectionalFusion withCross-viewGraphFilter (BF-CGF). To be specific, BF-CGF first learns a consistent graph embedding via performing the interaction between multi-view graphs and spectral embeddings with the perspective of the graph spectral domain and then considers seeking a consistent indicator matrix via the graph cut model from the consistent graph embedding and the similarity matrices. To solve the optimization problem of BF-CGF, we propose an efficient iterative algorithm and provide the corresponding convergence and complexity analyses. Extensive experimental results demonstrate that the proposed BF-CGF outperforms state-of-the-art competitors in most benchmark datasets.

Feature selection has achieved unprecedented success in obtaining sparse discriminative features. However, the existing methods almost use the$\ell _{2,p}$-norm constraint on transformation matrix to obtain sparse features, which introduces extra parameters and cannot obtain the features directly. In addition, existing algorithms only focused on the global structure and ignored the local structure, leading to poor performance when solving data with non-Gaussian distributions which a single center point cannot describe precisely. Based on above considerations, we propose a supervised feature selection via multi-center and local structure learning. We further introduce trace ratio criterion into our model in favor of improving the discriminant of features selected. In order to address the overlap problem, we use multiple center points to match the distribution of data and construct a$k$-Nearest Neighbor graph to explore the local structure of the data. In addition, we also propose an efficient method to optimize the transformation matrix with the$\ell _{2,0}$-norm constraint and can directly obtain the sparse features. We evaluate our method on Toy datasets and several real-world datasets, show improvement over state-of-the-art feature selection methods, and demonstrate the effectiveness of our model in dealing with non-Gaussian distributed data problems.

In recent years, graph-based multiview clustering methods have become a research hotspot in the clustering field. However, most existing methods lack consideration of cluster balance in their results. In fact, cluster balance is crucial in many real-world scenarios. Additionally, graph-based multiview clustering methods often suffer from high time consumption and cannot handle large-scale datasets. To address these issues, this paper proposes a novel graph-based multiview clustering method. The method is built upon the bipartite graph. Specifically, it employs a label propagation mechanism to update the smaller anchor label matrix rather than the sample label matrix, significantly reducing the computational cost. The introduced balance constraint in the proposed model contributes to achieving balanced clustering results. The entire clustering model combines information from multiple views through graph fusion. The joint graph and view weight parameters in the model are obtained through task-driven self-supervised learning. Moreover, the model can directly obtain clustering results without the need for the two-stage processing typically used in general spectral clustering. Finally, extensive experiments on toy datasets and real-world datasets are conducted to validate the superiority of the proposed method in terms of clustering performance, clustering balance, and time expenditure.

Since obtaining data labels is a time-consuming and laborious task, unsupervised feature selection has become a popular feature selection technique. However, the current unsupervised feature selection methods are facing three challenges: (1) they rely on a fixed similarity matrix derived from the original data, which will affect their performance; (2) due to the limitation of sparsity, they can only obtain sub-optimal solutions; (3) they have high computational complexity and cannot handle large-scale data. To solve this dilemma, we propose a fast unsupervised feature selection algorithm with bipartite graph and 2;0-norm constraint (BGCFS). We use the original data and the selected anchors to construct an adaptive bipartite graph in the subspace, and apply the l2,0-norm constraint to the projection matrix for feature selection. In this way, we can update the adaptive bipartite graph and the projection matrix simultaneously, and we can get the feature subset directly, without sorting the features. In addition, we propose an iterative algorithm that can solve the proposed problem globally to obtain a closed-form solution, and we provide a strict proof of convergence for it. Experiments on eight real data sets with different scales show that our method can select more valuable feature subsets more quickly

Recently, several robust principal component analysis (RPCA) models were presented to enhance the robustness of PCA by exploiting the robust norms as their loss functions. But an important problem is that they have no ability to discriminate outliers from correct samples. To solve this problem, we propose a RPCA method based on discriminant information (RPCA-DI). RPCA-DI disentangles the robust PCA with a two-step fashion: the identification and the processing of outliers. To identity outliers, a sample representation model based on entropy regularization is constructed to analyze the membership of data belonging to the principal component space(PC) and its orthogonal complement(OC), the discriminative information of data will be extracted based on measuring the differences of retained information on PC(or OC) of data. By this way, we can discriminate correct samples when we deal with outliers, which is more reasonable for robustness learning respective to previous works. In the noise processing step, in addition to considering the levels of noise, the resistance of the sample points to noise is also considered to prevent overfitting, thereby improving the generalization performance of RPCA-DI. Finally, an iterative algorithm is designed to solve the corresponding model. Compared with some state-of-art RPCA methods on artificial datasets, UCI datasets and face databases that verifies the effectiveness of RPCA-DI.

Fuzzy clustering algorithms have been widely used to reveal the possible hidden structure of data. However, with the increasing of data amount, large scale data has brought genuine challenges for fuzzy clustering. Most fuzzy clustering algorithms suffer from the long time-consumption problem since a large amount of distance calculations are involved to update the solution per iteration. To address this problem, we introduce the popular anchor graph technique into fuzzy clustering and propose a scalable fuzzy clustering algorithm referred to as Scalable Fuzzy Clustering with Anchor Graph (SFCAG). The main characteristic of SFCAG is that it addresses the scalability issue plaguing fuzzy clustering from two perspectives: anchor graph construction and membership matrix learning. Specifically, we select a small number of anchors and construct a sparse anchor graph, which is beneficial to reduce the computational complexity. We then formulate a trace ratio model, which is parameter-free, to learn the membership matrix of anchors to speed up the clustering procedure. In addition, the proposed method enjoys linear time complexity with the data size. Extensive experiments performed on both synthetic and real world datasets demonstrate the superiority (both effectiveness and scalability) of the proposed method over some representative large scale clustering methods.

Among a big body of recently developed algorithms for machine learning and data mining, a class of models using non-convex/non-smooth sparsity-inducing norms achieves promising results on many challenging tasks. An important problem faced with such models is to find an effective solution for the objective function with one or multiple intractable terms. Although a large number of optimization approaches have been developed, most of them are tailored to a specific model. Besides, these approaches generally introduce some additional parameters and no longer guarantee convergence. In this work, we first revisit some representative non-convex/non-smooth machine learning models, and then unity them into a generic formulation. Theoretically, we develop a simple yet efficient optimization framework, namely Iteratively Re-Weighted method (IRW), to solve such a class of models and provide the corresponding convergence analysis. Particularly, we validate our proposed method on two challenging machine learning tasks: multi-task regression and feature selection. Source codes are available at:https://github.com/KDD-Code/Sparse.git.

K-means is one of the most simple and popular clustering algorithms, which implemented as a standard clustering method in most of machine learning researches. The goal of K-means clustering is finding a set of cluster centers and minimizing the sum of squared distances between each sample and its nearest clustering center. In this paper, we proposed a novel K-means clustering algorithm, which reformulate the classical K-Means objective function as a trace maximization problem and then replace it with a new formulation. The proposed algorithm does not need to calculate the cluster centers in each iteration and requires fewer additional intermediate variables during the optimization process. In addition, we proposed an efficient iterative re-weighted algorithm to solve the involved optimization problem and provided the corresponding convergence analysis. The proposed algorithm keeps a consistent computational complexity as Lloyd's algorithm,$\mathcal {O}(ndk)$, but shows a faster convergence rate in experiments. Extensive experimental results on real world benchmark datasets show the effectiveness and efficiency of the proposed algorithm.

Multi-view clustering attracts considerable attention due to its effectiveness in unsupervised learning. However, previous multi-view spectral clustering methods include two separated steps: 1) Obtaining a spectral embedding; 2) Performing classical clustering methods. Although these methods have achieved promising performance, there is still some limitations. First, in computing spectral embedding, multi-view spectral clustering approaches exist high computational complexity since they usually need eigenvalue decomposition on laplacian matrix L. Its computational complexity is O(n3) where n is the number of samples; Second, in constructing similarity matrices, previous methods need to compute similarity between any two samples; Third, the two-stage approach only can obtain sub-optimal solution; Fourth, treating equally all views is unreasonable. To address these issues, we propose a Fast Multi-view Clustering via Prototype Graph (FMVPG) method. Specifically, the prototype graph is firstly constructed, and then simultaneously perform spectral embedding to obtain the real matrix and spectral rotation to get the indicator matrix. In addition, the Alternating Direction Method of Multipliers (ADMM) is used to solve the joint optimization problem. Further, we conduct extensive experiments to evaluate the proposed FMVPG approach. These experimental results show the comparable or even better clustering performance than the state-of-the-art approaches.

Spectral clustering is a vital clustering method and has been widely applied for data analysis and pattern reorganization. A routine of solving spectral clustering problem consists of two successive stages: (1) solving a relaxed continuous optimization problem to obtain a real-valued indicator solution (2) transform the real-valued indicator into a 0-1 discrete one as the final clustering result. However, we may lose the optimal solution with such a two-stage process. Besides, the spectral clustering has a high time complexity which limits the analysis of large-scale data. To alleviate these problems, this paper proposes an efficient spectral clustering framework that computes spectral embedding and improved spectral rotation simultaneously (SE-ISR). In addition, we also provide a parameter-free method (SE-ISR-PF) to automatically choose the trade-off parameter. Furthermore, with an anchor-based similarity matrix construction, it is scalable to large-scale data. An effective algorithm with a strict convergence proof is provided to solve the corresponding optimization problem. Experimental results on several benchmark datasets demonstrate that the proposed algorithm outperforms the state-of-art methods.

The recent work Unsupervised Linear Discriminant Analysis (Un-LDA) completes its clustering process during the alternating optimization by converting equivalently the objective and finally using the K-means algorithm. However, the K-means algorithm has its inherent drawbacks. It is hard for the K-means algorithm to deal well with some complex clustering cases where there are too many real clusters or non-convex clusters. In this paper, a novel clustering optimization method is presented to accomplish the clustering process in Un-LDA and the resulting method can be named Un-LDA(CD). Specifically, instead of the K-means algorithm, an elaborately designed coordinate descent algorithm is adopted to obtain the clusters after the objective function goes through a series of simple but deft equivalent conversions. Extensive experiments have demonstrated that the coordinate descent clustering solution for Un-LDA can outperform the original K-means based solution on the tested data sets especially those complex data sets with a pretty large number of real clusters.

In recent decades, unsupervised feature selection methods have become increasingly popular. Nevertheless, most of the existing unsupervised feature selection methods suffer from two major problems that lead to suboptimal solutions. Many methods impose a hard linear projection constraint on original data, which is overly strict in nature and not suitable for dealing with data sampled from nonlinear manifolds. Second, most existing methods usel2,p-norm (02S and SF2SOG, which can simultaneously learn optimal flexible projections and obtain an orthogonal sparse projection to directly select discriminative features by applyingl2,0-norm constraint. Moreover, we propose to explore the local structure of flexible embedding through preserving the manifold structure of original data and adaptively constructing an optimal graph in subspace. Thirdly, the novel iterative optimization algorithms are presented to solve objective functions guaranteeing convergence theoretically. Various evaluation experiments on synthetic and real-world datasets demonstrate the effectiveness and superiority of our proposed methods.

Graph-based semi-supervised learning, which further utilizes graph structure behind samples for boosting semi-supervised learning, gains convincing results in several machine learning tasks. Nevertheless, existing graph-based methods have shortcomings from two aspects. On the one hand, many of them concentrate on improving label propagation over the constructed graph through time-saving methods, e.g. path searching, without giving insights on constructing a proper graph accommodated to samples. On the other hand, some models are only devoted to constructing the appropriate graph resulting in a two-stage procedure, which may incur a suboptimal scenario. In this paper, we develop a joint learning method that considers both bipartite graph construction and label propagation simultaneously. With this configuration, the constructed graph is constantly adjusted by the smoothness term in the objective as the algorithm proceeds. The time complexity of our method gets significant improvement compared with traditional graph-based methods, and the experimental results on one synthetic dataset and several real-world benchmarks demonstrate the effectiveness and scalability of our proposed method.

Given an affinity graph of data samples, graph-based clustering aims to partition these samples into disjoint groups based on the affinities, and most previous works are based on spectral clustering. However, two problems among spectral-based methods heavily affect the clustering performance. Firstly, the randomness of post-processing procedures, such as$K$-means, affects the stability of clustering. Secondly, the separated stages of spectral-based methods, including graph construction, spectral embedding learning, and clustering decision, lead to mismatched problems. In this paper, we explore a structured graph learning (SGL) framework that aims to fuse these stages to improve clustering stability. Specifically, SGL adaptively learns a structured affinity graph that contains exact$k$connected components. Each connected component corresponds to a cluster so clustering assignments can be directly obtained according to the connectivity of the learned graph. In this way, SGL avoids the randomness brought by reliance on traditional post-processing procedures. Meanwhile, the graph construction and structured graph learning procedures happen simultaneously, which alleviates the mismatched problem effectively. Moreover, we propose an efficient algorithm to solve the involved optimization problems and discuss the connections between this work and previous works. Numerical experiments on several synthetic and real datasets demonstrate the effectiveness of our methods.

Considering many graph clustering methods are with quadratic or cubic time complexity and need post-processing to obtain the discrete solution. Combining with the anchor graph, we study a novel graph clustering model called Rank-$r$Discrete Matrix Factorization (DMF-RR), which is linear time complexity, and motivated by nonnegative matrix factorization (NMF). Instead of constraining the factor matrices of NMF to be nonnegative as many existed methods, we constrain them to indicator matrices. Thus, DMF-RR can obtain the discrete solution by directly solving the original problem without post-processing. Furthermore, considering the greater similarity between samples of the same category, an anchor graph is constructed as an input to capture essential clustering structure by utilizing the duality information between samples and anchors. Subsequently, an efficient and simple algorithm is proposed due to the nature of indicator matrices. Extensive experiments performed on synthetic and real-world datasets demonstrate the superiority of DMF-RR.

Multi-view clustering has attracted a lot of attention due to its ability to integrate information from distinct views, but how to improve efficiency is still a hot research topic. Anchor graph-based methods and k-means-based methods are two current popular efficient methods, however, both have limitations. Clustering on the derived anchor graph takes a while for anchor graph-based methods, and the efficiency of k-means-based methods drops significantly when the data dimension is large. To emphasize these issues, we developed an efficient multi-view k-means clustering method with multiple anchor graphs (EMKMC). It first constructs anchor graphs for each view and then integrates these anchor graphs using an improved k-means strategy to obtain sample categories without any extra post-processing. Since EMKMC combines the high-efficiency portions of anchor graph-based methods and k-means-based methods, its efficiency is substantially higher than current fast methods, especially when dealing with large-scale high-dimensional multi-view data. Extensive experiments demonstrate that, compared to other state-of-the-art methods, EMKMC can boost clustering efficiency by several to thousands of times while maintaining comparable or even exceeding clustering effectiveness.

Fuzzy K-Means (FKM) clustering regards each cluster as a fuzzy set and assigns each sample to multiple clusters with a certain degree of membership. However, conventional FKM methods perform clustering on original data directly where the intrinsic structure of data may be corrupted by the noise. According, the performance of these methods would be challenged. In this paper, we present a novel fuzzy K-Means clustering model to conduct clustering tasks on the flexible manifold. Technically, we perform fuzzy clustering based on the shrunk patterns which have desired manifold structure. The shrunk patterns can be viewed as an approximation to the original data; and a penalty term is employed to measure the mismatch between them. Moreover, we integrate the learning of shrunk patterns and the learning of membership degree between shrunk patterns and clusters into a unified framework. Furthermore, we extend the proposed model for projected FKM clustering to find a suitable subspace to fit the non-linear manifold structure of data, reduce the interference of the noise and redundant features and gather homogeneous samples together simultaneously. Two alternating iterative algorithms are derived to solve these two models, respectively. Extensive experimental results demonstrate the feasibility and effectiveness of our proposed clustering algorithms.

In traditional graph embedding methods, graph construction is sensitive to high-dimensional data with noise and outliers, making an effective exploration of the neighborhood structure of the data difficult. Besides, with these methods, constructing graphs and reducing dimensions are disconnected and cannot be mutually optimized. To address these problems, we propose an unsupervised dimensionality reduction method based on bipartite graph, named unsupervised adaptive bipartite graph embedding (UABGE). First, the anchors are generated from the raw data by K-means or random sampling. Second, the bipartite graph, which is constructed between the samples and the anchors in the low-dimensional subspace, utilizes the adaptive allocation method to assign neighbors for each sample, so that the local structure of high-dimensional data can be captured effectively. Third, we present an objective function that combines bipartite graph construction and projection matrix learning to achieve mutual optimization between them, which can be solved with an alternating optimization algorithm. Finally, the computational complexity and the convergence of the algorithm are analyzed. Experimental results on synthetic data and publicly available datasets illustrate the effectiveness of the proposed method.

Graph embedding is a widely used method for dimensionality reduction due to its computational effectiveness. The quality of the graph and the efficiency of graph construction will directly affect the performance and the efficiency of the graph embedding methods. However, in the unsupervised graph embedding methods, the graph is not considered as an optimized graph since there is no label that can be used to construct this graph. In addition, the running of traditional graph embedding methods become very time-consuming on large-scale datasets due to the high computational cost in the step of graph construction. Aiming to solve these problems, we propose an unsupervised dimensionality reduction method based on bipartite graph, called Unsupervised Optimized Bipartite Graph Embedding (UOBGE). Representative anchors are firstly identified in the data. Then, we construct the bipartite graph between the projected samples and the projected anchors and the intrinsic graph connecting all the projected sample pairs with equal weights, which keep the local and global geometric structures of the data, respectively. Finally, the bipartite graph and the projection matrix are optimized simultaneously by introducing an alternating optimization procedure. Extensive experiments on several datasets demonstrate that the effectiveness and efficiency of the proposed method.

With its simplicity and effectiveness, k-means is immensely popular, but it cannot perform well on complex nonlinear datasets. Multiple kernel k-means (MKKM) demonstrates the ability to describe highly complex nonlinear separable data structures. However, its speed requirement cannot scale as well as the data size grows beyond tens of thousands. Nowadays, digital data explosion mandates more scalable clustering methods to assist the machine learning tasks in easy-to-access form. To address the issue, we propose to employ the Nystrom scheme for MKKM clustering, termed scalable multiple kernel k-means clustering. It significantly reduces the computational complexity by replacing the original kernel matrix with a low-rank approximation. Analytically and empirically, we demonstrate that our method performs as well as existing state-of-the-art methods, but at a significantly lower compute cost, allowing us to scale the method more effectively for clustering tasks.

Multiple Kernel K-means (MKKM) uses various kernels from different sources to improve clustering performance. However, most of the existing models are non-convex, which is prone to be stuck into bad local optimum, especially with noise and outliers. To address the issue, we propose a novel Self-Paced and Discrete Multiple Kernel K-Means (SPD-MKKM). It learns the MKKM model in a meaningful order by progressing both samples and kernels from easy to complex, which is beneficial to avoid bad local optimum. In addition, whereas existing methods optimize in two stages: learning the relaxation matrix and then finding the discrete one by extra discretization, our work can directly gain the discrete cluster indicator matrix without extra process. What's more, a well-designed alternative optimization is employed to reduce the overall computational complexity via using the coordinate descent technique. Finally, thorough experiments performed on real-world datasets illustrated the excellence and efficacy of our method.

Recently, many real-world applications exploit multi-view data, which is collected from diverse domains or obtained from various feature extractors and reflect different properties or distributions of the data. In this work, a novel unsupervised multi-view framework is proposed to cluster such data. The proposed method, called Multi-View clustering with Adaptive Sparse Memberships and Weight Allocation (MVASM), pays more attention to constructing a common membership matrix with proper sparseness over different views and learns the centroid matrix and its corresponding weight of each view. Concretely, MVASM method attempts to learn a common and flexible sparse membership matrix to indicate the clustering, which explores the underlying consensus information of multiple views, and solves the multiple centroid matrices and weights to utilize the view-specific information and further modifies the above-mentioned membership matrix. In addition, the theoretical analysis, including the determination of the power exponent parameter, convergence analysis, and complexity analysis are also presented. Compared to the state-of-the-art methods, the proposed method improves the performance of clustering on different public datasets and demonstrates its reasonability and superiority.

In this paper, a novel low-rank structural model is proposed for segmenting data drawn from a high-dimensional space. Our method is based on the fact that all groups clustered from a high-dimensional dataset are distributed in multiple low-rank subspaces. In general, it’s a very difficult task to find the low-rank structures hidden in data. Different from the classical sparse subspace clustering (SSC) and low-rank representation (LRR) which all take two steps including building the affinity matrix and spectral clustering, we introduce a new rank constraint into our model. This constraint allows our model to learn a subspace indicator which can capture different clusters directly from the data without any postprocessing. To further approximate the rank constraint, a piecewise function is utilized as the relaxing item for the proposed model. Besides, under the subspace indicator constraints, the integer programming problem is avoided, which makes our algorithm more efficient and scalable. In addition, we prove the convergence of the proposed algorithm in theory and further discuss the general case in which subspaces don’t pass through the origin. Experiment results on both synthetic and real-world datasets demonstrate that our algorithm significantly outperforms the state-of-the-art methods.

Semi-supervised learning as one of most attractive problems in machine learning research field has aroused broad attentions in recent years. In this paper, we propose a novel locality preserved dimensionality reduction framework, named Semi-supervised Adaptive Local Embedding learning (SALE), which learns a local discriminative embedding by constructing a$k_1$Nearest Neighbors ($k_1$NN) graph on labeled data, so as to explore the intrinsic structure, i.e., sub-manifolds from non-Gaussian labeled data. Then, mapping all samples into learned embedding and constructing another$k_2$NN graph on all embedded data to explore the global structure of all samples. Therefore, the unlabeled data and their corresponding labeled neighbors can be clustered into same sub-manifold, so as to improve the discriminative power of embedded data. Furthermore, we propose two semi-supervised dimensionality reduction methods with orthogonal and whitening constraints based on proposed SALE framework. An efficient alternatively iterative optimization algorithm is developed to solve the NP-hard problem in our models. Extensive experiments conducted on several synthetic and real-world data sets demonstrate the superiorities of our methods on local structure exploration and classification task.

There are many successful spectral based unsupervised dimensionality reduction methods, including Laplacian Eigenmap (LE), Locality Preserving Projection (LPP), Spectral Regression (SR), etc. We find that LPP and SR are equivalent if the symmetric similarity matrix is doubly stochastic, Positive Semi-Definite (PSD) and with rank$p$, where$p$is the reduced dimension. Since solving SR is believed faster than solving LPP based on some related literature, the discovery promotes us to seek to construct such specific similarity matrix to speed up LPP solving procedures. We then propose an unsupervised linear method called Unsupervised Large Graph Embedding (ULGE). ULGE starts with a similar idea as LPP but adopts an efficient approach to construct anchor-based similarity matrix and then performs spectral analysis on it. Moreover, since conventional anchor generation strategies suffer kinds of problems, we propose an efficient and effective anchor generation strategy, called Balanced$K$-means based Hierarchical$K$-means (BHKH). The computational complexity of ULGE can reduce to$O(ndm)$, which is a significant improvement compared to conventional methods need$O(n^2d)$at least, where$n$,$d$and$m$are the number of samples, dimensions, and anchors, respectively. Extensive experiments on several publicly available datasets demonstrate the efficiency and effectiveness of the proposed method.

Fuzzy K-Means (FKM) clustering is of great importance for analyzing unlabeled data. FKM algorithms assign each data point to multiple clusters with some degree of certainty measured by the membership function. In these methods, the fuzzy membership degree matrix is obtained based on the calculation of the distance between data points in the original space. However, this operation may lead to suboptimal results because of the influence of noises and redundant features. Besides, some FKM clustering methods ignore the importance of the weighting exponent. In this paper, we propose a novel FKM method called Fuzzy K-Means Clustering With Discriminative Embedding. Within this method, we simultaneously conduct dimensionality reduction along with fuzzy membership degree learning. To retain most information in the embedding subspace and improve the robustness of this method, principal component analysis is incorporated into our framework. An iterative optimization algorithm is proposed to solve the model. To validate the efficacy of the proposed method, we perform comprehensive analyses, including convergence behavior, parameter determination and computational complexity. Moreover, we also match a appropriate weighting exponent for each data set. Experimental results on benchmark data sets show that the proposed method is more discriminative and effective for clustering tasks.

Feature selection is one of the most important techniques to deal with the high-dimensional data for a variety of machine learning and data mining tasks, such clustering, classification, and retrieval, etc. Fuzziness is a widespread nature of data in nature human society. However, most existing feature selection methods ignore the existence of fuzziness in the data, resulting in sub-optimal feature subsets. To address the problem, we propose a novel unsupervised feature selection method, called Unsupervised Discriminative Projection for Feature Selection (UDPFS) to select discriminative features by conducting fuzziness learning and sparse learning, simultaneously. Specifically, we use projection matrix transform data as its low-dimensional representation, which are partitioned into clusters by using membership matrix with sparse constraint. In addition,$\ell _{2, 1}$-norm regularization is applied to the projection matrix. Then, a discriminative projection matrix with row sparse is obtained by perform fuzziness learning and sparse learning, simultaneously. An effective alternative optimization algorithm is proposed to solve the objective function. Evaluate experimental results on several real-world datasets show the effectiveness and superiority of the proposed unsupervised feature selection method.

Feature selection has attracted a lot of attention in obtaining discriminative and non-redundant features from high-dimension data. Compared with traditional filter and wrapper methods, embedded methods can obtain a more informative feature subset by fully considering the importance of features in the classification tasks. However, the existing embedded methods emphasize the above importance of features and mostly ignore the correlation between the features, which leads to retain the correlated and redundant features with similar scores in the feature subset. To solve the problem, we propose a novel supervised embedded feature selection framework, called feature selection under global redundancy minimization in orthogonal regression (GRMOR). The proposed framework can effectively recognize redundant features from a global view of redundancy among the features. We also incorporate the large margin constraint into GRMOR for robust multi-class classification. Compared with the traditional embedded methods based on least square regression, the proposed framework utilizes orthogonal regression to preserve more discriminative information in the subspace, which can help accurately rank the importance of features in the classification tasks. Experimental results on twelve public datasets demonstrate that the proposed framework can obtain superior classification performance and redundancy removal performance than twelve other feature selection methods.

Inspired by the mean calculation of RPCA_OM and inductiveness of IRPCA, we first propose an inductive robust principal component analysis method with removing the optimal mean automatically, which is shorted as IRPCA_OM. Furthermore, IRPCA_OM is extended to Schatten-$p$norm and a more general framework (i.e., EIRPCA_OM) is presented. The objective function of EIRPCA_OM includes two terms, the first term is a robust reconstruction error term constrained by an$\ell _{2,1}$-norm and the second term is a regularization term constrained by a Schatten-$p$norm. The proposed EIRPCA_OM method is robust, inductive and accurate. However, on the high-dimensional data, it would spend a large computation cost in training stage. To this end, a fast version of EIRPCA_OM called as FEIRPCA_OM is proposed, and its basic idea is to eliminate the zero eigenvalues of data matrix. More importantly, an effective theoretical proof is presented to ensure that FEIRPCA_OM has faster processing speed than EIRPCA_OM when processing high-dimensional data, but without any performance loss. Based on it, we also can exchange the less performance loss for the higher computation efficiency by removing the small eigenvalues of data matrix. Experimental results on the public datasets demonstrate that FEIRPCA_OM works efficiently on the high-dimensional data.

Multi-view semi-supervised learning (SSL) has attracted great attention due to its effectiveness in information utilization of multiple views and labeled and unlabeled data to solve practical problems. However, most existing methods exhibit high computational complexity. Effective integration of the information on different views to achieve enhanced performance remains a challenging task. In this study, we combine an anchor-based approach with multi-view semi-supervised learning to address these problems. A novel multi-view SSL method called fast multi-view SSL (FMSSL) based on learned graph is proposed. Starting from the affinity graphs constructed by using an anchor-based strategy, FMSSL learns an optimal multi-view consensus graph by using feature and label information. The learned graph can jointly consider the relation of multiple views to approximate the manifold structure. The learned graph is then introduced into the SSL model as the weight matrix of a bipartite graph to simultaneously perform separate classification on the original samples and anchors. Accordingly, multi-view SSL can be efficiently performed, and the computational complexity can be significantly reduced. We propose an effective algorithm to optimize the objective function. Extensive experimental results on different real-world datasets demonstrate the effectiveness and efficiency of the proposed algorithm.

Multi-task learning has been observed by many researchers, which supposes that different tasks can share a low-rank common yet latent subspace. It means learning multiple tasks jointly is better than learning them independently. In this paper, we propose two novel multi-task learning formulations based on two regularization terms, which can learn the optimal shared latent subspace by minimizing the exactly k minimal singular values. The proposed regularization terms are the more tight approximations of rank minimization than trace norm. But it's an NP-hard problem to solve the exact rank minimization problem. Therefore, we design a novel re-weighted based iterative strategy to solve our models, which can tactically handle the exact rank minimization problem by setting a large penalizing parameter. Experimental results on benchmark datasets demonstrate that our methods can correctly recover the low-rank structure shared across tasks, and outperform related multi-task learning methods.

Linear discriminant analysis (LDA) is a popular technique to learn the most discriminative features for multi-class classification. A vast majority of existing LDA algorithms are prone to be dominated by the class with very large deviation from the others, i.e., edge class, which occurs frequently in multi-class classification. First, the existence of edge classes often makes the total mean biased in the calculation of between-class scatter matrix. Second, the exploitation of ℓ2-norm based between-class distance criterion magnifies the extremely large distance corresponding to edge class. In this regard, a novel self-weighted robust LDA with ℓ2,1-norm based pairwise between-class distance criterion, called SWRLDA, is proposed for multi-class classification especially with edge classes. SWRLDA can automatically avoid the optimal mean calculation and simultaneously learn adaptive weights for each class pair without setting any additional parameter. An efficient re-weighted algorithm is exploited to derive the global optimum of the challenging ℓ2,1-norm maximization problem. The proposed SWRLDA is easy to implement and converges fast in practice. Extensive experiments demonstrate that SWRLDA performs favorably against other compared methods on both synthetic and real-world datasets while presenting superior computational efficiency in comparison with other techniques.

Spectral clustering is a quite simple but effective method for solving graph clustering problem. It projects the original data points into a lower dimensional space with spectral embedding, and then relies on an algorithm to obtain the cluster labels. Since it involves eigendecomposition of the graph Laplacian matrix for embedding, spectral clustering has high time complexity and is not able to process large scale data. The performance of spectral clustering is also limited by a post-processing algorithm such as kmeans. To tackle the two issues, we propose a method called Orthogonal and Nonnegative Graph Reconstruction (ONGR) for large scale clustering. The two constraints serve as a structure constraint with which the graph reconstructed by the indicator matrix is structured. The proposed method has linear time complexity with respect to the data size that it mainly needs to implicitly construct a graph and iteratively perform economical singular value decomposition for a small size matrix. Moreover, the interpretability of the indicator matrix is offered due to the nonnegative constraint, and thus our method can provide the cluster labels with no post-processing. The experiments on benchmark datasets show the effectiveness of the proposed scalable clustering method.

Recently, with the explosive increase in Internet data, the traditional Graph-based Semi-Supervised Learning (GSSL) model is not suitable to deal with large scale data as the high computation complexity. Besides, GSSL models perform classification on a fixed input data graph. The quality of initialized graph has a great effect on the classification result. To solve this problem, in this paper, we propose a novel approach, named optimal bipartite graph-based SSL (OBGSSL). Instead of fixing the input data graph, we learn a new bipartite graph to make the result more robust. Based on the learned bipartite graph, the labels of the original data and anchors can be calculated simultaneously, which solves co-classification problem in SSL. Then, we use the label of anchor to handle out-of-sample problem, which preserves well classification performance and saves much time. The computational complexity of OBGSSL is O(ndmt+nm2), which is a significant improvement compared with traditional GSSL methods that need O(n2d+n3), where n, d, m and t are the number of samples, features anchors and iterations, respectively. Experimental results demonstrate the effectiveness and efficiency of our OBGSSL model.

Unsupervised feature selection has attracted more and more attention due to the rapid growth of the large amount of unlabelled and high-dimensional data. The performance of traditional spectral-based unsupervised methods always depends on the quality of constructed similarity matrix. However, real world data always contain a large number of noise samples and features that make the similarity matrix created by original data cannot be fully relied. We propose an unsupervised feature selection method which conducts feature selection and local structure learning simultaneously. Moreover, we add an important constraint on the similarity matrix to allow it to capture more accurate information of the data structure. To perform feature selection, orthogonal constraint and `2;p-norm are adopted on the projection matrix. An efficient and simple algorithm is derived to tackle the problem. We conduct comprehensive experiments on various benchmark data sets, including handwritten digit, face image, and biomedical data, to validate the effectiveness of the proposed approach.

Linear discriminant analysis (LDA) is one of commonly used supervised subspace learning methods. However, LDA will be powerless faced with the no-label situation. In this paper, the unsupervised LDA (Un-LDA) is proposed and first formulated as a seamlessly unified objective optimization which guarantees convergence during the iteratively alternative solving process. The objective optimization is in both the ratio trace and the trace ratio forms, forming a complete framework of a new approach to jointly clustering and unsupervised subspace learning. The extension of LDA into Un-LDA enables to not only complete unsupervised subspace learning via the explicitly presented subspace projection matrix but also simultaneously finish clustering and even clustering out-of-sample data via the explicitly presented transformation matrix. To overcome the difficulty in solving the non-convex objective optimization, we mathematically prove that the Un-LDA optimization in both forms can be transformed into the simple K-means clustering optimization when the subspace is determined. The Un-LDA optimization is eventually completed by alternatively optimizing the clusters using K-means and the subspace using the supervised LDA methods and iterating this whole process until convergence or stopping criterion. The experiments demonstrate that our proposed Un-LDA algorithms are comparable or even much superior to the counterparts.

Dimensionality reduction plays a significant role in high-dimensional data processing, and Linear Discriminant Analysis (LDA) is a widely used supervised dimensionality reduction approach. However, a major drawback of LDA is that it is incapable of extracting the local structure information, which is crucial for handling multimodal data. In this article, we propose a novel supervised dimensionality reduction method named Adaptive Local Linear Discriminant Analysis (ALLDA), which adaptively learns a k -nearest neighbors graph from data themselves to extract the local connectivity of data. Furthermore, the original high-dimensional data usually contains noisy and redundant features, which has a negative impact on the evaluation of neighborships and degrades the subsequent classification performance. To address this issue, our method learns the similarity matrix and updates the subspace simultaneously so that the neighborships can be evaluated in the optimal subspaces where the noises have been removed. Through the optimal graph embedding, the underlying sub-manifolds of data in intra-class can be extracted precisely. Meanwhile, an efficient iterative optimization algorithm is proposed to solve the minimization problem. Promising experimental results on synthetic and real-world datasets are provided to evaluate the effectiveness of proposed method.

With the rapid increase of the data size, it has increasing demands for selecting features by exploiting both labeled and unlabeled data. In this paper, we propose a novel semi-supervised embedded feature selection method. The new method extends the least square regression model by rescaling the regression coefficients in the least square regression with a set of scale factors, which is used for evaluating the importance of features. An iterative algorithm is proposed to optimize the new model. It has been proved that solving the new model is equivalent to solving a sparse model with a flexible and adaptable ℓ2;pnorm regularization. Moreover, the optimal solution of scale factors provides a theoretical explanation for why we can use {||w1||2, . . .,||wd||2} to evaluate the importance of features. Experimental results on eight benchmark data sets show the superior performance of the proposed method.

Traditional graph-based Semi-Supervised Learning (SSL) methods usually contain two separate steps. First, constructing an affinity matrix. Second, inferring the unknown labels. While such a two-step method has been successful, it cannot take full advantage of the correlation between affinity matrix and label information. In order to address the above problem, we propose a novel graph-based SSL method. It can learn the affinity matrix and infer the unknown labels simultaneously. Moreover, feature selection with auto-weighting is introduced to extract the effective and robust features. Further, the proposed method learns the data similarity matrix by assigning the adaptive neighbors for each data point based on the local distance. We solve the unified problem via an alternative minimization algorithm. Extensive experimental results on synthetic data and benchmark data show that the proposed method consistently outperforms the state-of-the-art approaches.

Semi-supervised learning models for multiview data are important in image classification tasks, since heterogeneous features are easy to obtain and semi-supervised schemes are economical and effective. To model the view importance, conventional graph-based multiview learning models learn a linear combination of views while assuming a priori weights distribution. In this paper, we present a novel structural regularized semi-supervised model for multiview data, termed Adaptive MUltiview SEmi-supervised model (AMUSE). Our new model learns weights from a priori graph structure, which is more reasonable than weight regularization. Theoretical analysis reveals the significant difference between AMUSE and the prior arts. An efficient optimization algorithm is provided to solve the new model. Experimental results on six real-world data sets demonstrate the effectiveness of the structural regularized weights learning scheme.

In many real-world applications, we are often confronted with high dimensional data which are represented by various heterogeneous views. How to cluster this kind of data is still a challenging problem due to the curse of dimensionality and effectively integration of different views. To address this problem, we propose two parameter-free weighted multi-view projected clustering methods which perform structured graph learning and dimensionality reduction simultaneously. We can use the obtained structured graph directly to extract the clustering indicators, without performing other discretization procedures as previous graph-based clustering methods have to do. Moreover, two parameter-free strategies are adopted to learn an optimal weight for each view automatically, without introducing a regularization parameter as previous methods do. Extensive experiments on several public datasets demonstrate that the proposed methods outperform other state-of-the-art approaches and can be used more practically.

With the explosive growth of data, the multi-view data is widely used in many fields, such as data mining, machine learning, computer vision, and so on. Because such data always has a complex structure, i.e., many categories, many perspectives of description and high dimension, how to formulate an accurate and reliable framework for the multi-view classification is a very challenging task. In this paper, we propose a novel multi-view classification method by using multiple multi-class Support Vector Machines (SVMs) with a novel collaborative strategy. Here, each multi-class SVM embeds the scaling factor to renewedly adjust the weight allocation of all features, which is beneficial to highlight more important and discriminative features. Furthermore, we adopt the decision function values to integrate multiple multi-class learners and introduce the confidence score across multiple classes to determine the final classification result. In addition, through a series of the mathematical deduction, we bridge the proposed model with the solvable problem and solve it through an alternating iteration optimization method. We evaluate the proposed method on several image and face datasets, and the experimental results demonstrate that our proposed method performs better than other state-of-the-art learning algorithms.

In this paper, we make an extension of K-means for the clustering of multiple means. The popular K-means clustering uses only one center to model each class of data. However, the assumption on the shape of the clusters prohibits it to capture the non-convex patterns. Moreover, many categories consist of multiple subclasses which obviously cannot be represented by a single prototype. We propose a K-Multiple-Means (KMM) method to group the data points with multiple sub-cluster means into specified k clusters. Unlike the methods which use the agglomerative strategies, the proposed method formalizes the multiple-means clustering problem as an optimization problem and updates the partitions of m sub-cluster means and k clusters by an alternating optimization strategy. Notably, the partition of the original data with multiple-means representation is modeled as a bipartite graph partitioning problem with the constrained Laplacian rank. We also show the theoretical analysis of the connection between our method and the K-means clustering. Meanwhile, KMM is linear scaled with respect to n. Experimental results on several synthetic and well-known real-world data sets are conducted to show the effectiveness of the proposed algorithm.

Sparse regression based feature selection method has been extensively investigated these years. However, because it has a non-convex constraint, i.e., $\ell _{2,0}$ℓ2,0-norm constraint, this problem is very hard to solve. In this paper, unlike most of the other methods which only solve its slack version by introducing sparsity regularization into objective function forcibly, a novel framework is proposed by us to solve the original $\ell _{2,0}$ℓ2,0-norm constrained sparse regression based feature selection problem. We transform our objective function into Linear Discriminant Analysis (LDA) by using a new label coding method, thus enabling our model to calculate the ratio of inter-class scatter to intra-class scatter of features which is the most widely used feature discrimination evaluation metric. According to that ratio, features can be selected by a simple sorting method. The projection gradient descent algorithm is introduced to further improve the performance of our algorithm by using the solution obtained before as its initial solution. This ensures the stability of this iterative algorithm. We prove that the proposed method can get the global optimal solution of this non-convex problem when all features are statistically independent. For the general case where features are statistically dependent, extensive experiments on six small sample size datasets and one large-scale dataset show that our algorithm has comparable or better classification capability comparing with other eight state-of-the-art feature selection methods by the SVM classifier. We also show that our algorithm can obtain a low loss value, which means the solution of our algorithm can get very close to this NP-hard problem’s real solution. What is more, because we solve the original $\ell _{2,0}$ℓ2,0-norm constrained problem, we avoid the heavy work of tuning the regularization parameter because its meaning is explicit in our method, i.e., the number of selected features. At last, we evaluate the stability of our algorithm from two perspectives, i.e., the objective function values and the selected features, by experiments. From both perspectives, our algorithm shows satisfactory stability performance.

Most existing multiview clustering methods take graphs, which are usually predefined independently in each view, as input to uncover data distribution. These methods ignore the correlation of graph structure among multiple views and clustering results highly depend on the quality of predefined affinity graphs. We address the problem of multiview clustering by seamlessly integrating graph structures of different views to fully exploit the geometric property of underlying data structure. The proposed method is based on the assumption that the intrinsic underlying graph structure would assign corresponding connected component in each graph to the same cluster. Different graphs from multiple views are integrated by using the Hadamard product since different views usually together admit the same underlying structure across multiple views. Specifically, these graphs are integrated into a global one and the structure of the global graph is adaptively tuned by a well-designed objective function so that the number of components of the graph is exactly equal to the number of clusters. It is worth noting that we directly obtain cluster indicators from the graph itself without performing further graph-cut or k-means clustering algorithms. Experiments show the proposed method obtains better clustering performance than the state-of-the-art methods.

Dimensionality reduction is a critical technology in the domain of pattern recognition, and linear discriminant analysis (LDA) is one of the most popular supervised dimensionality reduction methods. However, whenever its distance criterion of objective function uses$L_2$-norm, it is sensitive to outliers. In this paper, we propose a new formulation of linear discriminant analysis via joint$L_{2,1}$-norm minimization on objective function to induce robustness, so as to efficiently alleviate the influence of outliers and improve the robustness of proposed method. An efficient iterative algorithm is proposed to solve the optimization problem and proved to be convergent. Extensive experiments are performed on an artificial data set, on UCI data sets, and on four face data sets, which sufficiently demonstrates the efficiency of comparing to other methods and robustness to outliers of our approach.

In machine learning and data mining, dimensionality reduction is one of the main tasks. Linear Discriminant Analysis (LDA) is a widely used supervised dimensionality reduction algorithm and it has attracted a lot of research interests. Classical Linear Discriminant Analysis finds a subspace to minimize within-class distance and maximize between-class distance, where between-class distance is computed using arithmetic mean of all between-class distances. However, arithmetic mean between-class distance has some limitations. First, arithmetic mean gives equal weight to all between-class distances, and large between-class distance could dominate the result. Second, it does not consider pairwise between-class distance and thus some classes may overlap with each other in the subspace. In this paper, we propose two formulations of harmonic mean based Linear Discriminant Analysis: HLDA and HLDAp, to demonstrate the benefit of harmonic mean between-class distance and overcome the limitations of classical LDA. We compare our algorithm with 11 existing single-label algorithms on seven datasets and five existing multi-label algorithms on two datasets. On some single-label experiment data, the classification accuracy absolute percentage increase can reach 39 percent compared to state-of-art existing algorithms; on multi-label data, significant improvement on five evaluation metric has been achieved compared to existing algorithms.

As one of the most widely used clustering techniques, the fuzzy K-Means (also called FKM or FCM) assigns every data point to each cluster with a certain degree of membership. However, conventional FKM approach relies on the square data fitting term which is not robust to data outliers and ignores the prior information, which leads to unsatisfactory clustering results. In this paper, we present a novel and robust fuzzy K-Means clustering algorithm, namely Embedding Fuzzy K-Means with Nonnegative Spectral Clustering via Incorporating Side Information. The proposed method combines fuzzy K-Means with nonnegative spectral clustering into a unified model, and further takes the advantage of the prior knowledge of data pairs such that the quality of similarity graph is enhanced and the clustering performance is effectively improved. Besides, the ℓ2,1-norm loss function is adopted in the objective function, which achieves better robustness to outliers. Last, experimental results on benchmark datasets verify the effectiveness and superiority of the proposed clustering method.

During the past decades, many spectral clustering algorithms have been proposed. However, their high computational complexities hinder their applications on large-scale data. Moreover, most of them use a two-step approach to obtain the optimal solution, which may deviate from the solution by directly solving the original problem. In this paper, we propose a new optimization algorithm, namely Direct Normalized Cut (DNC), to directly optimize the normalized cut model. DNC has a quadratic time complexity, which is a significant reduction comparing with the cubic time complexity of the traditional spectral clustering. To cope with large-scale data, a Fast Normalized Cut (FNC) method with linear time and space complexities is proposed by extending DNC with an anchor-based strategy. In the new method, we first seek a set of anchors and then construct a representative similarity matrix by computing distances between the anchors and the whole data set. To find high quality anchors that best represent the whole data set, we propose a Balanced k-means (BKM) to partition a data set into balanced clusters and use the cluster centers as anchors. Then DNC is used to obtain the final clustering result from the representative similarity matrix. A series of experiments were conducted on both synthetic data and real-world data sets, and the experimental results show the superior performance of BKM, DNC and FNC.

This paper proposes a novel algorithm, named Non-Convex Calibrated Multi-Task Learning (NC-CMTL), for learning multiple related regression tasks jointly. Instead of utilizing the nuclear norm, NC-CMTL adopts a non-convex low rank regularizer to explore the shared information among different tasks. In addition, considering that the regularization parameter for each regression task desponds on its noise level, we replace the least squares loss function by square-root loss function. Computationally, as proposed model has a nonsmooth loss function and a non-convex regularization term, we construct an efcient re-weighted method to optimize it. Theoretically, we frst present the convergence analysis of constructed method, and then prove that the derived solution is a stationary point of original problem. Particularly, the regularizer and optimization method used in this paper are also suitable for other rank minimization problems. Numerical experiments on both synthetic and real data illustrate the advantages of NC-CMTL over several state-of-the-art methods.

In this paper, we make a multiview extension of the spectral rotation technique raised in single view spectral clustering research. Since spectral rotation is closely related to the Procrustes Analysis for points matching, we point out that classical Procrustes Average approach can be used for multiview clustering. Besides, we show that direct applying Procrustes Average (PA) in multiview tasks may not be optimal theoretically and empirically, since it does not take the clustering capacity differences of different views into consideration. Other than that, we propose an Adaptively Weighted Procrustes (AWP) approach to overcome the aforementioned deficiency. Our new AWP weights views with their clustering capacities and forms a weighted Procrustes Average problem accordingly. The optimization algorithm to solve the new model is computational complexity analyzed and convergence guaranteed. Experiments on five real-world datasets demonstrate the effectiveness and efficiency of the new models.

Clustering of customer transaction data is an important procedure to analyze customer behaviors in retail and e-commerce companies. Note that products from companies are often organized as a product tree, in which the leaf nodes are goods to sell, and the internal nodes (except root node) could be multiple product categories. Based on this tree, we propose the “personalized product tree”, named purchase tree, to represent a customer's transaction records. So the customers' transaction data set can be compressed into a set of purchase trees. We propose a partitional clustering algorithm, named PurTreeClust, for fast clustering of purchase trees. A new distance metric is proposed to effectively compute the distance between two purchase trees. To cluster the purchase tree data, we first rank the purchase trees as candidate representative trees with a novel separate density, and then select the top k customers as the representatives of k customer groups. Finally, the clustering results are obtained by assigning each customer to the nearest representative. We also propose a gap statistic based method to evaluate the number of clusters. A series of experiments were conducted on ten real-life transaction data sets, and experimental results show the superior performance of the proposed method.

Feature selection plays a significant role in dealing with high-dimensional data to avoid the curse of dimensionality. In many real applications, like video semantic recognition, handling few labeled and large unlabeled data samples from the same population is a recently addressed challenge in feature selection. To solve this problem, we propose a novel semi-supervised feature selection method via insensitive sparse regression (ISR). Specifically, we compute the soft label matrix by the special label propagation, which can predict the labels of the unlabeled data. To guarantee the robustness of ISR to the false labeled instances or outliers, we propose Insensitive Regression Model (IRM) by capped$l_2$-$l_p$-norm loss. The soft label is imposed as the weights of IRM to fully utilize the label information. Meanwhile, to perform feature selection, we incorporate$l_{2,q}$-norm regularizer with IRM as the structural sparsity constraint when$0 < q\leq 1$. Moreover, we put forward an effective approach for solving the formulated non-convex optimization problem. We analyze the performance of convergence rigorously and discuss the parameter determination problem. Extensive experimental results on several public data sets verify the effectiveness of our proposed algorithm in comparison with the state-of-art feature selection methods. Finally, we apply our method to video semantic recognition successfully.

With the advent of multi-view data, multi-view learning has become an important research direction in both machine learning and data mining. Considering the difficulty of obtaining labeled data in many real applications, we focus on the multi-view unsupervised feature selection problem. Traditional approaches all characterize the similarity by fixed and pre-defined graph Laplacian in each view separately and ignore the underlying common structures across different views. In this paper, we propose an algorithm named Multi-view Unsupervised Feature Selection with Adaptive Similarity and View Weight (ASVW) to overcome the above mentioned problems. Specifically, by leveraging the learning mechanism to characterize the common structures adaptively, we formulate the objective function by a common graph Laplacian across different views, together with the sparse ℓ2,p-norm constraint designed for feature selection. We develop an efficient algorithm to address the non-smooth minimization problem and prove that the algorithm will converge. To validate the effectiveness of ASVW, comparisons are made with some benchmark methods on real-world datasets. We also evaluate our method in the real sports action recognition task. The experimental results demonstrate the effectiveness of our proposed algorithm.

Linear discriminant analysis (LDA) is one of the most important supervised linear dimensional reduction techniques which seeks to learn low-dimensional representation from the original high-dimensional feature space through a transformation matrix, while preserving the discriminative information via maximizing the between-class scatter matrix and minimizing the within class scatter matrix. However, the conventional LDA is formulated to maximize the arithmetic mean of trace ratios which suffers from the domination of the largest objectives and might deteriorate the recognition accuracy in practical applications with a large number of classes. In this paper, we propose a new criterion to maximize the weighted harmonic mean of trace ratios, which effectively avoid the domination problem while did not raise any difficulties in the formulation. An efficient algorithm is exploited to solve the proposed challenging problems with fast convergence, which might always find the globally optimal solution just using eigenvalue decomposition in each iteration. Finally, we conduct extensive experiments to illustrate the effectiveness and superiority of our method over both of synthetic datasets and real-life datasets for various tasks, including face recognition, human motion recognition and head pose recognition. The experimental results indicate that our algorithm consistently outperforms other compared methods on all of the datasets.

Many feature extraction methods reduce the dimensionality of data based on the input graph matrix. The graph construction which reflects relationships among raw data points is crucial to the quality of resulting low-dimensional representations. To improve the quality of graph and make it more suitable for feature extraction tasks, we incorporate a new graph learning mechanism into feature extraction and add an interaction between the learned graph and the low-dimensional representations. Based on this learning mechanism, we propose a novel framework, termed as unsupervised single view feature extraction with structured graph (FESG), which learns both a transformation matrix and an ideal structured graph containing the clustering information. Moreover, we propose a novel way to extend FESG framework for multi-view learning tasks. The extension is named as unsupervised multiple views feature extraction with structured graph (MFESG), which learns an optimal weight for each view automatically without requiring an additional parameter. To show the effectiveness of the framework, we design two concrete formulations within FESG and MFESG, together with two efficient solving algorithms. Promising experimental results on plenty of real-world datasets have validated the effectiveness of our proposed algorithms.

Multi-label data with high dimensionality arise frequently in data mining and machine learning. It is not only time consuming but also computationally unreliable when we use high-dimensional data directly. Supervised dimensionality reduction approaches are based on the assumption that there are large amounts of labeled data. It is infeasible to label a large number of training samples in practice especially in multi-label learning. To address these challenges, we propose a novel algorithm, namely Semi-Supervised Multi-Label Dimensionality Reduction (SSMLDR), which can utilize the information from both labeled data and unlabeled data in an effective way. First, the proposed algorithm enlarges the multi-label information from the labeled data to the unlabeled data through a special designed label propagation method. It then learns a transformation matrix to perform dimensionality reduction by incorporating the enlarged multi-label information. Extensive experiments on a broad range of datasets validate the effectiveness of our approach against other well-established algorithms.

In multi-task learning, it is paramount to discover the relational structure of tasks and utilize the learned task structure. Previous works have been using the low-rank latent feature subspace to capture the task relations, and some of them aim to learn the group based relational structure of tasks. However, in many cases, the low-rank subspace may not exist for the specific group of tasks, thus using this paradigm would not work. To discover the task relational structures, we propose a novel multi-task learning method using the structured sparsity-inducing norms to automatically uncover the relations of tasks. Instead of imposing the low-rank constraint, our new model uses a more meaningful assumption, in which the tasks from the same relational group should share the common feature subspace. We can discover the group relational structure of tasks and learn the shared feature subspace for each task group, which help to improve the predictive performance. Our proposed algorithm avoids the high computational complexity of integer programming, thus it converges very fast. Empirical studies conducted on both synthetic and real-world data show that our method consistently outperforms related multi-task learning methods.

Metric learning aims at automatically learning a metric from pair or triplet based constraints in data, and it can be potentially beneficial whenever the notion of metric between instances plays a nontrivial role. In Mahalanobis distance metric learning, distance matrix M is in symmetric positive semi-definite cone, and in order to avoid overfitting and to learn a better Mahalanobis distance from weakly supervised constraints, the low-rank regularization has been often imposed on matrix M to learn the correlations between features and samples. As the approximations of the rank minimization function, the trace norm and Fantope have been utilized to regularize the metric learning objectives and achieve good performance. However, these low-rank regularization models are either not tight enough to approximate rank minimization or time-consuming to tune an optimal rank. In this paper, we introduce a novel metric learning model using the capped trace norm based regularization, which uses a singular value threshold to constraint the metric matrix M as low-rank explicitly such that the rank of matrix M is stable when the large singular values vary. The capped trace norm regularization can also be viewed as the adaptive Fantope regularization. We minimize singular values which are less than threshold value and the rank of M is not necessary to be k, thus our method is more stable and applicable in practice when we do not know the optimal rank of matrix M. We derive an efficient optimization algorithm to solve the proposed new model and the algorithm convergence proof is also provided in this paper. We evaluate our method on a variety of challenging benchmarks, such as LFW and Pubfig datasets. Face verification experiments are performed and results show that our method consistently outperforms the state-of-the-art metric learning algorithms.

As one of the most significant machine learning topics, clustering has been extensively employed in various kinds of area. Its prevalent application in scientific research as well as industrial practice has drawn high attention in this day and age. A multitude of clustering methods have been developed, among which the graph based clustering method using the affinity matrix has been laid great emphasis on. Recent research work used the doubly stochastic matrix to normalize the input affinity matrix and enhance the graph based clustering models. Although the doubly stochastic matrix can improve the clustering performance, the clustering structure in the doubly stochastic matrix is not clear as expected. Thus, post processing step is required to extract the final clustering results, which may not be optimal. To address this problem, in this paper, we propose a novel convex model to learn the structured doubly stochastic matrix by imposing low-rank constraint on the graph Laplacian matrix. Our new structured doubly stochastic matrix can explicitly uncover the clustering structure and encode the probabilities of pair-wise data points to be connected, such that the clustering results are enhanced. An efficient optimization algorithm is derived to solve our new objective. Also, we provide theoretical discussions that when the input differs, our method possesses interesting connections with K-means and spectral graph cut models respectively. We conduct experiments on both synthetic and benchmark datasets to validate the performance of our proposed method. The empirical results demonstrate that our model provides an approach to better solving the K-mean clustering problem. By using the cluster indicator provided by our model as initialization, K-means converges to a smaller objective function value with better clustering performance. Moreover, we compare the clustering performance of our model with spectral clustering and related double stochastic model. On all datasets, our method performs equally or better than the related methods.

Principal component analysis (PCA) has been widely applied to dimensionality reduction and data pre-processing for different applications in engineering, biology, social science, and the like. Classical PCA and its variants seek for linear projections of the original variables to obtain the low-dimensional feature representations with maximal variance. One limitation is that it is difficult to interpret the results of PCA. Besides, the classical PCA is vulnerable to certain noisy data. In this paper, we propose a Convex Sparse Principal Component Analysis (CSPCA) algorithm and apply it to feature learning. First, we show that PCA can be formulated as a low-rank regression optimization problem. Based on the discussion, thel2, 1-normminimization is incorporated into the objective function to make the regression coefficients sparse, thereby robust to the outliers. Also, based on the sparse model used in CSPCA, an optimal weight is assigned to each of the original feature, which in turn provides the output with good interpretability. With the output of our CSPCA, we can effectively analyze the importance of each feature under the PCA criteria. Our new objective function is convex, and we propose an iterative algorithm to optimize it. We apply the CSPCA algorithm to feature selection and conduct extensive experiments on seven benchmark datasets. Experimental results demonstrate that the proposed algorithm outperforms state-of-the-art unsupervised feature selection algorithms.

As an important matrix factorization model, Nonnegative Matrix Factorization (NMF) has been widely used in information retrieval and data mining research. Standard Nonnegative Matrix Factorization is known to use the Frobenius norm to calculate the residual, making it sensitive to noises and outliers. It is desirable to use robust NMF models for practical applications, in which usually there are many data outliers. It has been studied that the 2,1, or 1-norm can be used for robust NMF formulations to deal with data outliers. However, these alternatives still suffer from the extreme data outliers. In this paper, we present a novel robust capped norm orthogonal Nonnegative Matrix Factorization model, which utilizes the capped norm for the objective to handle these extreme outliers. Meanwhile, we derive a new efficient optimization algorithm to solve the proposed non-convex non-smooth objective. Extensive experiments on both synthetic and real datasets show our proposed new robust NMF method consistently outperforms related approaches.

With the rapid growth of modern technologies, Internet has reached almost every corner of the world. As a result, it becomes more and more important to manage and mine information contained in Web pages in different languages. Traditional supervised learning methods usually require a large amount of training data to obtain accurate and robust classification models. However, labeled Web pages did not increase as fast as the growth of Internet. The lack of sufficient training Web pages in many languages, especially for those in uncommonly used languages, makes it a challenge for traditional classification algorithms to achieve satisfactory performance. To address this, we observe that Web pages for a same topic from different languages usually share some common semantic patterns, though in different representation forms. In addition, we also observe that the associations between word clusters and Web page classes are another type of reliable carriers to transfer knowledge across languages. With these recognitions, in this article we propose a novel joint nonnegative matrix trifactorization (NMTF) based Dual Knowledge Transfer (DKT) approach for cross-language Web page classification. Our approach transfers knowledge from the auxiliary language, in which abundant labeled Web pages are available, to the target languages, in which we want to classify Web pages, through two different paths: word cluster approximation and the associations between word clusters and Web page classes. With the reinforcement between these two different knowledge transfer paths, our approach can achieve better classification accuracy. In order to deal with the large-scale real world data, we further develop the proposed DKT approach by constraining the factor matrices of NMTF to be cluster indicator matrices. Due to the nature of cluster indicator matrices, we can decouple the proposed optimization objective and the resulted subproblems are of much smaller sizes involving much less matrix multiplications, which make our new approach much more computationally efficient. We evaluate the proposed approach in extensive experiments using a real world cross-language Web page data set. Promising results have demonstrated the effectiveness of our approach that are consistent with our theoretical analyses.

Feature selection has been an important research topic in data mining, because the real data sets often have high-dimensional features, such as the bioinformatics and text mining applications. Many existing filter feature selection methods rank features by optimizing certain feature ranking criterions, such that correlated features often have similar rankings. These correlated features are redundant and don't provide large mutual information to help data mining. Thus, when we select a limited number of features, we hope to select the top non-redundant features such that the useful mutual information can be maximized. In previous research, Ding et al. recognized this important issue and proposed the minimum Redundancy Maximum Relevance Feature Selection (mRMR) model to minimize the redundancy between sequentially selected features. However, this method used the greedy search, thus the global feature redundancy wasn't considered and the results are not optimal. In this paper, we propose a new feature selection framework to globally minimize the feature redundancy with maximizing the given feature ranking scores, which can come from any supervised or unsupervised methods. Our new model has no parameter so that it is especially suitable for practical data mining application. Experimental results on benchmark data sets show that the proposed method consistently improves the feature selection results compared to the original methods. Meanwhile, we introduce a new unsupervised global and local discriminative feature selection method which can be unified with the global feature redundancy minimization framework and shows superior performance.

Many clustering methods partition the data groups based on the input data similarity matrix. Thus, the clustering results highly depend on the data similarity learning. Because the similarity measurement and data clustering are often conducted in two separated steps, the learned data similarity may not be the optimal one for data clustering and lead to the suboptimal results. In this paper, we propose a novel clustering model to learn the data similarity matrix and clustering structure simultaneously. Our new model learns the data similarity matrix by assigning the adaptive and optimal neighbors for each data point based on the local distances. Meanwhile, the new rank constraint is imposed to the Laplacian matrix of the data similarity matrix, such that the connected components in the resulted similarity matrix are exactly equal to the cluster number. We derive an efficient algorithm to optimize the proposed challenging problem, and show the theoretical analysis on the connections between our method and the K-means clustering, and spectral clustering. We also further extend the new clustering model for the projected clustering to handle the high-dimensional data. Extensive empirical results on both synthetic data and real-world benchmark data sets show that our new clustering methods consistently outperforms the related clustering approaches.

Most semi-supervised learning models propagate the labels over the Laplacian graph, where the graph should be built beforehand. However, the computational cost of constructing the Laplacian graph matrix is very high. On the other hand, when we do classification, data points lying around the decision boundary (boundary points) are noisy for learning the correct classifier and deteriorate the classification performance. To address these two challenges, in this paper, we propose an adaptive semi-supervised learning model. Different from previous semi-supervised learning approaches, our new model needn't construct the graph Laplacian matrix. Thus, our method avoids the huge computational cost required by previous methods, and achieves a computational complexity linear to the number of data points. Therefore, our method is scalable to large-scale data. Moreover, the proposed model adaptively suppresses the weights of boundary points, such that our new model is robust to the boundary points. An efficient algorithm is derived to alternatively optimize the model parameter and class probability distribution of the unlabeled data, such that the induction of classifier and the transduction of labels are adaptively unified into one framework. Extensive experimental results on six real-world data sets show that the proposed semi-supervised learning model outperforms other related methods in most cases.

The low-rank regression model has been studied and applied to capture the underlying classes/tasks correlation patterns, such that the regression/classification results can be enhanced. In this paper, we will prove that the low-rank regression model is equivalent to doing linear regression in the linear discriminant analysis (LDA) subspace. Our new theory reveals the learning mechanism of low-rank regression, and shows that the low-rank structures exacted from classes/tasks are connected to the LDA projection results. Thus, the low-rank regression efficiently works for the high-dimensional data.

Along with increasing popularity of social websites, online users rely more on the trustworthiness information to make decisions, extract and filter information, and tag and build connections with other users. However, such social network data often suffer from severe data sparsity and are not able to provide users with enough information. Therefore, trust prediction has emerged as an important topic in social network research. Traditional approaches are primarily based on exploring trust graph topology itself. However, research in sociology and our life experience suggest that people who are in the same social circle often exhibit similar behaviors and tastes. To take advantage of the ancillary information for trust prediction, the challenge then becomes what to transfer and how to transfer. In this article, we address this problem by aggregating heterogeneous social networks and propose a novel joint social networks mining (JSNM) method. Our new joint learning model explores the user-group-level similarity between correlated graphs and simultaneously learns the individual graph structure; therefore, the shared structures and patterns from multiple social networks can be utilized to enhance the prediction tasks. As a result, we not only improve the trust prediction in the target graph but also facilitate other information retrieval tasks in the auxiliary graphs. To optimize the proposed objective function, we use the alternative technique to break down the objective function into several manageable subproblems. We further introduce the auxiliary function to solve the optimization problems with rigorously proved convergence. The extensive experiments have been conducted on both synthetic and real- world data. All empirical results demonstrate the effectiveness of our method.

Along with the increasing popularity of social web sites, users rely more on the trustworthiness information for many online activities among users. However, such social network data often suffers from severe data sparsity and are not able to provide users with enough information. Therefore, trust prediction has emerged as an important topic in social network research. Traditional approaches explore the topology of trust graph. Previous research in sociology and our life experience suggest that people who are in the same social circle often exhibit similar behavior and tastes. Such ancillary information, is often accessible and therefore could potentially help the trust prediction. In this paper, we address the link prediction problem by aggregating heterogeneous social networks and propose a novel joint manifold factorization (JMF) method. Our new joint learning model explores the user group level similarity between correlated graphs and simultaneously learns the individual graph structure, therefore the shared structures and patterns from multiple social networks can be utilized to enhance the prediction tasks. As a result, we not only improve the trust prediction in the target graph, but also facilitate other information retrieval tasks in the auxiliary graphs. To optimize the objective function, we break down the proposed objective function into several manageable sub-problems, then further establish the theoretical convergence with the aid of auxiliary function. Extensive experiments were conducted on real world data sets and all empirical results demonstrated the effectiveness of our method.

The low-rank matrix completion problem is a fundamental machine learning problem with many important applications. The standard low-rank matrix completion methods relax the rank minimization problem by the trace norm minimization. However, this relaxation may make the solution seriously deviate from the original solution. Meanwhile, most completion methods minimize the squared prediction errors on the observed entries, which is sensitive to outliers. In this paper, we propose a new robust matrix completion method to address these two problems. The joint Schatten p-norm and ℓp-norm are used to better approximate the rank minimization problem and enhance the robustness to outliers. The extensive experiments are performed on both synthetic data and real world applications in collaborative filtering and social network link prediction. All empirical results show our new method outperforms the standard matrix completion methods.

This paper addresses the scalability issue in spectral analysis which has been widely used in data management applications. Spectral analysis techniques enjoy powerful clustering capability while suffer from high computational complexity. In most of previous research, the bottleneck of computational complexity of spectral analysis stems from the construction of pairwise similarity matrix among objects, which costs at least O(n2) where n is the number of the data points. In this paper, we propose a novel estimator of the similarity matrix using K-means accumulative consensus matrix which is intrinsically sparse. The computational cost of the accumulative consensus matrix is O(nlogn). We further develop a Non-negative Matrix Factorization approach to derive clustering assignment. The overall complexity of our approach remains O(nlogn). In order to validate our method, we (1) theoretically show the local preserving and convergent property of the similarity estimator, (2) validate it by a large number of real world datasets and compare the results to other state-of-the-art spectral analysis, and (3) apply it to large-scale data clustering problems. Results show that our approach uses much less computational time than other state-of-the-art clustering methods, meanwhile provides comparable clustering qualities. We also successfully apply our approach to a 5-million dataset on a single machine using reasonable time. Our techniques open a new direction for high-quality large-scale data analysis.

Feature selection is an essential component of data mining. In many data analysis tasks where the number of data point is much less than the number of features, efficient feature selection approaches are desired to extract meaningful features and to eliminate redundant ones. In the previous study, many data mining techniques have been applied to tackle the above challenging problem. In this paper, we propose a new ℓ2,1-norm SVM, that is, multi-class hinge loss with a structured regularization term for all the classes to naturally select features for multi-class without bothering further heuristic strategy. Rather than directly solving the multi-class hinge loss with ℓ2,1-norm regularization minimization, which has not been solved before due to its optimization difficulty, we are the first to give an efficient algorithm bridging the new problem with a previous solvable optimization problem to do multi-class feature selection. A global convergence proof for our method is also presented. Via the proposed efficient algorithm, we select features across multiple classes with jointly sparsity, i.e., each feature has either small or large score over all classes. Comprehensive experiments have been performed on six bioinformatics data sets to show that our method can obtain better or competitive performance compared with exiting state-of-art multi-class feature selection approaches.

The fast growth of Internet and modern technologies has brought data involving objects of multiple types that are related to each other, called as Multi-Type Relational data. Traditional clustering methods for single-type data rarely work well on them, which calls for new clustering techniques, called as high-order co-clustering (HOCC), to deal with the multiple types of data at the same time. A major challenge in developing HOCC methods is how to effectively make use of all available information contained in a multi-type relational data set, including both inter-type and intra-type relationships. Meanwhile, because many real world data sets are often of large sizes, clustering methods with computationally efficient solution algorithms are of great practical interest. In this paper, we first present a general HOCC framework, named as Orthogonal Nonnegative Matrix Tri-factorization (O-NMTF), for simultaneous clustering of multi-type relational data. The proposed O-NMTF approach employs Nonnegative Matrix Tri-Factorization (NMTF) to simultaneously cluster different types of data using the inter-type relationships, and incorporate intra-type information through manifold regularization, where, different from existing works, we emphasize the importance of the orthogonal ties of the factor matrices of NMTF. Based on O-NMTF, we further develop a novel Fast Nonnegative Matrix Tri-Factorization (F-NMTF) approach to deal with large-scale data. Instead of constraining the factor matrices of NMTF to be nonnegative as in existing methods, F-NMTF constrains them to be cluster indicator matrices, a special type of nonnegative matrices. As a result, the optimization problem of the proposed method can be decoupled, which results in sub problems of much smaller sizes requiring much less matrix multiplications, such that our new algorithm scales well to real world data of large sizes. Extensive experimental evaluations have demonstrated the effectiveness of our new approaches.

The lack of sufficient labeled Web pages in many languages, especially for those uncommonly used ones, presents a great challenge to traditional supervised classification methods to achieve satisfactory Web page classification performance. To address this, we propose a novel Nonnegative Matrix Tri-factorization (NMTF) based Dual Knowledge Transfer (DKT) approach for cross-language Web page classification, which is based on the following two important observations. First, we observe that Web pages for a same topic from different languages usually share some common semantic patterns, though in different representation forms. Second, we also observe that the associations between word clusters and Web page classes are a more reliable carrier than raw words to transfer knowledge across languages. With these recognitions, we attempt to transfer knowledge from the auxiliary language, in which abundant labeled Web pages are available, to target languages, in which we want classify Web pages, through two different paths: word cluster approximations and the associations between word clusters and Web page classes. Due to the reinforcement between these two different knowledge transfer paths, our approach can achieve better classification accuracy. We evaluate the proposed approach in extensive experiments using a real world cross-language Web page data set. Promising results demonstrate the effectiveness of our approach that is consistent with our theoretical analyses.

Semi-supervised learning has been successfully applied to many fields such as knowledge management, information retrieval and data mining as it can utilize both labeled and unlabeled data. In this paper, we propose a general semi-supervised framework for multi-class categorization. Many classical supervised and semi-supervised method dealing with binary classification or multi-class classification including the standard regularization and the manifold regularization can be viewed as special cases of this framework. Based on this framework, we propose a novel method called multi-class unlabeled constrained SVM(MCUCSVM) and its special case: multi-class Laplacian SVM(MCLapSVM). We then put forward a general kernel version semi-supervised dual coordinate descent algorithm to efficiently solve MCUCSVM and makes it more applicable to problems with large number of classes and large scale labeled data. Both rigorous theory and promising experimental results on four real datasets show the great performance and remarkable efficiency of MCUCSVM and MCLapSVM.

This paper presents a new algorithm for Nonlinear Dimensionality Reduction (NLDR). Our algorithm is developed under the conceptual framework of compatible mapping. Each such mapping is a compound of a tangent space projection and a group of splines. Tangent space projection is estimated at each data point on the manifold, through which the data point itself and its neighbors are represented in tangent space with local coordinates. Splines are then constructed to guarantee that each of the local coordinates can be mapped to its own single global coordinate with respect to the underlying manifold. Thus, the compatibility between local alignments is ensured. In such a work setting, we develop an optimization framework based on reconstruction error analysis, which can yield a global optimum. The proposed algorithm is also extended to embed out of samples via spline interpolation. Experiments on toy data sets and real-world data sets illustrate the validity of our method.