Tensor Robust Principal Component Analysis (TRPCA), which aims to recover the low-rank and sparse components from their sum, has drawn intensive interest in recent years. Most existing TRPCA methods adopt the tensor nuclear norm (TNN) and the tensor ℓ1 norm as the regularization terms for the low-rank and sparse components, respectively. However, TNN treats each singular value of the low-rank tensor L equally and the tensor ℓ1 norm shrinks each entry of the sparse tensor S with the same strength. It has been shown that larger singular values generally correspond to prominent information of the data and should be less penalized. The same goes for large entries in S in terms of absolute values. In this paper, we propose a Double Auto-weighted TRPCA (DATRPCA) method. Instead of using predefined and manually set weights merely for the low-rank tensor as previous works, DATRPCA automatically and adaptively assigns smaller weights and applies lighter penalization to significant singular values of the low-rank tensor and large entries of the sparse tensorsimultaneously. We have further developed an efficient algorithm to implement DATRPCA based on the Alternating Direction Method of Multipliers (ADMM) framework. In addition, we have also established the convergence analysis of the proposed algorithm. The results on both synthetic and real-world data demonstrate the effectiveness of DATRPCA for low-rank tensor recovery, color image recovery and background modelling.

This article presents a generalized collaborative representation-based classification (GCRC) framework, which includes many existing representation-based classification (RC) methods, such as collaborative RC (CRC) and sparse RC (SRC) as special cases. This article also advances the GCRC theory by exploring theoretical conditions on the general regularization matrix. A key drawback of CRC and SRC is that they fail to use the label information of training data and are essentially unsupervised in computing the representation vector. This largely compromises the discriminative ability of the learned representation vector and impedes the classification performance. Guided by the GCRC theory, we propose a novel RC method referred to as discriminative RC (DRC). The proposed DRC method has the following three desirable properties: 1) discriminability: DRC can leverage the label information of training data and is supervised in both representation and classification, thus improving the discriminative ability of the representation vector; 2) efficiency: it has a closed-form solution and is efficient in computing the representation vector and performing classification; and 3) theory: it also has theoretical guarantees for classification. Experimental results on benchmark databases demonstrate both the efficacy and efficiency of DRC for multiclass classification.

Representation-based classification (RC) methods, such as sparse RC, have shown great potential in face recognition (FR) in recent years. Most previous RC methods are based on the conventional regression models, such as lasso regression, ridge regression, or group lasso regression. These regression models essentially impose a predefined assumption on the distribution of the noise variable in the query sample, such as the Gaussian or Laplacian distribution. However, the complicated noises in practice may violate the assumptions and impede the performance of these RC methods. In this paper, we propose a modal regression (MR)-based atomic representation and classification (MRARC) framework to alleviate such limitations. MR is a robust regression framework which aims to reveal the relationship between the input and response variables by regressing toward the conditional mode function. Atomic representation is a general atomic norm regularized linear representation framework which includes many popular representation methods, such as sparse representation, collaborative representation, and low-rank representation as special cases. Unlike previous RC methods, the MRARC framework does not require the noise variable to follow any specific predefined distributions. This gives rise to the capability of MRARC in handling various complex noises in reality. Using MRARC as a general platform, we also develop four novel RC methods for unimodal and multimodal FR, respectively. In addition, we devise a general optimization algorithm for the unified MRARC framework based on the alternating direction method of multipliers and half-quadratic theory. The experiments on real-world data validate the efficacy of MRARC for robust FR and reconstruction.

Representation-based classification (RC) methods such as sparse RC (SRC) have attracted great interest in pattern recognition recently. Despite their empirical success, few theoretical results are reported to justify their effectiveness. In this paper, we establish the theoretical guarantees for a general unified framework termed as atomic representation-based classification (ARC), which includes most RC methods as special cases. We introduce a new condition called atomic classification condition (ACC), which reveals important geometric insights for the theory of ARC. We show that under such condition ARC is provably effective in correctly recognizing any new test sample, even corrupted with noise. Our theoretical analysis significantly broadens the range of conditions under which RC methods succeed for classification in the following two aspects: (1) prior theoretical advances of RC are mainly concerned with the single SRC method while our theory can apply to the general unified ARC framework, including SRC and many other RC methods; and (2) previous works are confined to the analysis of noiseless test data while we provide theoretical guarantees for ARC using both noiseless and noisy test data. Numerical results are provided to validate and complement our theoretical analysis of ARC and its important special cases for both noiseless and noisy test data.

A joint sparse representation (JSR) method has shown superior performance for the classification of hyperspectral images (HSIs). However, it is prone to be affected by outliers in the HSI spatial neighborhood. In order to improve the robustness of JSR, we propose a maximum likelihood estimation (MLE)-based JSR (MLEJSR) model, which replaces the traditional quadratic loss function with an MLE-like estimator for measuring the joint approximation error. The MLE-like estimator is actually a function of coding residuals. Given some priors on the coding residuals, the MLEJSR model can be easily converted to an iteratively reweighted JSR problem. Choosing a reasonable weight function, the effect of inhomogeneous neighboring pixels or outliers can be dramatically reduced. We provide a theoretical analysis of MLEJSR from the viewpoint of recovery error and evaluate its empirical performance on three public hyperspectral data sets. Both the theoretical and experimental results demonstrate the effectiveness of our proposed MLEJSR method, especially in the case of large noise.

Various greedy algorithms have been developed for sparse signal recovery in recent years. However, most of them utilize the l2 norm based loss function and sensitive to non-Gaussian noises and outliers. This paper proposes a Cauchy matching pursuit (CauchyMP) algorithm for robust sparse representation and classification. By leveraging a Cauchy estimator based loss function, the proposed approach can robustly learn the sparse representation of noisy data corrupted by various severe noises. As a greedy algorithm, CauchyMP is also computationally efficient. We also develop a CauchyMP based classifier for robust classification with application to face recognition. The experiments on the datasets with gross corruptions demonstrate the efficacy and robustness of CauchyMP for learning robust sparse representation.

Big data research has become a globally hot topic in recent years. One of the core problems in big data learning is how to extract effective information from the huge data. In this paper, we propose a Markov resampling algorithm to draw useful samples for handling coefficient-based regularized regression (CBRR) problem. The proposed Markov resampling algorithm is a selective sampling method, which can automatically select uniformly ergodic Markov chain (u.e.M.c.) samples according to transition probabilities. Based on u.e.M.c. samples, we analyze the theoretical performance of CBRR algorithm and generalize the existing results on independent and identically distributed observations. To be specific, when the kernel is infinitely differentiable, the learning rate depending on the sample size $m$ can be arbitrarily close to $\mathcal {O}(m^{-1})$ under a mild regularity condition on the regression function. The good generalization ability of the proposed method is validated by experiments on simulated and real data sets.

As an efficient sparse representation algorithm, orthogonal matching pursuit (OMP) has attracted massive attention in recent years. However, OMP and most of its variants estimate the sparse vector using the mean square error criterion, which depends on the Gaussianity assumption of the error distribution. A violation of this assumption, e.g., non-Gaussian noise, may lead to performance degradation. In this paper, a correntropy matching pursuit (CMP) method is proposed to alleviate this problem of OMP. Unlike many other matching pursuit methods, our method is independent of the error distribution. We show that CMP can adaptively assign small weights on severely corrupted entries of data and large weights on clean ones, thus reducing the effect of large noise. Our another contribution is to develop a robust sparse representation-based recognition method based on CMP. Experiments on synthetic and real data show the effectiveness of our method for both sparse approximation and pattern recognition, especially for noisy, corrupted, and incomplete data.

Representation-based classifiers (RCs) including sparse RC (SRC) have attracted intensive interest in pattern recognition in recent years. In our previous work, we have proposed a general framework called atomic representation-based classifier (ARC) including many popular RCs as special cases. Despite the empirical success, ARC and conventional RCs utilize the mean square error (MSE) criterion and assign the same weights to all entries of the test data, including both severely corrupted and clean ones. This makes ARC sensitive to the entries with large noise and outliers. In this work, we propose an information-theoretic ARC (ITARC) framework to alleviate such limitation of ARC. Using ITARC as a general platform, we develop three novel representation-based classifiers. The experiments on public real-world datasets demonstrate the efficacy of ITARC for robust pattern recognition.

This note corrects an error in the proof of corollary 1 of Li et al. ( 2014 ). The original claim of the contraction principle in appendix D of Li et al. no longer holds.

Recently, a large family of representation-based classification methods have been proposed and attracted great interest in pattern recognition and computer vision. This paper presents a general framework, termed as atomic representation-based classifier (ARC), to systematically unify many of them. By defining different atomic sets, most popular representation-based classifiers (RCs) follow ARC as special cases. Despite good performance, most RCs treat test samples separately and fail to consider the correlation between the test samples. In this paper, we develop a structural ARC (SARC) based on Bayesian analysis and generalizing a Markov random field-based multilevel logistic prior. The proposed SARC can utilize the structural information among the test data to further improve the performance of every RC belonging to the ARC framework. The experimental results on both synthetic and real-database demonstrate the effectiveness of the proposed framework.

UNLABELLED: The previously known works studying the generalization ability of support vector machine classification (SVMC) algorithm are usually based on the assumption of independent and identically distributed samples. In this paper, we go far beyond this classical framework by studying the generalization ability of SVMC based on uniformly ergodic Markov chain (u.e.M.c.) samples. We analyze the excess misclassification error of SVMC based on u.e.M.c. samples, and obtain the optimal learning rate of SVMC for u.e.M.c. SAMPLES: We also introduce a new Markov sampling algorithm for SVMC to generate u.e.M.c. samples from given dataset, and present the numerical studies on the learning performance of SVMC based on Markov sampling for benchmark datasets. The numerical studies show that the SVMC based on Markov sampling not only has better generalization ability as the number of training samples are bigger, but also the classifiers based on Markov sampling are sparsity when the size of dataset is bigger with regard to the input dimension.

Representation-based classifiers (RCs) have attracted considerable attention in face recognition in recent years. However, most existing RCs use the mean square error (MSE) criterion as the cost function, which relies on the Gaussianity assumption of the error distribution and is sensitive to non-Gaussian noise. This may severely degrade the performance of MSE-based RCs in recognizing facial images with random occlusion and corruption. In this paper, we present a minimum error entropy-based atomic representation (MEEAR) framework for face recognition. Unlike existing MSE-based RCs, our framework is based on the minimum error entropy criterion, which is not dependent on the error distribution and shown to be more robust to noise. In particular, MEEAR can produce discriminative representation vector by minimizing the atomic norm regularized Renyi's entropy of the reconstruction error. The optimality conditions are provided for general atomic representation model. As a general framework, MEEAR can also be used as a platform to develop new classifiers. Two effective MEE-based RCs are proposed by defining appropriate atomic sets. The experimental results on popular face databases show that MEEAR can improve both the recognition accuracy and the reconstructed results compared with the state-of-the-art MSE-based RCs.

In this paper, we consider online support vector machine (SVM) classification learning algorithms with uniformly ergodic Markov chain (u.e.M.c.) samples. We establish the bound on the misclassification error of an online SVM classification algorithm with u.e.M.c. samples based on reproducing kernel Hilbert spaces and obtain a satisfactory convergence rate. We also introduce a novel online SVM classification algorithm based on Markov sampling, and present the numerical studies on the learning ability of online SVM classification based on Markov sampling for benchmark repository. The numerical studies show that the learning performance of the online SVM classification algorithm based on Markov sampling is better than that of classical online SVM classification based on random sampling as the size of training samples is larger.

Preference learning has caused great attention in machining learning. In this letter we propose a learning framework for pairwise loss based on empirical risk minimization of U-processes via Rademacher complexity. We first establish a uniform version of Bernstein inequality of U-processes of degree 2 via the entropy methods. Then we estimate the bound of the excess risk by using the Bernstein inequality and peeling skills. Finally, we apply the excess risk bound to the pairwise preference and derive the convergence rates of pairwise preference learning algorithms with squared loss and indicator loss by using the empirical risk minimization with respect to U-processes.

The large number of spectral bands acquired by hyperspectral imaging sensors allows us to better distinguish many subtle objects and materials. Unlike other classical hyperspectral image classification methods in the multivariate analysis framework, in this paper, a novel method using functional data analysis (FDA) for accurate classification of hyperspectral images has been proposed. The central idea of FDA is to treat multivariate data as continuous functions. From this perspective, the spectral curve of each pixel in the hyperspectral images is naturally viewed as a function. This can be beneficial for making full use of the abundant spectral information. The relevance between adjacent pixel elements in the hyperspectral images can also be utilized reasonably. Functional principal component analysis is applied to solve the classification problem of these functions. Experimental results on three hyperspectral images show that the proposed method can achieve higher classification accuracies in comparison to some state-of-the-art hyperspectral image classification methods.

This paper considers the generalization ability of two regularized regression algorithms [least square regularized regression (LSRR) and support vector machine regression (SVMR)] based on non-independent and identically distributed (non-i.i.d.) samples. Different from the previously known works for non-i.i.d. samples, in this paper, we research the generalization bounds of two regularized regression algorithms based on uniformly ergodic Markov chain (u.e.M.c.) samples. Inspired by the idea from Markov chain Monto Carlo (MCMC) methods, we also introduce a new Markov sampling algorithm for regression to generate u.e.M.c. samples from a given dataset, and then, we present the numerical studies on the learning performance of LSRR and SVMR based on Markov sampling, respectively. The experimental results show that LSRR and SVMR based on Markov sampling can present obviously smaller mean square errors and smaller variances compared to random sampling.

A sparsity-based model has led to interesting results in hyperspectral image (HSI) classification. Sparse representation from a test sample is used to identify the class label. However, an ℓ1-based sparse algorithm sometimes yields unstable sparse representation. Inspired by recent progress in manifold learning, two manifold-based sparse representation algorithms are proposed to exploit the local structure of the test samples in corresponding sparse representations for enforcing smoothness across neighboring samples' sparse representations. Using techniques from regularization and local invariance, two manifold-based regularization terms are incorporated into the ℓ1-based objective function. Extensive experiments show that our proposed algorithms obtain excellent classification performance on three classic HSIs.

In this letter, we consider a density-level detection (DLD) problem by a coefficient-based classification framework with [Formula: see text]-regularizer and data-dependent hypothesis spaces. Although the data-dependent characteristic of the algorithm provides flexibility and adaptivity for DLD, it leads to difficulty in generalization error analysis. To overcome this difficulty, an error decomposition is introduced from an established classification framework. On the basis of this decomposition, the estimate of the learning rate is obtained by using Rademacher average and stepping-stone techniques. In particular, the estimate is independent of the capacity assumption used in the previous literature.

Ranking is always an important task in machine learning and information retrieval, e.g., collaborative filtering, recommender systems, drug discovery, etc. A kernel-based stochastic gradient descent algorithm with the least squares loss is proposed for ranking in this paper. The implementation of this algorithm is simple, and an expression of the solution is derived via a sampling operator and an integral operator. An explicit convergence rate for leaning a ranking function is given in terms of the suitable choices of the step size and the regularization parameter. The analysis technique used here is capacity independent and is novel in error analysis of ranking learning. Experimental results on real-world data have shown the effectiveness of the proposed algorithm in ranking tasks, which verifies the theoretical analysis in ranking error.

In this paper, a hierarchical feature extraction method is proposed for image recognition. The key idea of the proposed method is to extract an effective feature, called local neural response (LNR), of the input image with nontrivial discrimination and invariance properties by alternating between local coding and maximum pooling operation. The local coding, which is carried out on the locally linear manifold, can extract the salient feature of image patches and leads to a sparse measure matrix on which maximum pooling is carried out. The maximum pooling operation builds the translation invariance into the model. We also show that other invariant properties, such as rotation and scaling, can be induced by the proposed model. In addition, a template selection algorithm is presented to reduce computational complexity and to improve the discrimination ability of the LNR. Experimental results show that our method is robust to local distortion and clutter compared with state-of-the-art algorithms.

Fisher linear discriminant (FLD) is a well-known method for dimensionality reduction and classification that projects high-dimensional data onto a low-dimensional space where the data achieves maximum class separability. The previous works describing the generalization ability of FLD have usually been based on the assumption of independent and identically distributed (i.i.d.) samples. In this paper, we go far beyond this classical framework by studying the generalization ability of FLD based on Markov sampling. We first establish the bounds on the generalization performance of FLD based on uniformly ergodic Markov chain (u.e.M.c.) samples, and prove that FLD based on u.e.M.c. samples is consistent. By following the enlightening idea from Markov chain Monto Carlo methods, we also introduce a Markov sampling algorithm for FLD to generate u.e.M.c. samples from a given data of finite size. Through simulation studies and numerical studies on benchmark repository using FLD, we find that FLD based on u.e.M.c. samples generated by Markov sampling can provide smaller misclassification rates compared to i.i.d. samples.

Elastic-net regularization is a successful approach in statistical modeling. It can avoid large variations which occur in estimating complex models. In this paper, elastic-net regularization is extended to a more general setting, the matrix recovery (matrix completion) setting. Based on a combination of the nuclear-norm minimization and the Frobenius-norm minimization, we consider the matrix elastic-net (MEN) regularization algorithm, which is an analog to the elastic-net regularization scheme from compressive sensing. Some properties of the estimator are characterized by the singular value shrinkage operator. We estimate the error bounds of the MEN regularization algorithm in the framework of statistical learning theory. We compute the learning rate by estimates of the Hilbert-Schmidt operators. In addition, an adaptive scheme for selecting the regularization parameter is presented. Numerical experiments demonstrate the superiority of the MEN regularization algorithm.

This paper presents a novel dictionary learning method for image denoising, which removes zero-mean independent identically distributed additive noise from a given image. Choosing noisy image itself to train an over-complete dictionary, the dictionary trained by traditional sparse coding methods contains noise information. Through mathematical derivation of equation, we found that a lower bound of dictionary is related with the level of noise in dictionary learning. The proposed idea is to take advantage of the noise information for designing a sparse coding algorithm called improved sparse coding (ISC), which effectively suppresses the noise influence for training a dictionary. This denoising framework utilizes the effective \nmethod, which is based on sparse representations over trained dictionaries. Acquiring an over-complete dictionary by ISC mainly includes three stages. Firstly, we utilize \nK-means method to group the noisy image patches. Secondly, each dictionary is trained by ISC in corresponding class. Finally, an over-complete dictionary is merged \nby these dictionaries. Theory analysis and experimental results both demonstrate that the proposed method yields excellent performance.

Local learning algorithm has been widely used in single-frame super-resolution reconstruction algorithm, such as neighbor embedding algorithm [1] and locality preserving constraints algorithm [2]. Neighbor embedding algorithm is based on manifold assumption, which defines that the embedded neighbor patches are contained in a single manifold. While manifold assumption does not always hold. In this paper, we present a novel local learning-based image single-frame SR reconstruction algorithm with kernel ridge regression (KRR). Firstly, Gabor filter is adopted to extract texture information from low-resolution patches as the feature. Secondly, each input low-resolution feature patch utilizes K nearest neighbor algorithm to generate a local structure. Finally, KRR is employed to learn a map from input low-resolution (LR) feature patches to high-resolution (HR) feature patches in the corresponding local structure. Experimental results show the effectiveness of our method.

In this paper, we propose a local semi-supervised learning-based algorithm for single-image super-resolution. Different from most of example-based algorithms, the information of test patches is considered during learning local regression functions which map a low-resolution patch to a high-resolution patch. Localization strategy is generally adopted in single-image super-resolution with nearest neighbor-based algorithms. However, the poor generalization of the nearest neighbor estimation decreases the performance of such algorithms. Though the problem can be fixed by local regression algorithms, the sizes of local training sets are always too small to improve the performance of nearest neighbor-based algorithms significantly. To overcome the difficulty, the semi-supervised regression algorithm is used here. Unlike supervised regression, the information about test samples is considered in semi-supervised regression algorithms, which makes the semi-supervised regression more powerful. Noticing that numerous test patches exist, the performance of nearest neighbor-based algorithms can be further improved by employing a semi-supervised regression algorithm. Experiments verify the effectiveness of the proposed algorithm.

Semisupervised learning has been of growing interest over the past years and many methods have been proposed. While existing semisupervised methods have shown some promising empirical performances, their development has been based largely on heuristics. In this paper, we investigate semisupervised multicategory classification with an imperfect mixture density model. In the proposed model, the training data come from a probability distribution, which can be modeled imperfectly by an identifiable mixture distribution. Furthermore, we propose a semisupervised multicategory classification method and establish its generalization error bounds. The theoretical analysis illustrates that the proposed method can utilize unlabeled data effectively and can achieve fast convergence rate.