This paper proposes a novel robust Fisher Discriminant Analysis (FDA) for discriminative subspace learning in the presence of outliers. The proposed approach is motivated by the maximum-likelihood perspective on FDA and its connection to Kullback-Leibler (KL) divergence minimization. Within the probabilistic FDA framework, we develop a robust model by adopting the $\alpha $ -divergence as a flexible alternative to the KL divergence. The resulting method induces an adaptive redescending weighting scheme, in which each observation is weighted according to its statistical compatibility with the model, where the robustness level continuously controlled by $\alpha $ . As $\alpha $ decreases from 1, the influence of outliers is progressively suppressed, while classical FDA is recovered at $\alpha = 1$ . Combined with a two-fold iterative optimization procedure, the proposed method mitigates contamination at both the class-modeling stage and the projection-learning stage. We further provide theoretical analysis of the robustness mechanism, convergence, and computational complexity analysis to support the effectiveness and efficiency of the proposed method. Extensive experiments on synthetic data and multiple public image datasets under diverse contamination settings demonstrate that the proposed method consistently outperforms representative robust FDA variants and related approaches.

Canonical correlation analysis (CCA) is a widely used multivariate analysis technique for explaining the relation between two sets of variables. It achieves this goal by finding linear combinations of the variables with maximal correlation. Recently, under the assumption that leading canonical directions are sparse, various penalized CCA procedures have been proposed for high dimensional data applications. However, all these procedures have the inconvenience of not preserving the sparsity among the retained leading canonical directions. To address this issue, two new sparse CCA methods are proposed in this paper. The first method is obtained by diagonal thresholding of two square matrices derived from the cross-covariance matrix of the two sets of variables where each matrix characterizes one set of variables. A model selection criterion is used to select the number of variables to retain from each matrix diagonal. The second method is derived within an adaptive alternating penalized least squares framework where the 1 2-norm is used as a penalty promoting block sparsity. Compared to existing sparse CCA methods, the proposed methods have the advantage of preserving the sparsity across the retained canonical loading vectors. Their performance are illustrated in an extended experimental study which shows the superior performance of the proposed methods.

In this paper, we address the problem of robust direction-of-arrival (DOA) estimation in unknown spatially cor-related noise fields using sensor arrays composed of subarrays in sparse configurations. In such arrays, the noise covariance matrix has a block-diagonal structure. The proposed robust DOA estimation method is derived from a parametric distribution divergence, α-divergence. Our approach can be viewed as an extension of existing Maximum Likelihood (ML) iterative procedures. The degree of robustness is controlled by the parameter α: as α→1, the proposed method converges to the traditional ML approach, while for α < 1, our method effectively mitigates the impact of potential outliers. Moreover, simulation studies show that our robust DOA estimation not only handles two types of outliers better than the ML method, but also exhibits high breakdown point properties.

Detecting small objects in optical images and videos is a significant challenge in numerous intelligent transportation and autonomous systems. State-of-the-art generic object detection methods fail to accurately localize and identify such small objects (e.g., pedestrians, small vehicles, obstacles). Because small objects occupy only a small area in the input image (e.g.,$32 \times 32$pixels or less), the information extracted from such a small area is not always rich enough to support decision-making. Multidisciplinary strategies are being developed by researchers working at the interface of deep learning and computer vision to enhance the performance of Small Object Detection (SOD). In this paper, we provide a comprehensive review of over 160 research papers published between 2017 and 2022 in order to survey this growing subject. This paper summarizes the existing literature and provides a taxonomy that illustrates the broad picture of current research. We further explore methods to boost the performance of small object detection in maritime settings, where enhanced performance is crucial for ensuring safety and managing traffic. Detecting small objects in the maritime environment requires additional considerations and the current survey aims to review the advanced techniques addressing those aspects. In addition, the popular SOD datasets for generic and maritime applications are discussed, and also well-known evaluation metrics for the state-of-the-art methods on some of the datasets are provided. The link to these datasets appears inhttps://github.com/arekavandi/Datasets_SOD.

Canonical correlation analysis (CCA) is a widely used mutivariate statistical technique for exploring the relationship between two multivariable datasets. It extracts existing relationship information by finding pairs of linear combinations from the two sets of variables with maximum correlation. In some applications however, the observed datasets may be contaminated by outliers and the standard CCA methods are sensitive to the presence of outliers in the datasets. In this paper, a robust CCA (RCCA) algorithm is presented. It is obtained using the interpretation of CCA as a latent variable model with two Gaussian random vectors and a robust loss function derived from the $\alpha$-divergence as an alternative to maximum likelihood. Compared to existing robust CCA approaches, the proposed loss has the advantage of belonging to class of redescending M-estimators, guaranteeing inference stability for large deviation from the Gaussian nominal noise model. Experimental results on simulated and real datasets show that the proposed RCCA outperforms some existing robust and standard CCA methods.

In this paper, novel robust principal component analysis (RPCA) methods are proposed to exploit the local structure of datasets. The proposed methods are derived by minimizing the α -divergence between the sample distribution and the Gaussian density model. The α- divergence is used in different frameworks to represent variants of RPCA approaches including orthogonal, non-orthogonal, and sparse methods. We show that the classical PCA is a special case of our proposed methods where the α- divergence is reduced to the Kullback-Leibler (KL) divergence. It is shown in simulations that the proposed approaches recover the underlying principal components (PCs) by down-weighting the importance of structured and unstructured outliers. Furthermore, using simulated data, it is shown that the proposed methods can be applied to fMRI signal recovery and Foreground-Background (FB) separation in video analysis. Results on real world problems of FB separation as well as image reconstruction are also provided.

In this paper, we generalize the well-known Expectation Maximization (EM) algorithm using the α−divergence for Gaussian Mixture Model (GMM). This approach is used in robust subspace detection when the number of parameters is kept small to avoid overfitting and large estimation variances. The level of robustness can be tuned by the parameter α. When α → 1, our method is equivalent to the standard EM approach and for α < 1 the method is robust against potential outliers. Simulation results show that the method outperforms the standard EM when it comes to mismatches between noise models and their realizations. In addition, we use the proposed method to detect active brain areas using collected functional Magnetic Resonance Imaging (fMRI) data during task-related experiments.

We studied the problem of robust subspace tracking (RST) in contaminated environments. Leveraging the fast approximated power iteration and α-divergence, a novel robust algorithm called αFAPI was developed for tracking the underlying principal subspace of streaming data over time. αFAPI is fast and it outperforms many RST methods while only having a low complexity linear to the data dimension. Some experiments were conducted to illustrate the performance of αFAPI.

This article addresses the robust estimation of the output layer linear parameters in a radial basis function network (RBFN). A prominent method used to estimate the output layer parameters in an RBFN with the predetermined hidden layer parameters is the least-squares estimation, which is the maximum-likelihood (ML) solution in the specific case of the Gaussian noise. We highlight the connection between the ML estimation and minimizing the Kullback-Leibler (KL) divergence between the actual noise distribution and the assumed Gaussian noise. Based on this connection, a method is proposed using a variant of a generalized KL divergence, which is known to be more robust to outliers in the pattern recognition and machine-learning problems. The proposed approach produces a surrogate-likelihood function, which is robust in the sense that it is adaptive to a broader class of noise distributions. Several signal processing experiments are conducted using artificially generated and real-world data. It is shown that in all cases, the proposed adaptive learning algorithm outperforms the standard approaches in terms of mean-squared error (MSE). Using the relative increase in the MSE for different noise conditions, we compare the robustness of our proposed algorithm with the existing methods for robust RBFN training and show that our method results in overall improvement in terms of absolute MSE values and consistency.

Robust variants of Wald, Rao and likelihood ratio (LR) tests for the detection of a signal subspace in a signal interference subspace corrupted by contaminated Gaussian noise are proposed in this paper. They are derived using the α- divergence, and the trade-off between the robustness and the power (the probability of detection) of the tests is adjustable using a single hyperparameter α . It is shown that when α→ 1 , these tests are equivalent to their well known classical counterparts. For example the robust LR test coincides with the LR test or the matched subspace detector (MSD). Asymptotic results are provided to support the proposed tests and robustness to outliers is obtained using values of . Numerical experiments illustrating the performance of these tests on simulated, real functional magnetic resonance imaging (fMRI), hyperspectral and synthetic aperture radar (SAR) data are also presented.

The problem of detecting a subspace signal in colored Gaussian noise with unknown covariance matrix is investigated when the training data may contain samples with target signal. The target signal is assumed that it lies in a subspace spanned by columns of a known matrix. To develop the test, an ad hoc approach, similar to the classical adaptive matched filter (AMF) is used where instead of the maximum likelihood (ML) estimator of the covariance, the minimum α-divergence based estimator is substituted in the likelihood ratio. This test just depends on the single parameter α and as a special case can be turned to the AMF. For a range of α, the proposed test has the benefits of being robust to outliers and the existence of other targets in the training data. Numerical examples illustrating that the proposed detector can achieve better detection rates in such a scenario while providing almost the same performance in a target free scenario are presented.

The problem of detecting a subspace signal in the presence of subspace interference and contaminated Gaussian noise with unknown variance is investigated. The target signal is assumed to lie in a subspace spanned by the columns of a known matrix. To develop the test, the same steps used in the generalized likelihood ratio test (GLRT) are used where instead of the maximum likelihood (ML) estimator of the parameters, the minimum α-divergence based estimator is substituted in the test to increase the robustness of the test against contaminations in noise. This test depends on the single parameter α and as the special case corresponds to the well known GLRT. Numerical examples illustrating that the proposed test can achieve better detection rates in such scenarios are presented. Moreover, the test is applied to real fMRI dataset to detect the active area of the brain for some task-related inputs.

In this paper, a new robust principal component analysis (RPCA) method which enables us to exploit the main components of a given corrupted data with non Gaussian outliers is proposed. This method is based on the α-divergence which is a parametric measure from information geometry. The proposed method is adjustable using a hyperparameter α and reduces to the classical PCA as a particular case. In order to derive the main components, the α-divergence between the empirical data distribution and the assumed model for the distribution is minimized with respect to the unknown parameters. The singular value decomposition (SVD) of the estimated covariance matrix is then used to exploit the main direction of the data. The proposed method is applied to some video and signal processing applications and the results show the superiority of the proposed method over classical PCA and other existing robust methods.

Dictionary learning algorithms have been successfully used to solve a variety of signal and image processing problems. In some applications however, the observed signals may be contaminated by outliers and have a multi-subpsace structure that enables block-sparse signal representations. Based on the observation that the observed signals can be approximated as a sum of low rank matrices, a new algorithm for learning a block-structured dictionary in the presence of outliers is proposed. The proposed algorithm is obtained using a robust α-divergence based data fitting term in the algorithm cost function and derived via sequential penalized low rank matrix approximation. Experimental results illustrating the performance of the proposed algorithm compared to some state-of-the-art algorithms are provided.

Canonical correlation analysis (CCA) describes the relationship between two sets of variables by finding linear combinations of the variables with maximal correlation. Recently, under the assumption that the leading canonical correlation directions are sparse, various procedures have been proposed for many high-dimensional applications to improve the interpretability of CCA. However all these procedures have the inconvenience of not preserving the sparsity among the retained leading canonical directions. To address this issue, a new sparse CCA method is proposed in this paper. It is derived within a penalized alternative least squares framework where the 12-norm is used as a penalty promoting block sparsity. The proposed method has the advantage of generating the same sparsity pattern across all retained canonical correlation components. The performance of the proposed method is demonstrated in a simulation study.

In this paper, a robust sequential dictionary learning (DL) algorithm is presented. It is obtained by using a robust loss function in the data fidelity term of the DL objective instead of the usual quadratic loss. The proposed robust loss function is derived from the α-divergence as an alternative to the Kullback-Leibler divergence which leads to a quadratic loss. Compared to other robust approaches, the proposed loss has the advantage of belonging to class of redescending M-estimators, guaranteeing inference stability for large deviation from the Gaussian nominal noise model. The algorithm is derived via adaptive sequential penalized rank-l matrix approximation using a block coordinate descent approach to obtain the vector pairs of different rank-1 matrices. Performance comparison with similar robust DL algorithms on digit recognition highlights efficacy of the proposed algorithm.

Adaptive subspace detectors (ASD) generalize matched subspace detectors (MSD) by accounting for possible correlation. Both ASD and MSD are derived using the generalized likelihood ratio test (GLRT). While MSD assumes there is no correlation between observations, ASD estimates a sample covariance matrix of possibly correlated samples using signal-free observations. In this paper, we address the performance of the ASD when the number of secondary data is insufficient and the observed signal lies in higher dimensional space. Such high dimensional spaces are frequently encountered in functional magnetic resonance imaging (fMRI) data for the analysis of brain activation detection. We propose a methodology that works based on the latent variables in a lower dimensional space. A low-rank decomposition of the sample covariance matrix is derived based on the singular value decomposition (SVD) and an adaptive basis selection method is used to decide which eigen-vectors are useful in data projection. Performing detection in the lower dimensional subspace has the benefit of reducing the number of parameters which need to be estimated. Simulation results show superiority of our proposed adaptive reduced subspace detector (ARSD) over conventional ASD in term of probability of detection.

Dictionary learning algorithms have been successfully applied to a number of signal and image processing problems. In some applications however, the observed signals may have a multi-subpsace structure that enables block-sparse signal representations. Based on the observation that the observed signals can be approximated as a sum of low rank matrices, a new algorithm for learning a block-structured dictionary for block-sparse signal representations is proposed. It's derived via sequential penalized low rank matrix approximation, where a block coordinate descent approach is used to estimate the matrix pairs that form the different low rank matrix approximations. Experimental results on synthetic and standard gray-scale images illustrating the performance of the proposed algorithm are provided.

Functional Near-InfraRed Spectroscopy (fNIRS) has gained widespread acceptance as a non-invasive neuroimaging modality for monitoring functional brain activities. fNIRS uses light in the near infra-red spectrum (600-900 nm) to penetrate human brain tissues and estimates the oxygenation conditions based on the proportion of light absorbed. In order to get reliable results, artefacts and noise need to be separated from fNIRS physiological signals. This paper focuses on removing motion-related artefacts. A new motion artefact removal algorithm based on robust parameter estimation is proposed. Results illustrate that the proposed algorithm can outperform the state-of-art algorithms in removing motion artefacts. Moreover, the proposed algorithm is robust in estimating the parameters under different interference conditions.

High dimensional data and systems with many degrees of freedom are often characterized by covariance matrices. In this paper, we consider the problem of simultaneously estimating the dimension of the principal (dominant) subspace of these covariance matrices and obtaining an approximation to the subspace. This problem arises in the popular principal component analysis (PCA), and in many applications of machine learning, data analysis, signal and image processing, and others. We first present a novel method for estimating the dimension of the principal subspace. We then show how this method can be coupled with a Krylov subspace method to simultaneously estimate the dimension and obtain an approximation to the subspace. The dimension estimation is achieved at no additional cost. The proposed method operates on a model selection framework, where the novel selection criterion is derived based on random matrix perturbation theory ideas. We present theoretical analyses which (a) show that the proposed method achieves strong consistency (i.e., yields optimal solution as the number of data-points n → ∞), and (b) analyze conditions for exact dimension estimation in the finite n case. Using recent results, we show that our algorithm also yields near optimal PCA. The proposed method avoids forming the sample covariance matrix (associated with the data) explicitly and computing the complete eigen-decomposition. Therefore, the method is inexpensive, which is particularly advantageous in modern data applications where the covariance matrices can be very large. Numerical experiments illustrate the performance of the proposed method in various applications.

In this paper, a robust sequential dictionary learning (DL) algorithm is presented. The proposed algorithm is motivated from the maximum likelihood perspective on dictionary learning and its link to the minimization of the Kullback-Leibler divergence. It is obtained by using a robust loss function in the data fidelity term of the DL objective instead of the usual quadratic loss. The proposed robust loss function is derived from the α-divergence as an alternative to the Kullback-Leibler divergence, which leads to a quadratic loss. Compared to other robust approaches, the proposed loss has the advantage of belonging to class of redescending M-estimators, guaranteeing inference stability from large deviations from the Gaussian nominal noise model. The algorithm is obtained by solving a sequence of penalized rank-1 matrix approximation problems, where the ℓ1-norm is introduced as a penalty promoting sparsity and then using a block coordinate descent approach to estimate the unknowns. Performance comparison with similar robust DL algorithms on digit recognition, background removal, and gray-scale image denoising is performed highlighting efficacy of the proposed algorithm.

Principal component analysis (PCA) is widely used for feature extraction and dimension reduction in pattern recognition and data analysis. Despite its popularity, the reduced dimension obtained from the PCA is difficult to interpret due to the dense structure of principal loading vectors. To address this issue, several methods have been proposed for sparse PCA, all of which estimate loading vectors with few non-zero elements. However, when more than one principal component is estimated, the associated loading vectors do not possess the same sparsity pattern. Therefore, it becomes difficult to determine a small subset of variables from the original feature space that have the highest contribution in the principal components. To address this issue, an adaptive block sparse PCA method is proposed. The proposed method is guaranteed to obtain the same sparsity pattern across all principal components. Experiments show that applying the proposed sparse PCA method can help improve the performance of feature selection for image processing applications. We further demonstrate that our proposed sparse PCA method can be used to improve the performance of blind source separation for functional magnetic resonance imaging data.

Data-driven methods, such as principal component analysis and independentcomponent analysis, have been successfully applied to functionalmagnetic resonance imaging (fMRI) data in particular and neuro-imaging data in general. A central issue of thesemethods is the importance of correctly selecting the number of components to be used in the factor model. This issue is often addressed using a model selection criterion, where the goodness-of-fit term is obtained from the log-likelihood function. In this paper, an alternative criterion is proposed for selecting the number of components. Unlike existingmodel selection criteria that use the log-likelihood function, the proposed goodness-of-fit termuses the sum of squares of the smallest eigenvalues of the sample covariance matrix. The proposed criterion is obtained from the asymptotic distribution of the goodness-of-fit term, for which consistency is established. This criterion has a straight-forward implementation and is shown to outperform conventional model selection criteria used in fMRI data analysis. Experiments are conducted using simulated and real fMRI data, in which improved performance is obtained by the proposed criterion, both in terms of accuracy and consistency under data variabilities.

In recent years, data driven methods have been successfully used for analyzing multi-subject functional magnetic resonance imaging (fMRI) datasets. These methods attempt to learn shared spatial activation maps (SM) or voxel time courses (TC) from temporally or spatially concatenated fMRI datasets respectively. Most of the methods proposed so far do not distinguish whether a particular SM/TC is a group level component or only present in a certain subject dataset. In this paper we present a new two stage algorithm which aims to separate the joint and sub-specific information from the temporally concatenated multi-subject datasets. The proposed method is based on the singular value decomposition (SVD) and penalized rank-one matrix approximation. Simulation experiments are used to demonstrate this ability of the proposed algorithm followed by validation on real experimental task-fMRI datasets.

In recent history, dictionary learning (DL) methods have been successfully used for analyzing multi-subject functional magnetic resonance imaging. These algorithms try to learn group-level spatial activation maps (SM) or voxel time courses (TC) from temporally or spatially concatenated fMRI datasets respectively. However, in multi-subject fMRI studies, we are interested in both group-level TCs as well as SMs. In this paper, we propose a DL algorithm which combines temporally and spatially concatenated fMRI datasets to learn not only the shared TC/SM pairs but also the subject-specific ones. We do this by separating group-level information and sub-specific information from each subject fMRI dataset. Performance of the proposed algorithm is illustrated using simulated as well as experimental task fMRI datasets.

Canonical correlation analysis (CCA) is a data driven method that has been successfully used in functional magnetic resonance imaging (fMRI) data analysis. Standard CCA extracts meaningful information from a data set by seeking pairs of linear combinations from two sets of variables with maximum pairwise correlation. So far, however, this method has been used without incorporating prior information available for fMRI data. In this paper, we address this issue by proposing a new CCA method named PCCA (for projection CCA). PCCA is obtained by using the discrete cosine transform (DCT) to create a basis for a span that better characterizes the fMRI data set. Employing DCT guides the estimated canonical variates, yielding a more computationally efficient CCA procedure. The proposed method can be seen as a regularized CCA method where regularization is introduced via basis expansion. The advantages of the proposed PCCA algorithm over the standard CCA are illustrated on both simulated data and real fMRI data from a resting state experiment.

This letter deals with a hybrid joint diagonalization problem considering both Hermitian and transpose congruence. Such problem can be encountered in certain noncircular signal analysis applications including blind source separation. We introduce new Jacobi-like algorithms using Givens or a combination of Givens and hyperbolic rotations. These algorithms are compared with state-of-the-art methods and their performance gain, especially in the high dimensional case, is assessed through simulation experiments including examples related to blind separation of noncircular sources.

We address the problem of two-dimensional (2-D) whitening of face image matrices. The proposed method whitens the distribution of rows and columns of an image matrix. The main contribution of this letter is to investigate some aspects of recently published studies on this topic. We point out that existing methods on 2-D whitening for face recognition overlook the rows as potential random vectors, implying that the image matrix is only a collection of independent identically distributed column vectors. This study shows that this one-sided whitening approach relies on an incorrect assumption for image data. We show that one-sided whitening replaces the covariance matrix along one dimension with the identity matrix. Our proposed, truely 2-D, whitening transform does not enforce any such restriction on the row- or column-covariance matrices. A second contribution of our letter is to illustrate a method to confirm that the rows and columns of front-pose face images are in fact approximately normally distributed. This validation has not been addressed previously, although it plays a crucial role in deriving a linear whitening transform.

Multiple-set canonical correlation analysis (mCCA) is a generalization of canonical correlation analysis (CCA) to three or more sets of variables. It aims to study the relationships between several sets of variables and it subsumes a number of interesting multivariate data analysis techniques as special cases. The quality and interpretability of the mCCA components are likely to be affected by the usefulness and relevance of each set of variables. Therefore, it is an important issue to identify each set of significant variables that are active in the relationships between sets. In this paper mCCA is extended to address the issue of variable set selection. Specifically a block sparse multiple set canonical correlation analysis (BSmCCA) algorithm is proposed to combine mCCA with ℓ2-norm type penalty in a unified framework. Within this framework sets of variables that are not necessarily relevant are removed. This makes BSmCCA a flexible method for analyzing for Multi-Subject functional magnetic resonance imaging (fMRI) data sets. The performances of the proposed BSmCCA algorithm are illustrated through on block design paradigm finger taping fMRI datasets.

Sequential dictionary learning algorithms has gained widespread acceptance in functional magnetic resonance imaging (fMRI) data analysis. However, many problems in fMRI data analysis involve the analysis of multiple-subject fMRI data sets and the existing algorithms do not extend naturally to this case. In this paper we propose an algorithm dedicated to multiple-subject fMRI data analysis. The algorithm is named SMSDL for sequential multi-subject dictionary learning and differs from existing dictionary learning algorithms in its dictionary update stage. This algorithm is derived by using a variation of the power algorithm in the dictionary update stage to extract the common information among the multiple-subject fMRI data sets. The results of the proposed dictionary learning algorithm is a set of time courses which are common to the whole group of subjects and an individual spatial response pattern for each of the subjects in the group. The performance of the proposed algorithm are illustrated through a simulation and an application on real fMRI datasets.

This letter considers the problem of dictionary learning for sparse signal representation whose atoms have low mutual coherence. To learn such dictionaries, at each step, we first update the dictionary using the method of optimal directions (MOD) and then apply a dictionary rank shrinkage step to decrease its mutual coherence. In the rank shrinkage step, we first compute a rank 1 decomposition of the column-normalized least squares estimate of the dictionary obtained from the MOD step. We then shrink the rank of this learned dictionary by transforming the problem of reducing the rank to a nonnegative garrotte estimation problem and solving it using a path-wise coordinate descent approach. We establish theoretical results that show that the rank shrinkage step included will reduce the coherence of the dictionary, which is further validated by experimental results. Numerical experiments illustrating the performance of the proposed algorithm in comparison to various other well-known dictionary learning algorithms are also presented.

Many machine learning and data-related applications require the knowledge of approximate ranks of large data matrices at hand. This letter presents two computationally inexpensive techniques to estimate the approximate ranks of such matrices. These techniques exploit approximate spectral densities, popular in physics, which are probability density distributions that measure the likelihood of finding eigenvalues of the matrix at a given point on the real line. Integrating the spectral density over an interval gives the eigenvalue count of the matrix in that interval. Therefore, the rank can be approximated by integrating the spectral density over a carefully selected interval. Two different approaches are discussed to estimate the approximate rank, one based on Chebyshev polynomials and the other based on the Lanczos algorithm. In order to obtain the appropriate interval, it is necessary to locate a gap between the eigenvalues that correspond to noise and the relevant eigenvalues that contribute to the matrix rank. A method for locating this gap and selecting the interval of integration is proposed based on the plot of the spectral density. Numerical experiments illustrate the performance of these techniques on matrices from typical applications.

Sequential dictionary learning via the K-SVD algorithm has been revealed as a successful alternative to conventional data driven methods, such as independent component analysis for functional magnetic resonance imaging (fMRI) data analysis. fMRI data sets are however structured data matrices with notions of spatio-temporal correlation and temporal smoothness. This prior information has not been included in the K-SVD algorithm when applied to fMRI data analysis. In this paper, we propose three variants of the K-SVD algorithm dedicated to fMRI data analysis by accounting for this prior information. The proposed algorithms differ from the K-SVD in their sparse coding and dictionary update stages. The first two algorithms account for the known correlation structure in the fMRI data by using the squared Q, R-norm instead of the Frobenius norm for matrix approximation. The third and last algorithms account for both the known correlation structure in the fMRI data and the temporal smoothness. The temporal smoothness is incorporated in the dictionary update stage via regularization of the dictionary atoms obtained with penalization. The performance of the proposed dictionary learning algorithms is illustrated through simulations and applications on real fMRI data.

Sequential dictionary learning algorithms have been successfully applied to functional magnetic resonance imaging (fMRI) data analysis. fMRI data sets are, however, structured data matrices with the notions of temporal smoothness in the column direction. This prior information, which can be converted into a constraint of smoothness on the learned dictionary atoms, has seldomly been included in classical dictionary learning algorithms when applied to fMRI data analysis. In this paper, we tackle this problem by proposing two new sequential dictionary learning algorithms dedicated to fMRI data analysis by accounting for this prior information. These algorithms differ from the existing ones in their dictionary update stage. The steps of this stage are derived as a variant of the power method for computing the SVD. The proposed algorithms generate regularized dictionary atoms via the solution of a left regularized rank-one matrix approximation problem where temporal smoothness is enforced via regularization through basis expansion and sparse basis expansion in the dictionary update stage. Applications on synthetic data experiments and real fMRI data sets illustrating the performance of the proposed algorithms are provided.

Canonical correlation analysis (CCA) is a well-known technique used to characterize the relationship between two sets of multidimensional variables by finding linear combinations of variables with maximal correlation. Sparse CCA or regularized CCA are two widely used variants of CCA because of the improved interpretability of the former and the better performance of the later. So far the cross-matrix product of the two sets of multidimensional variables has been widely used for the derivation of these variants. In this paper a new algorithm for sparse CCA is proposed. This algorithm differs from the existing ones in their derivation which is based on penalized rank one matrix approximation and the orthogonal projectors onto the space spanned by the two sets of multidimensional variables instead of the simple cross-matrix product. The performance and effectiveness of the proposed algorithm are tested on simulated experiments. On these results it can be observed that they outperform the state of the art sparse CCA algorithms.

Sequential dictionary learning via the K-SVD algorithm has been revealed as a successful alternative to conventional data driven methods such as independent component analysis (ICA) for functional magnetic resonance imaging (fMRI) data analysis. fMRI data sets are however structured data matrices with notions of spatio-temporal correlation. This prior information has not been included in the K-SVD algorithm when applied in fMRI data analysis. In this paper we remedy to this situation by proposing a variant of the K-SVD algorithm dedicated to fMRI data analysis by taking into account this prior information. The proposed algorithm accounts for the known correlation structure in the fMRI data by using the squared Q, R-norm instead of the Frobenius norm for rank one approximation in the dictionary update stage. The performance of the proposed algorithm is illustrated through simulations and applications on a real fMRI data set.

This paper addresses the problem of scanner induced low frequency drift estimation in order to improve the significance of functional magnetic resonance imaging (fMRI) data for statistical analysis. A novel technique is presented to estimate the drift parameters using a sparse general linear model (sGLM) framework. The fMRI signal is modeled as a linear mixture of several signals such as low frequency trend, brain hemodynamic, physiological noise and unexplained signal variations. These signals are considered as underlying sources and sparse dictionary learning (SDL) is used to estimate them. The superior performance of the proposed technique compared to other detrending techniques is illustrated using a simulation study. Furthermore, the proposed technique is validated using real fMRI data, which shows its better capability to estimate drift in presence of spatiotemporal dependencies.

Dictionary learning algorithms have received widespread acceptance when it comes to data analysis and signal representations problems. These algorithms alternate between two stages: the sparse coding stage and dictionary update stage. In all existing dictionary learning algorithms the use of sparsity has been limited to the sparse coding stage while presenting differences in the dictionary update stage which can be achieved sequentially or in parallel. The singular value decomposition (SVD) has been successfully used for sequential dictionary update. In this paper we propose a dictionary learning algorithm that include a sparsity constraint also in the dictionary update stage. The cost function used to include sparsity in the dictionary update stage is derived using the link between SVD and rank one matrix approximation. The effectiveness of the proposed dictionary learning method is tested on synthetic data and an image processing application. The results reveal that including a sparsity constraint in the dictionary update stage is not a bad idea.

We consider the problem of selecting the optimal orders of vector autoregressive (VAR) models for fMRI data. Many previous studies used model order of one and ignored that it may vary considerably across data sets depending on different data dimensions, subjects, tasks, and experimental designs. In addition, the classical information criteria (IC) used (e.g., the Akaike IC (AIC)) are biased and inappropriate for the high-dimensional fMRI data typically with a small sample size. We examine the mixed results on the optimal VAR orders for fMRI, especially the validity of the order-one hypothesis, by a comprehensive evaluation using different model selection criteria over three typical data types--a resting state, an event-related design, and a block design data set--with varying time series dimensions obtained from distinct functional brain networks. We use a more balanced criterion, Kullback's IC (KIC) based on Kullback's symmetric divergence combining two directed divergences. We also consider the bias-corrected versions (AICc and KICc) to improve VAR model selection in small samples. Simulation results show better small-sample selection performance of the proposed criteria over the classical ones. Both bias-corrected ICs provide more accurate and consistent model order choices than their biased counterparts, which suffer from overfitting, with KICc performing the best. Results on real data show that orders greater than one were selected by all criteria across all data sets for the small to moderate dimensions, particularly from small, specific networks such as the resting-state default mode network and the task-related motor networks, whereas low orders close to one but not necessarily one were chosen for the large dimensions of full-brain networks.

We consider the problem of identifying large-scale effective connectivity of brain networks from fMRI data. Standard vector autoregressive (VAR) models fail to estimate reliably networks with large number of nodes. We propose a new method based on factor modeling for reliable and efficient high-dimensional VAR analysis of large networks. We develop a subspace VAR (SVAR) model from a factor model (FM), where observations are driven by a lower-dimensional subspace of common latent factors with an AR dynamics. We consider two variants of principal components (PC) methods that provide consistent estimates for the FM hence the implied SVAR model, even of large dimensions. Information criterion is used to select the optimal subspace dimension. We established asymptotic normality and convergence rates for the estimated SVAR coefficients matrix. Evaluation on simulated resting-state fMRI shows that the SVAR models are more robust and produce better connectivity estimates than the classical model for a moderately-large network analysis. Results on real data by varying the subspace dimensions identify strong connections in the default mode network and reveal hierarchical connectivity of resting-state networks with distinct functional relevance.

Functional near-infrared spectroscopy (fNIRS) signals offer an interesting alternative to functional magnetic resonance imaging (fMRI) when investigating the temporal dynamics of brain region responses during activations. The hemodynamic response function (HRF) is the object of primary interest to neuroscientists in this case. Making use of a semiparametric model to characterize the oxygenated (HbO) and deoxygenated (HbR) fNIRS time-series and a sparsity assumption on the HRF, a new method for non-parametric HRF estimation from a single fNIRS signal is derived in this paper. The proposed method consistently estimates the HRF using a profile least square estimator obtained using the local polynomial smoothing technique applied to estimate the drift and introducing a regularization penalty in the minimization problem to promote sparsity of the HRF coefficients. The performance of the proposed method is assessed on both simulated and fNIRS data from a finger tapping experiment.

Blurring is a common source of image degradation in many applications. Blind image deblurring (BID) is an apposite approach for blur removal in real images. Being an ill-posed linear inverse problem, a regularized and well constrained approach is required for a credible solution of BID model. Recently sparse representation base modeling emerged as an efficacious tool in image processing community, with application as regularizer in inverse problems. In this work the sparsity constraint is fused with the non-negative matrix approximation to address the BID problem. An alternative-iterative frame work is developed to estimate the non-negative sparse approximation of the sharp image and blurring kernel. With sparsity constraint, an estimate of the sharp image is obtained without solving the ill-posed deconvolution model. Although similar formulation has been proposed but unlike other BID methods the proposed approach is parameter free and requires no prior statistics. The experimental results validate comparatively better performance of proposed method against the other methods.

Image restoration is a significant inverse problem in image processing community. We present an iterative alternating minimization of Kullback Leibler divergence (KLD) for an optimized image denoising. It is obtained by modeling the original image and the additive noise as multivariate Gaussian processes with unknown covariance matrices in wavelet domain. The original image and noise parameters are estimated by minimizing KLD between a model family of probability distributions defined using the linear image degradation model and a desired family of probability distributions constrained to be concentrated on the observed noisy image. The wavelet coefficients are modeled using the class of Gaussian Scale Mixture (GSM), which represents the heavy-tailed statistical distribution, suitable for natural images. The algorithm provides closed form expressions for the parameters updates and converge only in few iterations. The efficiency of proposed method is demonstrated through numerical simulations, both visually and in terms of signal to noise ratio.

This letter proposes an algorithm for linear whitening that minimizes the mean squared error between the original and whitened data without using the truncated eigendecomposition (ED) of the covariance matrix of the original data. This algorithm uses Lanczos vectors to accurately approximate the major eigenvectors and eigenvalues of the covariance matrix of the original data. The major advantage of the proposed whitening approach is its low computational cost when compared with that of the truncated ED. This gain comes without sacrificing accuracy, as illustrated with an experiment of whitening a high-dimensional fMRI data set.

Nonparametric hemodynamic response function (HRF) estimation in functional near-infrared spectroscopy (fNIRS) data plays an important role when investigating the temporal dynamics of a brain region response during activations. Assuming the drift arising from both physical and physiological effects in fNIRS data is Lipschitz continuous; a novel algorithm for joint HRF and drift estimation is derived in this paper. The proposed algorithm estimates the HRF by applying a first-order differencing to the fNIRS time series samples in order to remove the drift effect. An estimate of the drift is then obtained using a wavelet thresholding technique applied to the residuals generated by removing the estimated induced activation response from the fNIRS time-series. It is shown that the proposed HRF estimator is √N consistent whereas the estimator of the drift is asymptotically optimal. The de-drifted fNIRS oxygenated (HbO) and deoxygenated (HbR) hemoglobin responses are then obtained by removing the corresponding estimated drifts from the fNIRS time-series. Its performance is assessed using both simulated and real fNIRS data sets. The application results reveal that the proposed joint HRF and drift estimation method is efficient both computationally and in terms of accuracy. In comparison to traditional model based methods used for HRF estimation, the proposed novel method avoids the selection of a model to remove the drift component. As a result, the proposed method finds an optimal estimate of the fNIRS drift and offers a model-free approach to de-drift the HbO/HbR responses.

Non-parametric hemodynamic response function (HRF) estimation in noisy functional near-infrared spectroscopy (fNIRS) plays an important role when investigating the temporal dynamics of a brain region response during activations. Assuming the drift Lipschitz continuous; a new algorithm for non-parametric HRF estimation from the oxygenated (HbO) and deoxygenated (HbR) fNIRS time-series is derived in this paper. The proposed algorithm estimates the HRF by applying a first order differencing to the fNIRS time series samples. It is shown that the proposed HRF estimator is √N consistent. Its performance is assessed using both simulated and a real fNIRS data set obtained from a motor activity experiment. The application results reveal that the proposed HRF estimation method is efficient both computationally and in terms of accuracy.

Blurred image restoration is a longstanding and critical research problem. We addressed this problem using Expectation Maximization (EM) based approach in wavelet domain. The sparsity property of wavelet coefficients is modeled using the class of Gaussian Scale Mixture (GSM), which represents the heavy-tailed statistical distribution, suitable for natural images. The underlying original image and noise parameters are estimated by alternating EM iterations based on available and hidden data sets, where regularization is introduced using an intermediate variable. Although similar formulations have been proposed before but the resulting optimization problems have been computationally demanding, where our formulation is simple to implement and converge in few iterations. Simulation results are presented to demonstrate the quality of our method both visually and in terms of signal to noise ratio improvement.

Detecting and characterizing causal interdependencies and couplings between different activated brain areas from functional neuroimage time series measurements of their activity constitutes a significant step toward understanding the process of brain functions. In this letter, we make the simple point that all current statistics used to make inferences about directed influences in functional neuroimage time series are variants of the same underlying quantity. This includes directed transfer entropy, transinformation, Kullback-Leibler formulations, conditional mutual information, and Granger causality. Crucially, in the case of autoregressive modeling, the underlying quantity is the likelihood ratio that compares models with and without directed influences from the past when modeling the influence of one time series on another. This framework is also used to derive the relation between these measures of directed influence and the complexity or the order of directed influence. These results provide a framework for unifying the Kullback-Leibler divergence, Granger causality, and the complexity of directed influence.

Hemodynamic response function (HRF) estimation in noisy functional magnetic resonance imaging (fMRI) plays an important role when investigating the temporal dynamic of a brain region response during activations. Nonparametric methods which allow more flexibility in the estimation by inferring the HRF at each time sample have provided improved performance in comparison to the parametric methods. In this paper, the mixed-effects model is used to derive a new algorithm for nonparametric maximum likelihood HRF estimation. In this model, the random effect is used to better account for the variability of the drift. Contrary to the usual approaches, the proposed algorithm has the benefit of considering an unknown and therefore flexible drift matrix. This allows the effective representation of a broader class of drift signals and therefore the reduction of the error in approximating the drift component. Estimates of the HRF and the hyperparameters are derived by iterative minimization of the Kullback-Leibler divergence between a model family of probability distributions defined using the mixed-effects model and a desired family of probability distributions constrained to be concentrated on the observed data. The performance of proposed method is demonstrated on simulated and real fMRI data, the latter originating from both event-related and block design fMRI experiments.

Existing polyp detection methods rely heavily on curvature-based characteristics to differentiate between lesions. These assume that the discrete triangulated surface mesh or volume closely approximates a smooth continuous surface. However, this is often not the case and because curvature is computed as a local feature and a second-order differential quantity, the presence of noise significantly affects its estimation. For this reason, a more global feature is required to provide an accurate description of the surface at hand. In this paper, a novel method incorporating a local neighborhood around the centroid of a surface patch is proposed. This is done using geodesic rings which accumulate curvature information in a neighborhood around this centroid. This geodesic-ring neighborhood approximates a single smooth, continuous surface upon which curvature and orientation estimation methods can be applied. A new global shape index, S is also introduced and computed. These curvature and orientation values will be used to classify the surface as either a bulbous polyp, ridge-like fold or semiplanar structure. Experimental results show that this method is promising (100% sensitivity, 100% specificity for lesions > 10 mm) for distinguishing between bulbous polyps, folds and planar-like structures in the colon.

A new algorithm for maximum-likelihood blind image restoration is presented in this paper. It is obtained by modeling the original image and the additive noise as multivariate Gaussian processes with unknown covariance matrices. The blurring process is specified by its point spread function, which is also unknown. Estimations of the original image and the blur are derived by alternating minimization of the Kullback-Leibler divergence between a model family of probability distributions defined using the linear image degradation model and a desired family of probability distributions constrained to be concentrated on the observed data. The algorithm presents the advantage to provide closed form expressions for the parameters to be updated and to converge only after few iterations. A simulation example that illustrates the effectiveness of the proposed algorithm is presented.

A new algorithm for Maximum likelihood blind image restoration is presented in this paper. It is obtained by modeling the original image and the additive noise as multivariate Gaussian processes with unknown covariance matrices. The blurring process is specified by its point spread function, which is also unknown. Estimations of the original image and the blur are derived by alternating minimization of the Kullback-Leibler divergence. The algorithm presents the advantage to provide closed form expressions for the parameters to be updated and to converge only after few iterations. A simulation example that illustrates the effectiveness of the proposed algorithm is presented.

Existing polyp detection methods rely heavily on curvature-based characteristics to differentiate between lesions. However, as curvature is a local feature and a second order differential quantity, simply inspecting the curvature at a point is not sufficient. In this paper, we propose to inspect a local neighbourhood around a candidate point using curvature maps. This candidate point is pre-identified using the geodesic centroid of a surface patch containing vertices with positive point curvature values corresponding to convex shaped protrusions. Geodesic rings are then constructed around this candidate point and point curvatures around these rings are accumulated to produce curvature maps. From this, a cumulative shape property, S for a given neighbourhood radius can be computed and used for identifying bulbous polyps which typically have a high S value, and its corresponding 'neck' region. We show that a threshold value of S > 0.48 is sufficient to discriminate between polyps and non polyps with 100% sensitivity and specificity for bulbous polyps > 10mm.

Computed tomographic colonography (CTC) is a promising alternative to traditional invasive colonoscopic methods used in the detection and removal of cancerous growths, or polyps in the colon. Existing algorithms for CTC typically use a classifier to discriminate between true and false positives generated by a polyp candidate detection system. However, these classifiers often suffer from a phenomenon termed the curse of dimensionality, whereby there is a marked degradation in the performance of a classifier as the number of features used in the classifier is increased. In addition an increase in the number of features used also contributes to an increase in computational complexity and demands on storage space. This paper demonstrates the benefits of feature selection with the aim at increasing specificity while preserving sensitivity in a polyp detection system. It also compares the performances of an individual (F-score) and mutual information (MI) method for feature selection on a polyp candidate database, in order to select a subset of features for optimum CAD performance. Experimental results show that the performance of SVM+MI seems to be better for a small number of features used, but the SVM+Fscore method seems to dominate when using the 30-50 best ranked features. On the whole, the AUC measures are able to reach 0.8-0.85 for the top ranked 20-40 features using MI or F-score methods compared with 0.65-0.7 when using all 100 features in the worst-case scenario.

A fundamental question in functional MRI (fMRI) data analysis is to declare pixels either activated or non-activated with respect to the experimental design. A new statistical test for detecting activated pixels in fMRI data is proposed. The test is based on comparing the dimension of the parametric models fitted to the voxels fMRI time series data with and without controlled activation-baseline pattern. The Bayesian information criterion, is used for this comparison. This test has the advantage of not requiring any user-specified threshold to be estimated. The effectiveness of the proposed fMRI activation detection method is illustrated on real experimental data.

In this brief, the image restoration problem is approached as a learning system problem, in which a model is to be selected and parameters are estimated. Although the parameters which correspond to the restored image can easily be obtained, their quality depend heavily on a proper choice of the regularization parameter that controls the tradeoff between fidelity to the blurred noisy observed image and the smoothness of the restored image. By analogy between the model selection philosophy that constitutes a fundamental task in systems learning and the choice of the regularization parameter, two criteria are proposed in this brief for selecting the regularization parameter. These criteria are based on Bayesian arguments and the Kullback-Leibler divergence and they can be considered as extensions of the Bayesian information criterion (BIC) and the Akaike information criterion (AIC) for the image restoration problem.

Image restoration necessitates the choice of a regularization parameter that controls the trade-off between fidelity to the blurred noisy observed image and the smoothness of the restored image. The choice of this parameter for which several estimators have been proposed is crucial for the quality of the restored image. In this letter, two estimators for choosing the regularization parameter are proposed. One is a simple closed-form approximation to the minimum of the selection criterion, and the other is an approximation to the minimum of a mean squared error (MSE)-based criterion.

The Akaike information criterion (AIC) is a widely used tool for model selection. AIC is derived as an asymptotically unbiased estimator of a function used for ranking candidate models which is a variant of the Kullback-Leibler divergence between the true model and the approximating candidate model. Despite the Kullback-Leibler's computational and theoretical advantages, what can become inconvenient in model selection applications is their lack of symmetry. Simple examples can show that reversing the role of the arguments in the Kullback-Leibler divergence can yield substantially different results. In this paper, three new functions for ranking candidate models are proposed. These functions are constructed by symmetrizing the Kullback-Leibler divergence between the true model and the approximating candidate model. The operations used for symmetrizing are the average, geometric, and harmonic means. It is found that the original AIC criterion is an asymptotically unbiased estimator of these three different functions. Using one of these proposed ranking functions, an example of new bias correction to AIC is derived for univariate linear regression models. A simulation study based on polynomial regression is provided to compare the different proposed ranking functions with AIC and the new derived correction with AICc.

In recent years, algorithms to perform Independent Component Analysis in blind identification, localisation of sources or more general in data analysis have been developed. Prominent example certainly is the socalled FastICA algorithms from the Finnish school. In this paper we will generalise the FastICA algorithm considered as a discrete dynamical system on the unit sphere to the case where all units converge simultaneously, i.e., we consider some kind of parallel FastICA algorithm living on the orthogonal group. In addition we present a local convergence analysis for the algorithms proposed in this paper building on earlier work. It turns out that one can treat these type of algorithms in a similar manner as the Rayleigh quotient iteration, well known in numerical linear algebra, i.e. considering the algorithm as a discrete dynamical system on a suitable manifold. The algorithms presented here are compared by several numerical experiments and simulations.

The Kullback information criterion, KIC, and its univariate bias-corrected version, KIC/sub c/, are two recently developed criteria for model selection. A small sample model selection criterion for vector autoregressive models is developed. The proposed criterion is named KIC/sub vc/, where the notation "vc" stands for vector correction, and it can be considered as an extension of KIC for vector autoregressive models. KIC/sub vc/ is an unbiased estimator of a variant of the Kullback symmetric divergence, assuming that the true model is correctly specified or overfitted. Simulation results shows that the proposed criterion estimates the model order more accurately than any other asymptotically efficient method when applied to vector autoregressive model selection in small samples.

The Kullback information criterion, KIC and its multivariate bias-corrected version, KIC/sub VC/ are two alternatively developed criteria for model selection. The two criteria can be viewed as estimators of the expected Kullback symmetric divergence. In this paper, a new criterion is proposed in order to select a well fitted model for an extrapolation case. The proposed criterion is named, PKIC, where "P" stands for prediction, and is derived as an exact unbiased estimator of an adapted cost function that is based on the Kullback symmetric divergence and the future design matrix. PKIC is an unbiased estimator of its cost function assuming that the true model is correctly specified or overfitted. A simulation study illustrating that model selection with PKIC performs well for some extrapolation cases is presented.

The Kullback information criterion (KIC) is a recently developed tool for statistical model selection (Cavanaugh, J.E., Statistics and Probability Letters, vol.42, p.333-43, 1999). KIC serves as an asymptotically unbiased estimator of a variant of the Kullback symmetric divergence, known also as J-divergence. A bias correction of the Kullback symmetric information criterion is derived for linear models. The correction is of particular use when the sample size is small or when the number of fitted parameters is of a moderate to large fraction of the sample size. For linear regression models, the corrected method, called KICc, is an exactly unbiased estimator of a variant of the Kullback symmetric divergence between the true unknown model and the candidate fitted model. Furthermore, KICc is found to provide better model order choice than any other asymptotically efficient methods when applied to autoregressive time series models.