Cluster analysis by nonnegative low-rank approximations has experienced a remarkable progress in the past decade. However, the majority of such approximation approaches are still restricted to nonnegative matrix factorization (NMF) and suffer from the following two drawbacks: 1) they are unable to produce balanced partitions for large-scale manifold data which are common in real-world clustering tasks; 2) most existing NMF-type clustering methods cannot automatically determine the number of clusters. We propose a new low-rank learning method to address these two problems, which is beyond matrix factorization. Our method approximately decomposes a sparse input similarity in a normalized way and its objective can be used to learn both cluster assignments and the number of clusters. For efficient optimization, we use a relaxed formulation based on Data- Cluster-Data random walk, which is also shown to be equivalent to low-rank factorization of the doubly-stochastically normalized cluster incidence matrix. The probabilistic cluster assignments can thus be learned with a multiplicative majorization-minimization algorithm. Experimental results show that the new method is more accurate both in terms of clustering large-scale manifold data sets and of selecting the number of clusters.

Information divergence that measures the difference between two nonnegative matrices or tensors has found its use in a variety of machine learning problems. Examples are Nonnegative Matrix/Tensor Factorization, Stochastic Neighbor Embedding, topic models, and Bayesian network optimization. The success of such a learning task depends heavily on a suitable divergence. A large variety of divergences have been suggested and analyzed, but very few results are available for an objective choice of the optimal divergence for a given task. Here we present a framework that facilitates automatic selection of the best divergence among a given family, based on standard maximum likelihood estimation. We first propose an approximated Tweedie distribution for the β-divergence family. Selecting the best β then becomes a machine learning problem solved by maximum likelihood. Next, we reformulate α-divergence in terms of β-divergence, which enables automatic selection of α by maximum likelihood with reuse of the learning principle for β-divergence. Furthermore, we show the connections between γ- and β-divergences as well as Renyi- and α-divergences, such that our automatic selection framework is extended to non-separable divergences. Experiments on both synthetic and real-world data demonstrate that our method can quite accurately select information divergence across different learning problems and various divergence families.

Nonnegative Matrix Factorization (NMF) is a promising relaxation technique for clustering analysis. However, conventional NMF methods that directly approximate the pairwise similarities using the least square error often yield mediocre performance for data in curved manifolds because they can capture only the immediate similarities between data samples. Here we propose a new NMF clustering method which replaces the approximated matrix with its smoothed version using random walk. Our method can thus accommodate farther relationships between data samples. Furthermore, we introduce a novel regularization in the proposed objective function in order to improve over spectral clustering. The new learning objective is optimized by a multiplicative Majorization-Minimization algorithm with a scalable implementation for learning the factorizing matrix. Extensive experimental results on real-world datasets show that our method has strong performance in terms of cluster purity.

High dimensional data are often modeled by signal plus noise where the signal belongs to a low dimensional manifold contaminated with high dimensional noise. Estimating the signal subspace when the noise is Gaussian and the signal is non-Gaussian is the main focus of this paper. We assume that the Gaussian noise variance can be high, so standard denoising approaches like Principal Component Analysis fail. The approach also differs from standard Independent Component Analysis in that no independent signal factors are assumed. This model is called non-Gaussian subspace/component analysis (NGCA). The previous approaches proposed for this subspace analysis use the fourth cumulant matrix or the Hessian of the logarithm of characteristic functions, which both have some practical and theoretical issues. We propose to use sample Density Gradient Covariances, which are similar to the Fisher information matrix for estimating the non-Gaussian subspace. Here, we use nonparametric kernel density estimator to estimate the gradients of density functions. Moreover, we extend the notion of non-Gaussian subspace analysis to a supervised version where the label or response information is present. For the supervised non-Gaussian subspace analysis, we propose to use conditional density gradient covariances which are computed by conditioning on the discretized response variable. A non-asymptotic analysis of density gradient covariance is also provided which relates the error of estimating the population DGC matrix using sample DGC to the number of dimensions and the number of samples.

Multiplicative updates have been widely used in approximative nonnegative matrix factorization (NMF) optimization because they are convenient to deploy. Their convergence proof is usually based on the minimization of an auxiliary upper-bounding function, the construction of which however remains specific and only available for limited types of dissimilarity measures. Here we make significant progress in developing convergent multiplicative algorithms for NMF. First, we propose a general approach to derive the auxiliary function for a wide variety of NMF problems, as long as the approximation objective can be expressed as a finite sum of monomials with real exponents. Multiplicative algorithms with theoretical guarantee of monotonically decreasing objective function sequence can thus be obtained. The solutions of NMF based on most commonly used dissimilarity measures such as α- and β-divergence as well as many other more comprehensive divergences can be derived by the new unified principle. Second, our method is extended to a nonseparable case that includes e.g., γ-divergence and Rényi divergence. Third, we develop multiplicative algorithms for NMF using second-order approximative factorizations, in which each factorizing matrix may appear twice. Preliminary numerical experiments demonstrate that the multiplicative algorithms developed using the proposed procedure can achieve satisfactory Karush-Kuhn-Tucker optimality. We also demonstrate NMF problems where algorithms by the conventional method fail to guarantee descent at each iteration but those by our principle are immune to such violation.

We introduce a probabilistic version of the self-organizing map (SOM) where we model the uncertainty of both the model vectors and the data. While uncertainty information about the data is often not available, this property becomes very useful when the method is combined in a hierarchical manner with probabilistic principal component analysis (PCA), where we do estimate uncertainty of the principal components and the weights. We apply the hierarchical model to the domain of collaborative filtering, where probabilistic PCA is a popular approach due to its robustness for tackling many missing values in the data. The main focus in this paper is for recommendation systems about movies, where the movie rating data matrix of size people times movies is available, but contains lots of missing values. The matrix is first decomposed into a matrix product of people times features and features times movies by PCA. Then we apply the probabilistic SOM to both of those matrices separately. The uncertainty is large when a person (or a movie) has only a few ratings. The experiments with Movielens and Netflix data show an improvement over traditional SOM.

A variant of nonnegative matrix factorization (NMF) which was proposed earlier is analyzed here. It is called projective nonnegative matrix factorization (PNMF). The new method approximately factorizes a projection matrix, minimizing the reconstruction error, into a positive low-rank matrix and its transpose. The dissimilarity between the original data matrix and its approximation can be measured by the Frobenius matrix norm or the modified Kullback-Leibler divergence. Both measures are minimized by multiplicative update rules, whose convergence is proven for the first time. Enforcing orthonormality to the basic objective is shown to lead to an even more efficient update rule, which is also readily extended to nonlinear cases. The formulation of the PNMF objective is shown to be connected to a variety of existing NMF methods and clustering approaches. In addition, the derivation using Lagrangian multipliers reveals the relation between reconstruction and sparseness. For kernel principal component analysis (PCA) with the binary constraint, useful in graph partitioning problems, the nonlinear kernel PNMF provides a good approximation which outperforms an existing discretization approach. Empirical study on three real-world databases shows that PNMF can achieve the best or close to the best in clustering. The proposed algorithm runs more efficiently than the compared NMF methods, especially for high-dimensional data. Moreover, contrary to the basic NMF, the trained projection matrix can be readily used for newly coming samples and demonstrates good generalization.

Stochastic Neighbor Embedding (SNE) has shown to be quite promising for data visualization. Currently, the most popular implementation, t-SNE, is restricted to a particular Student t-distribution as its embedding distribution. Moreover, it uses a gradient descent algorithm that may require users to tune parameters such as the learning step size, momentum, etc., in finding its optimum. In this paper, we propose the Heavy-tailed Symmetric Stochastic Neighbor Embedding (HSSNE) method, which is a generalization of the t-SNE to accommodate various heavy-tailed embedding similarity functions. With this generalization, we are presented with two difficulties. The first is how to select the best embedding similarity among all heavy-tailed functions and the second is how to optimize the objective function once the heave-tailed function has been selected. Our contributions then are: (1) we point out that various heavy-tailed embedding similarities can be characterized by their negative score functions. Based on this finding, we present a parameterized subset of similarity functions for choosing the best tail-heaviness for HSSNE; (2) we present a fixed-point optimization algorithm that can be applied to all heavy-tailed functions and does not require the user to set any parameters; and (3) we present two empirical studies, one for unsupervised visualization showing that our optimization algorithm runs as fast and as good as the best known t-SNE implementation and the other for semi-supervised visualization showing quantitative superiority using the homogeneity measure as well as qualitative advantage in cluster separation over t-SNE.

We present a method for exploratory data analysis of large spatiotemporal data sets such as global longtime climate measurements, extending our previous work on semiblind source separation of climate data. The method seeks fast changing components whose variances exhibit slow behavior with specific temporal structure. The algorithm is developed in the framework of denoising source separation. It finds sources iteratively and alternates between estimating the variance structure of extracted sources and using the structure to find new source estimates. The performance of the algorithm is first demonstrated on a simple example of a semiblind source separation problem with artificially generated signals. Then, the proposed technique is applied to the global surface temperature measurements coming from the NCEP/NCAR re-analysis project. Fast changing temperature components whose variances have prominent annual and decadal structures are extracted. The extracted annual components reflect higher temperature variability over the continents during winters. The components with slower changing variances might correspond to some interesting weather phenomena characterized by slowly changing temperature variability in specific regions.

FastICA is one of the most popular algorithms for independent component analysis (ICA), demixing a set of statistically independent sources that have been mixed linearly. A key question is how accurate the method is for finite data samples. We propose an improved version of the FastICA algorithm which is asymptotically efficient, i.e., its accuracy given by the residual error variance attains the Cramér-Rao lower bound (CRB). The error is thus as small as possible. This result is rigorously proven under the assumption that the probability distribution of the independent signal components belongs to the class of generalized Gaussian (GG) distributions with parameter alpha, denoted GG(alpha) for alpha > 2. We name the algorithm efficient FastICA (EFICA). Computational complexity of a Matlab implementation of the algorithm is shown to be only slightly (about three times) higher than that of the standard symmetric FastICA. Simulations corroborate these claims and show superior performance of the algorithm compared with algorithm JADE of Cardoso and Souloumiac and nonparametric ICA of Boscolo et al. on separating sources with distribution GG (alpha) with arbitrary alpha, as well as on sources with bimodal distribution, and a good performance in separating linearly mixed speech signals.

The fast independent component analysis (FastICA) algorithm is one of the most popular methods to solve problems in ICA and blind source separation. It has been shown experimentally that it outperforms most of the commonly used ICA algorithms in convergence speed. A rigorous local convergence analysis has been presented only for the so-called one-unit case, in which just one of the rows of the separating matrix is considered. However, in the FastICA algorithm, there is also an explicit normalization step, and it may be questioned whether the extra rotation caused by the normalization will affect the convergence speed. The purpose of this paper is to show that this is not the case and the good convergence properties of the one-unit case are also shared by the full algorithm with symmetrical normalization. A local convergence analysis is given for the general case, and the global behavior is illustrated numerically for two sources and two mixtures in several typical cases.

In this paper, we enhance and analyze the Evolving Tree (ETree) data analysis algorithm. The suggested improvements aim to make the system perform better while still maintaining the simple nature of the basic algorithm. We also examine the system's behavior with many different kinds of tests, measurements and visualizations. We compare the ETree's performance against classical data analysis methods and very similar modern systems. We find that the ETree is a suitable method for unsupervised analysis of huge data sets.

We propose using Independent Component Analysis (ICA) as an advanced pre-processing tool for blind suppression of interfering jammer signals in direct sequence spread spectrum communication systems utilizing antenna arrays. The role of ICA is to provide a jammer-mitigated signal to the conventional detection. If the jammer signal is weak or absent, preprocessing by ICA is not advisable. Therefore we also consider two possible switching schemes, called pre-switching and post-switching, which activate the ICA-based jammer canceller only when it is expected to improve conventional detection. ICA-RAKE pre-switching is less complex, while post-switching performs better, especially when the jammer is pulsed in nature. Simulations are given to illustrate the achieved performance gains for single- and multi-path channels.

This paper derives a closed-form expression for the Cramer-Rao bound (CRB) on estimating the source signals in the linear independent component analysis problem, assuming that all independent components have finite variance. It is also shown that the fixed-point algorithm known as FastICA can approach the CRB (the estimate can be nearly efficient) in two situations: (1) when the distribution of the sources is not too much different from Gaussian, for the symmetric version of the algorithm using any of the custom nonlinear functions (pow3, tanh, gauss); (2) when the distribution of the sources is very different from Gaussian (e.g. has long tails) and the nonlinear function in the algorithm equals the score function of each independent component.

Denoising source separation (DSS), a recently developed source separation framework, was applied to extracting components exhibiting slow, interannual temporal behavior from climate data. Three datasets with daily measurements were used: surface temperature, sea level pressure and precipitation around the globe. For all datasets, the first extracted component captured the well-known El Nino-Southern Oscillation phenomenon and the second component was close to the derivative of the first one. Several other components with slow dynamics were extracted and together the components appear to capture essential features of the slow-dynamics state of the climate system. The first two components were identified reliably but the following components may have remained mixed, nonlinear DSS could identify the physically most meaningful rotation among them but only linear DSS was within the scope of this paper. This paper offers a simple demonstration of exploratory data analysis of climate data by DSS and suggests future lines of research.

We present and examine a tree-shaped self-organizing neural network called the evolving tree. It has been designed to efficiently analyze large databases. We show how the system's architecture makes it efficient for these kinds of tasks. We then analyze the system's behaviour under different circumstances. We find that, despite being simple, the evolving tree is a suitable tool for unsupervised data analysis.

Two pricing problems are solved by using asymmetric multiagent reinforcement learning methods in this paper. In the first problem, a flat pricing scenario, there are two competing brokers that sell identical products to customers and compete on the basis of price. The second problem is a hierarchical pricing scenario, where a supplier sells products to two competing brokers. In both cases, the methods converged and led to very promising results. We present a brief literature survey of pricing models based on reinforcement learning, introduce the basic concepts of Markov games and solve two pricing problems based on multiagent reinforcement learning.

Summary form only given. Learning the implicit structure of data in large-scale applications like document and image mining or multivariate signal analysis helps in understanding the underlying causes and phenomena. The result of learning is a new explanation or compressed representation of the observation data, which lead to improved decisions. In artificial neural networks, the representation is usually a clustering of the data, a discrete map, or a lower-dimensional manifold in the observation space. The talk covered some of the paradigms of artificial neural learning based on self-organization, principal, and independent component analysis, and efficient algorithms for their computation. Many examples from the author's research group are used to illuminate the concepts and methods.

The instantaneous noise-free linear mixing model in independent component analysis is largely a solved problem under the usual assumption of independent nongaussian sources and full column rank mixing matrix. However, with some prior information on the sources, like positivity, new analysis and perhaps simplified solution methods may yet become possible. In this letter, we consider the task of independent component analysis when the independent sources are known to be nonnegative and well grounded, which means that they have a nonzero pdf in the region of zero. It can be shown that in this case, the solution method is basically very simple: an orthogonal rotation of the whitened observation vector into nonnegative outputs will give a positive permutation of the original sources. We propose a cost function whose minimum coincides with nonnegativity and derive the gradient algorithm under the whitening constraint, under which the separating matrix is orthogonal. We further prove that in the Stiefel manifold of orthogonal matrices, the cost function is a Lyapunov function for the matrix gradient flow, implying global convergence. Thus, this algorithm is guaranteed to find the nonnegative well-grounded independent sources. The analysis is complemented by a numerical simulation, which illustrates the algorithm.

Changes in a dynamical process are often detected by monitoring selected indicators directly obtained from the process observations, such as the mean values or variances. Standard change detection algorithms such as the Shewhart control charts or the cumulative sum (CUSUM) algorithm are often based on such first- and second-order statistics. Much better results can be obtained if the dynamical process is properly modeled, for example by a nonlinear state-space model, and then the accuracy of the model is monitored over time. The success of the latter approach depends largely on the quality of the model. In practical applications like industrial processes, the state variables, dynamics, and observation mapping are rarely known accurately. Learning from data must be used; however, methods for the simultaneous estimation of the state and the unknown nonlinear mappings are very limited. We use a novel method of learning a nonlinear state-space model, the nonlinear dynamical factor analysis (NDFA) algorithm. It takes a set of multivariate observations over time and fits blindly a generative dynamical latent variable model, resembling nonlinear independent component analysis. We compare the performance of the model in process change detection to various traditional methods. It is shown that NDFA outperforms the classical methods by a wide margin in a variety of cases where the underlying process dynamics changes.

We consider the task of independent component analysis when the independent sources are known to be nonnegative and well-grounded, so that they have a nonzero probability density function (pdf) in the region of zero. We propose the use of a "nonnegative principal component analysis (nonnegative PCA)" algorithm, which is a special case of the nonlinear PCA algorithm, but with a rectification nonlinearity, and we conjecture that this algorithm will find such nonnegative well-grounded independent sources, under reasonable initial conditions. While the algorithm has proved difficult to analyze in the general case, we give some analytical results that are consistent with this conjecture and some numerical simulations that illustrate its operation.

Development of content-based image retrieval (CBIR) techniques has suffered from the lack of standardized ways for describing visual image content. Luckily, the MPEG-7 international standard is now emerging as both a general framework for content description and a collection of specific agreed-upon content descriptors. We have developed a neural, self-organizing technique for CBIR. Our system is named PicSOM and it is based on pictorial examples and relevance feedback (RF). The name stems from "picture" and the self-organizing map (SOM). The PicSOM system is implemented by using tree structured SOMs. In this paper, we apply the visual content descriptors provided by MPEG-7 in the PicSOM system and compare our own image indexing technique with a reference system based on vector quantization (VQ). The results of our experiments show that the MPEG-7-defined content descriptors can be used as such in the PicSOM system even though Euclidean distance calculation, inherently used in the PicSOM system, is not optimal for all of them. Also, the results indicate that the PicSOM technique is a bit slower than the reference system in starting to find relevant images. However, when the strong RF mechanism of PicSOM begins to function, its retrieval precision exceeds that of the reference system.

Adaptation is an effective method for improving classification accuracy and a committee structure can in general improve on its members' performance. Therefore an adaptive committee structure is a tempting approach. Rejection may be used in handwriting recognition to improve performance through either directing the problematic character to a special classifier that handles such hard cases or discarding it. The experiments in this paper compare several fundamentally different approaches to implementing rejection in an adaptive committee classifier. A dynamically expanding context (DEC) - based committee is used for evaluating these approaches. The results show that if the rejected classes are handled with a 50% error rate, the performance is improved. A scheme in which there is an adjustable threshold for distance-based rejection is an effective method for implementing rejection in this setting.

This work describes a prototype-based online handwritten character recognition system and a two-phase recognition scheme aimed to speed up the recognition. In the first phase, the prototype set is pruned and ordered on the basis of preclassification performed with heavily down-sampled characters and prototypes. In the second phase, the final classification is performed without down-sampling by using the reduced set of prototypes. Two down-sampling methods, a linear and nonlinear one, have been analyzed to see their properties regarding the recognition time and accuracy.

This article addresses the problem of blind source separation from time-varying noisy mixtures using a state variable model and recursive estimation. An estimate of each source signal is produced real time at the arrival of new observed mixture vector. The goal is to perform the separation and attenuate noise simultaneously, as well as to adapt to changes that occur in the mixing system. The observed data are projected along the eigenvectors in signal subspace. The subspace is tracked real time. Source signals are modeled using low-order AR (autoregressive) models, and noise is attenuated by trading off between the model and the information provided by measurements. The type of zero-memory nonlinearity needed in separation is determined on-line. Predictor-corrector filter structures are proposed, and their performance is investigated in simulation using biomedical and communications signals at different noise levels and a time-varying mixing system. In quantitative comparison to other widely used methods, significant improvement in output signal-to-noise ratio is achieved.

In this article the use of shape features in content-based image retrieval is studied. The emphasis is on techniques which do not demand object segmentation. PicSOM, the image retrieval system used in the experiments, requires that features are represented by constant-sized feature vectors for which the Euclidean distance can be used as a similarity measure. The shape features suggested here are edge histograms and Fourier transform based features computed for an edge image in Cartesian and polar coordinate planes. The results show that both local and global shape features are important clues of shapes in an image.

Methods for controlling the adaptation process of an online handwritten character recognizer are studied. The classifier is based on the k-nearest neighbor rule and it is adapted to a new writing style by adding new prototypes, deactivating confusing prototypes, and reshaping existing prototypes in a self-supervised fashion. The dissimilarity measure used for the comparison of characters is a nonlinear curve matching method base on dynamic time warping algorithm. Time needed for the evaluation of the dissimilarity measure for a single character depends linearly on the size of the prototype set. The purpose of the control methods is to increase the classifier's tolerance to malformed or mislabelled learning samples and to limit the growth of the prototype set. The control methods either set an upper limit for the number of prototypes per class or switch the adaptation of a particular character class on or off depending on the earlier performance of the classifier.

We have developed an adaptive handwriting recognizer for isolated Latin characters in which the adaptive behavior is based on the dynamically expanding context (DEC) algorithm. In our current system, the outputs of a set of static classifiers are combined in a committee machine, whose rules are adapted. Every misclassified character gives rise to adding a new DEC rule to the rule set of the committee. When the existing rules fail to produce a correct recognition output, more and more context information is utilized in forming the new DEC rules. Not only the first-ranking outputs from the member classifiers but also the second-ranking ones can be taken into account when forming the DEC rules. In the experiments described in this paper, various options in the implementation of the DEC committee classifier are evaluated. The results of the experiments show that the system is capable of fast adaptation to the user's handwriting and lead to lowered recognition error rates.

We have developed an adaptive online recognizer that is suitable for recognizing isolated alphanumeric characters. It is based on the k nearest neighbor rule. Various dissimilarity measures, all based on dynamic time warping (DTW), have been studied. The main focus of this work is on online adaptation. The adaptation is performed by modifying the prototype set of the classifier according to its recognition performance and the user's writing style. These adaptations include: (1) adding new prototypes, (2) inactivating confusing prototypes, and (3) reshaping existing prototypes. The reshaping algorithm is based on learning vector quantization (LVQ). The writers are allowed to use their own natural style of writing, and the adaptation is carried out during normal use in a self-supervised fashion and thus remains otherwise unnoticed by the user.

We compare the performance of five prominent neural or semi-neural algorithms designed for independent component analysis (ICA) using three different real-world data sets. The task is either to find interesting directions in the data for visualisation purposes or blind source separation. We develop criteria for selecting the most meaningful basis vectors of ICA or measuring the goodness of results. The comparison reveals characteristic differences between the studied neural ICA algorithms, complementing our previous results (1998) obtained for artificially generated data.

We introduce a very fast method for estimating over-complete bases of independent components from image data. This is based on the concept of quasi-orthogonality, which means that in a very high-dimensional space, there can be a large, over-complete set of vectors that are almost orthogonal to each other. Thus we may estimate an over-complete basis by using one-unit ICA algorithms and forcing only partial decorrelation between the different independent components. The method can be implemented using a modification of the FastICA algorithm, which leads to a computationally highly efficient method.

Subsystems for online recognition of handwriting are needed in personal digital assistants (PDA) and other portable handheld devices. We have developed a recognition system which enhances its accuracy by applying continuous adaptation to the user's writing style. The forms of adaptation we have experimented with take place simultaneously with the normal operation of the system and therefore, there is no need for separate training period of the device. The present implementation uses dynamic time warping (DTW) in matching the input characters with the stored prototypes. The DTW algorithm implemented with dynamic programming (DP) is, however both time and memory consuming. In our current research we have experimented with methods that transform the elastic templates to pixel images which can then be recognized by using statistical or neural classification. The particular neural classifier we have used is the local subspace classifier (LSC) of which we have developed an adaptive version.

Content-based image retrieval is an important approach to the problem of processing the increasing amount of visual data. It is based on automatically extracted features from the content of the images, such as color, texture, shape and structure. We have started a project to study methods for content-based image retrieval using the self-organizing map (SOM) as the image similarity scoring method. Our image retrieval system, named PicSOM, can be seen as a SOM-based approach to relevance feedback which is a form of supervised learning to adjust the subsequent queries based on the user's responses during the information retrieval session. In PicSOM, a separate tree structured SOM (TS-SOM) is trained for each feature vector type in use. The system then adapts to the user's preferences by returning her more images from those SOMs where her responses have been most densely mapped.

With the continuous increase in computing power, it has become possible to process and classify masses of natural data, such as statistical information, images, speech, as well as other kinds of signals and measurements coming from very different sources. Many problems occur in industry, finance, remote sensing, medicine, and natural sciences, to mention only a few main fields, in which one needs efficient tools for visualization, prediction, clustering, and profiling. Often the explicit modelling of the processes underlying the measurements is very hard and so inferences from the measurement data must be made by learning methods. A widely used class of learning algorithms are the neural learning paradigms. In this paper, emphasis is on unsupervised neural learning. Especially the techniques of self-organizing maps and independent component analysis are reviewed and shown to be useful in this context. Some examples are shown on applications of these techniques on financial data analysis.

In this paper, we compare the performance of five prominent neural or adaptive algorithms designed for Independent Component Analysis (ICA) and blind source separation (BSS). In the first part of the study, we use artificial data for comparing the accuracy, convergence speed, computational load, and other relevant properties of the algorithms. In the second part, the algorithms are applied to three different real-world data sets. The task is either blind source separation or finding interesting directions in the data for visualisation purposes. We develop criteria for selecting the most meaningful basis vectors of ICA and measuring the quality of the results. The comparison reveals characteristic differences between the studied ICA algorithms. The most important conclusions of our comparison are robustness of the ICA algorithms with respect to modest modeling imperfections, and the superiority of fixed-point algorithms with respect to the computational load.

Sparse coding is a method for finding a representation of data in which each of the components of the representation is only rarely significantly active. Such a representation is closely related to the techniques of independent component analysis and blind source separation. In this paper, we investigate the application of sparse coding for image feature extraction. We show how sparse coding can be used to extract wavelet-like features from natural image data. As an application of such a feature extraction scheme, we show how to apply a soft-thresholding operator on the components of sparse coding in order to reduce Gaussian noise. Methods based on sparse coding have the important benefit over wavelet methods that the features are determined solely by the statistical properties of the data, while the wavelet transformation relies heavily on certain abstract mathematical properties that may be only weakly related to the properties of the natural data.

We (1998) have developed methods for the description of monochrome and color textures with models of multidimensional co-occurrence distributions. The models are histograms of quantized multidimensional co-occurrence vectors obtained using the code words of vector quantizer as indexes of histogram bins. In the present study, the color texture analysis is further developed by selecting the co-occurring color components and the number of code vectors to minimize the classification error. The genetic algorithm is used for the optimization, and the iterative searches for the best parameters are performed by a vector quantizer with a short training time: the two-stage vector quantizer. The reduced multidimensional color histograms of 2-by-2-pixel values provide significantly higher classification accuracies than two- or three-dimensional histograms of intra- and interpixel co-occurrences. They also performed better than a Markov random field model.

Subspace methods are a powerful class of statistical pattern classification algorithms. The subspaces form semiparametric representations of the pattern classes in the form of principal components. In this sense, subspace classification methods are an application of classical optimal data compression techniques. Additionally, the subspace formalism can be given a neural network interpretation. There are learning versions of the subspace classification methods, in which error-driven learning procedures are applied to the subspaces in order to reduce the number of misclassified vectors. An algorithm for iterative selection of the subspace dimensions is presented in this paper. Likewise, a modified formula for calculating the projection lengths in the subspaces is investigated. The principle of adaptive learning in subspace methods can further be applied to feature extraction. In our work, we have studied two adaptive feature extraction schemes. The adaptation process is directed by errors occurring in the classifier. Unlike most traditional classifier models which take the preceding feature extraction stage as given, this scheme allows for reducing the loss of information in the feature extraction stage. The enhanced overall classification performance resulting from the added adaptivity is demonstrated with experiments in which recognition of handwritten digits has been used as an exemplary application.

We have developed methods for the classification of textures with multidimensional co-occurrence histograms. Gray levels of several pixels with a given spatial arrangement are first compressed linearly and the resulting multidimensional vectors are quantized using the self-organizing map. Histograms of quantized vectors are classified by matching them with precomputed texture model histograms. In the present study, a multiple resolution technique in linear compression of pixel values is evaluated. The multiple resolution linear compression was made with a local wavelet transform. The vectors were quantized with the tree-structured variant of the self-organizing map. In the tree-structured self-organizing map, the quantization error is reduced, in comparison to the traditional tree-structured codebook, by limited lateral searches in topologically-ordered neighborhoods. The performance of multiresolution texture histograms was compared with single-resolution histograms. The histogram method was compared with three well-established methods: co-occurrence matrices, Gaussian Markov random fields, and multiresolution Gabor energies. The results for a set of natural textures showed that the performance of single- and multiresolution texture histograms was similar. Thus, the benefit of multiresolution analysis was overridden by the multidimensionality of our texture models. Our method gave significantly higher classification accuracies than the three other methods.

Textures are frequently described using co-occurrence histograms of gray levels at two pixels in a given relative position. Analysis of several co-occurring pixel values may benefit texture description but is impeded by the exponential growth of histogram size. To make use of multidimensional histograms, we have developed methods for their reduction. The method described here uses linear compression, dimension optimization, and vector quantization. Experiments with natural textures showed that multidimensional histograms reduced with the new method provided higher classification accuracies than the channel histograms and the wavelet packet signatures. The new method was significantly faster than our previous one.

In blind source separation one tries to separate statistically independent unknown source signals from their linear mixtures without knowing the mixing coefficients. Such techniques are currently studied actively both in statistical signal processing and unsupervised neural learning. We apply neural blind separation techniques developed in our laboratory to the extraction of features from natural images and to the separation of medical EEG signals. The new analysis method yields features that describe the underlying data better than for example classical principal component analysis. We discuss difficulties related with real-world applications of blind signal processing, too.

We introduce a novel fast algorithm for independent component analysis, which can be used for blind source separation and feature extraction. We show how a neural network learning rule can be transformed into a fixedpoint iteration, which provides an algorithm that is very simple, does not depend on any user-defined parameters, and is fast to converge to the most accurate solution allowed by the data. The algorithm finds, one at a time, all nongaussian independent components, regardless of their probability distributions. The computations can be performed in either batch mode or a semiadaptive manner. The convergence of the algorithm is rigorously proved, and the convergence speed is shown to be cubic. Some comparisons to gradient-based algorithms are made, showing that the new algorithm is usually 10 to 100 times faster, sometimes giving the solution in just a few iterations.

Pattern classification using neural networks and statistical methods is discussed. We give a tutorial overview in which popular classifiers are grouped into distinct categories according to their underlying mathematical principles; also, we assess what makes a classifier neural. The overview is complemented by two case studies using handwritten digit and phoneme data that test the performance of a number of most typical neural-network and statistical classifiers. Four methods of our own are included: reduced kernel discriminant analysis, the learning k-nearest neighbors classifier, the averaged learning subspace method, and a version of kernel discriminant analysis.

Independent component analysis (ICA) is a recently developed, useful extension of standard principal component analysis (PCA). The ICA model is utilized mainly in blind separation of unknown source signals from their linear mixtures. In this application only the source signals which correspond to the coefficients of the ICA expansion are of interest. In this paper, we propose neural structures related to multilayer feedforward networks for performing complete ICA. The basic ICA network consists of whitening, separation, and basis vector estimation layers. It can be used for both blind source separation and estimation of the basis vectors of ICA. We consider learning algorithms for each layer, and modify our previous nonlinear PCA type algorithms so that their separation capabilities are greatly improved. The proposed class of networks yields good results in test examples with both artificial and real-world data.

Pattern classification using neural networks and statistical methods is discussed and a taxonomy based on their underlying mathematical principles is presented. Typical neural network and statistical classifiers are then compared in a case study using handwritten digit data.

The self-organizing map (SOM) method is a new, powerful software tool for the visualization of high-dimensional data. It converts complex, nonlinear statistical relationships between high-dimensional data into simple geometric relationships on a low-dimensional display. As it thereby compresses information while preserving the most important topological and metric relationships of the primary data elements on the display, it may also be thought to produce some kind of abstractions. The term self-organizing map signifies a class of mappings defined by error-theoretic considerations. In practice they result in certain unsupervised, competitive learning processes, computed by simple-looking SOM algorithms. Many industries have found the SOM-based software tools useful. The most important property of the SOM, orderliness of the input-output mapping, can be utilized for many tasks: reduction of the amount of training data, speeding up learning nonlinear interpolation and extrapolation, generalization, and effective compression of information for its transmission.

A generic, modular, neural network-based feature extraction and pattern classification system is proposed for finding essentially two-dimensional objects or object parts from digital images in a distortion tolerant manner, The distortion tolerance is built up gradually by successive blocks in a pipeline architecture. The system consists of only feedforward neural networks, allowing efficient parallel implementation. The most time and data-consuming stage, learning the relevant features, is wholly unsupervised and can be made off-line. The consequent supervised stage where the object classes are learned is simple and fast. The feature extraction is based on distortion tolerant Gabor transformations, followed by minimum distortion clustering by multilayer self-organizing maps. Due to the unsupervised learning strategy, there is no need for preclassified training samples or other explicit selection for training patterns during the training, which allows a large amount of training material to be used at the early stages, A supervised, one-layer subspace network classifier on top of the feature extractor is used for object labeling. The system has been trained with natural images giving the relevant features, and human faces and their parts have been used as the object classes for testing. The current experiments indicate that the feature space has sufficient resolution power for a moderate number of classes with rather strong distortions.

It is shown that frequency sensitive competitive learning (FSCL), one version of the recently improved competitive learning (CL) algorithms, significantly deteriorates in performance when the number of units is inappropriately selected. An algorithm called rival penalized competitive learning (RPCL) is proposed. In this algorithm, not only is the winner unit modified to adapt to the input for each input, but its rival (the 2nd winner) is delearned by a smaller learning rate. RPCL can be regarded as an unsupervised extension of Kohonen's supervised LVQ2. RPCL has the ability to automatically allocate an appropriate number of units for an input data set. The experimental results show that RPCL outperforms FSCL when used for unsupervised classification, for training a radial basis function (RBF) network, and for curve detection in digital images.

A method has been developed to calculate 2-D motion in a sequence of time-varying images. The method, called motion detection using randomized Hough transform (MDRHT), is based on the randomized Hough transform (RHT). The RHT decreases considerably the time consumption and memory requirements of the Hough transform. The idea of the MDRHT is to pick randomly point pairs from two images and calculate the translation with them. The points can be e.g. edge points of the original images. This approach can avoid difficulties of standard segmentation methods like overlapping and covering, and has the advantages provided by the RHT. The method can be generalized by picking more than two points. After a brief review of the RHT applied to motion detection, the extended algorithm to calculate both translation and rotation is represented in this paper.>

For the classical k-means clustering algorithm, the problem of selecting an appropriate k is a hard problem and affects the performance of k-means strongly. When used for clustering analysis, the conventional competitive learning (CL) algorithms also have a similar crucial problem-the selection of an appropriate number of neural units. The performance of frequency sensitive competitive learning (FSCL)-one version of the improved CL algorithms, also significantly deteriorates when the number of units is inappropriately selected. The paper proposes a new algorithm called rival penalized competitive learning (RPCL), which has the ability of automatically allocating an appropriate number of units for an input data set. The experimental results have shown that RPCL outperforms FSCL significantly when they are used for unsupervised classification, and supervised classification through the radial basis function net.>

Further developments on RHT are introduced here. The fundamental framework and the main components of this method are systematically elaborated. Its advantages are further confirmed. The deep mechanisms behind these advantages are exposed. Several improved algorithms for implementing RHT are proposed and tested by experiments on images with various kinds of strong noise. The results show that the advantages of RHT are quite robust. Moreover, the theoretical results are also given to show quantitatively that the complexities of RHT algorithms are considerably smaller than those of the HT.>

A new modification of the subspace pattern recognition method, called the dual subspace pattern recognition (DSPR) method, is proposed, and neural network models combining both constrained Hebbian and anti-Hebbian learning rules are developed for implementing the DSPR method. An experimental comparison is made by using our model and a three-layer forward net with backpropagation learning. The results illustrate that our model can outperform the backpropagation model in suitable applications.

A method is developed for calculating the Hough transform (HT) and completing the task of finding global features from binary edge images. The method is based on the fact that a single parameter space point can be determined uniquely with a pair, triple, or generally n-tuple of points from the original picture. Such n-tuples of points can be chosen randomly from the edge image, giving the method the name randomized Hough transform (RHT). The new algorithm reduces the computation time and memory use of the HT drastically. In the standard HT one must calculate one parameter curve in the parameter space for one pixel, whereas in the RHT only one parameter point has to be solved for one n-tuple of points, and the presence of a specific curve in the image is quickly revealed by the accumulation of a small number of parameter points.>

The topological feature map (TFM) algorithm introduced by T. Kohenen (1982) implements two important properties: a vector quantization (VQ) and a topology-preserving mapping. A tree-structured TFM (TSTFM) is presented as a computationally inexpensive alternative to the TFM algorithm. The computational complexity of the TSTFM isO(logN) rather thanO(N) for the TFM. In addition, the TSTFM has some new properties that prove to be useful for VQ and in the context of visual perception: increased performance in VQ compared to the tree-structured VQ of A. Buzo et al. (1980) and hierarchical mapping of code vectors

A single neuron with Hebbian-type learning for the connection weights, and with nonlinear internal feedback, has been shown to extract the statistical principal components of its stationary input pattern sequence. A generalization of this model to a layer of neuron units is given, called the Subspace Network, which yields a multi-dimensional, principal component subspace. This can be used as an associative memory for the input vectors or as a module in nonsupervised learning of data clusters in the input space. It is also able to realize a powerful pattern classifier based on projections on class subspaces. Some classification results for natural textures are given.

Two statistical measures, based on contingency tables, were compared in trying to reveal the periodic or quasiperiodic structure of a texture from its cooccurrence histogram: the chi /sup 2/ and the kappa statistics. Both are sensitive to periodicity, but the kappa statistic is clearly better in the sense that the chi /sup 2/ can attain high values for displacements quite different from the actual period, as long as the cooccurrence histogram shows a specific structure associating the two relevant variables. If periodicity is the property looked for, then statistical agreement, measured by the kappa statistic, is the property that will best reveal it.>

A local spatial convolution filter is suggested whose design is based on approximating a global optimal restoration filter, the Parametric Projection Filter (PPF). The PPF tries to construct a restored signal or image that is optimally close to each individual original signal, with the noise level as low as possible. The local filter provides a restoration that is as close as possible to the PPF restoration. The design is accomplished by least squares techique in Fourier domain. Once the PPF is available, the main task is to solve a set of linear normal equations whose coefficient size depends on the size of the local filter. It is shown by examples using both one- and two-dimensional signals that the restorations are satisfactory even when the filter size is considerably smaller than the width of the degradation.

An iterative method of constructing projection matrices on the intersection of subspaces is considered, using a product of elementary matrices. A convergence theorem with two natural assumptions, concerning the gain sequence and the order of representations of the matrices, is established. The result has an application in forming associative memories.