We consider the problem of testing whether r ≥ 2 samples are drawn from the same continuous distribution F(x). The test statistic we will study in some detail is defined as the maximum of the circular differences of the empirical distribution functio

We present a noise robust PCA algorithm which is an extension of the Oja subspace algorithm and allows tuning the noise sensitivity. We derive a loss function which is minimized by this algorithm and interpret it in a noisy PCA setting. Results on the local stability analysis of this algorithm are given and it is shown that the locally stable equilibria are those which minimize the loss function.

Cluster validity has been mainly used to evaluate the quality of individual clusters, and compare the whole partitions resulting from different or same (using different parameters) clustering algorithms. However, depending on the application, the demands for a validity measure may differ, resulting the necessity of introducing new measures which will suit to the problem under investigation. We introduce a new statistical concept for quantitative validation of fMRI analysis methods based on the exploratory data analysis algorithms. A "greedy" algorithm which combines partitions on a sequential basis is applied on 100 runs of a clustering algorithm with randomized cluster initialization. As a result, a fuzzy partition is derived where every data point is assigned to a cluster with a certain degree of "support". The purpose of this measure is to check the stability of the resulting clustering structures and improve the diagnostic reliability of fMRI post-processing strategies.

In this paper we present a voting scheme for fuzzy cluster algorithms. This voting method allows us to combine several runs of cluster algorithms resulting in a common partition. This helps us to tackle the problem of choosing the appropriate clustering method for a data set where we have no a priori information about it. We mathematically derive the algorithm from theoretical considerations. Experiments show that the voting algorithm finds structurally stable results. Several cluster validity indexes show the improvement of the voting result in comparison to simple fuzzy voting.

We consider autoregressive neural network (AR-NN) processes driven by additive noise and demonstrate that the characteristic roots of the shortcuts-the standard conditions from linear time-series analysis-determine the stochastic behavior of the overall AR-NN process. If all the characteristic roots are outside the unit circle, then the process is ergodic and stationary. If at least one characteristic root lies inside the unit circle, then the process is transient. AR-NN processes with characteristic roots lying on the unit circle exhibit either ergodic, random walk, or transient behavior. We also analyze the class of integrated AR-NN (ARI-NN) processes and show that a standardized ARI-NN process "converges" to a Wiener process. Finally, least-squares estimation (training) of the stationary models and testing for nonstationarity is discussed. The estimators are shown to be consistent, and expressions on the limiting distributions are given.

Within the last years various principal component analysis (PCA) algorithms have been proposed. In this paper we use a general framework to describe those PCA algorithms which are based on Hebbian learning. For an important subset of these algorithms, the local algorithms, we fully describe their equilibria, where all lateral connections are set to zero and their local stability. We show how the parameters in the PCA algorithms have to be chosen in order to get an algorithm which converges to a stable equilibrium which provides principal component extraction.

In a typical linear data compression system the representation variables resulting from the coding operation are assumed totally reliable and therefore the solution in the mean-squared-error sense is an orthogonal projector to the so-called principal component subspace. When the representation variables are contaminated by additive noise which is uncorrelated with the signal, the problem is called noisy principal component analysis (NPCA) and the optimal MSE solution is not a trivial extension of PCA. We first show that the problem is not well defined unless we impose explicit or implicit constraints on either the coding or the decoding operator. Second, orthogonality is not a property of the optimal solution under most constraints. Third, the signal components may or may not be reconstructed depending on the noise level. As the noise power increases, we observe rank reduction in the optimal solution under most reasonable constraints. In these cases it appears that it is preferable to omit the smaller signal components rather than attempting to reconstruct them. This phenomenon has similarities with classical information theoretical results, notably the water-filling analogy, found in parallel additive Gaussian noise channels. Finally, we show that standard Hebbian-type PCA learning algorithms are not optimally robust to noise, and propose a new Hebbian-type learning algorithm which is optimally robust in the NPCA sense.

We propose a new approach for leave-one-out cross-validation of neural-network classifiers called "cross-validation with active pattern selection" (CV/APS). In CV/APS, the contribution of the training patterns to network learning is estimated and this information is used for active selection of CV patterns. On the tested examples, the computational cost of CV can be drastically reduced with only small or no errors.

In this paper we consider the principal component analysis (PCA) and vector quantization (VQ) neural networks for image compression. We present a method where the PCA and VQ steps are adaptively combined. A learning algorithm for this combined network is derived. We demonstrate that this approach can improve the results of the successive application of the individually optimal methods.

The principal component pyramid is a hierarchical neural network which can successfully be employed in image compression and feature extraction of images. Previously, the construction of the network from the corresponding pyramid was done on a case by case basis. In this paper we automate this process by giving formulas describing the size of the network and the number of weight constraints in the net.

Networks of linear units are the simplest kind of networks, where the basic questions related to learning, generalization, and self-organization can sometimes be answered analytically. We survey most of the known results on linear networks, including: 1) backpropagation learning and the structure of the error function landscape, 2) the temporal evolution of generalization, and 3) unsupervised learning algorithms and their properties. The connections to classical statistical ideas, such as principal component analysis (PCA), are emphasized as well as several simple but challenging open questions. A few new results are also spread across the paper, including an analysis of the effect of noise on backpropagation networks and a unified view of all unsupervised algorithms.

This paper presents a new method for image compression by neural networks. First, we show that we can use neural networks in a py(cid:173) ramidal framework, yielding the so-called PCA pyramids. Then we present an image compression method based on the PCA pyramid, which is similar to the Laplace pyramid and wavelet transform. Some experimental results with real images are reported. Finally, we present a method to combine the quantization step with the learning of the PCA pyramid.

Recently Barron (1993) has given rates for hidden layer feedforward networks with sigmoid activation functions approximating a class of functions satisfying a certain smoothness condition. These rates do not depend on the dimension of the input space. We extend Barron's results to feedforward networks with possibly nonsigmoid activation functions approximating mappings and their derivatives simultaneously. Our conditions are similar but not identical to Barron's, but we obtain the same rates of approximation, showing that the approximation error decreases at rates as fast as n −1/2 , where n is the number of hidden units. The dimension of the input space appears only in the constants of our bounds.

We give a rigorous analysis of the convergence properties of a backpropagation algorithm for recurrent networks containing either output or hidden layer recurrence. The conditions permit data generated by stochastic processes with considerable dependence. Restrictions are offered that may help assure convergence of the network parameters to a local optimum, as some simulations illustrate.

The behavior of neural network learning algorithms with a small, constant learning rate, epsilon, in stationary, random input environments is investigated. It is rigorously established that the sequence of weight estimates can be approximated by a certain ordinary differential equation, in the sense of weak convergence of random processes as epsilon tends to zero. As applications, backpropagation in feedforward architectures and some feature extraction algorithms are studied in more detail.