Self-supervised speech representation models, particularly those leveraging transformer architectures, have demonstrated remarkable performance across various tasks such as speech recognition, speaker identification, and emotion detection. Recent studies on transformer models revealed high redundancy between layers and the potential for significant pruning, which we will investigate here for transformer-based speech representation models. We perform a detailed analysis of layer similarity in speech representation models using three similarity metrics: cosine similarity, centered kernel alignment, and mutual nearest-neighbor alignment. Our findings reveal a block-like structure of high similarity, suggesting two main processing steps and significant redundancy of layers. We demonstrate the effectiveness of pruning transformer-based speech representation models without the need for post-training, achieving up to 40% reduction in transformer layers while maintaining over 95% of the model’s predictive capacity. Furthermore, we employ a knowledge distillation method to substitute the entire transformer stack with mimicking layers, reducing the network size by 95-98% and the inference time by up to 94%. This substantial decrease in computational load occurs without considerable performance loss, suggesting that the transformer stack is almost completely redundant for downstream applications of speech representation models.

Suppressor variables can influence model predictions without being dependent on the target outcome, and they pose a significant challenge for Explainable AI (XAI) methods. These variables may cause false-positive feature attributions, undermining the utility of explanations. Although effective remedies exist for linear models, their extension to non-linear models and instance-based explanations has remained limited. We introduce PatternLocal, a novel XAI technique that addresses this gap. PatternLocal begins with a locally linear surrogate, e.g., LIME, KernelSHAP, or gradient-based methods, and transforms the resulting discriminative model weights into a generative representation, thereby suppressing the influence of suppressor variables while preserving local fidelity. In extensive hyperparameter optimization on the XAI-TRIS benchmark, PatternLocal consistently outperformed other XAI methods and reduced false-positive attributions when explaining non-linear tasks, thereby enabling more reliable and actionable insights. We further evaluate PatternLocal on an EEG motor imagery dataset, demonstrating physiologically plausible explanations.

In this work, we propose a Multi-Window Masked Autoencoder (MW-MAE) fitted with a novel Multi-Window Multi-Head Attention (MW-MHA) module that facilitates the modelling of local-global interactions in every decoder transformer block through attention heads of several distinct local and global windows. Empirical results on ten downstream audio tasks show that MW-MAEs consistently outperform standard MAEs in overall performance and learn better general-purpose audio representations, along with demonstrating considerably better scaling characteristics. Investigating attention distances and entropies reveals that MW-MAE encoders learn heads with broader local and global attention. Analyzing attention head feature representations through Projection Weighted Canonical Correlation Analysis (PWCCA) shows that attention heads with the same window sizes across the decoder layers of the MW-MAE learn correlated feature representations which enables each block to independently capture local and global information, leading to a decoupled decoder feature hierarchy.

Bayesian Optimization (BO) is a popular method for black-box optimization, which relies on uncertainty as part of its decision-making process when deciding which experiment to perform next. However, not much work has addressed the effect of uncertainty on the performance of the BO algorithm and to what extent calibrated uncertainties improve the ability to find the global optimum. In this work, we provide an extensive study of the relationship between the BO performance (regret) and uncertainty calibration for popular surrogate models and acquisition functions, and compare them across both synthetic and real-world experiments. Our results show that Gaussian Processes, and more surprisingly, Deep Ensembles are strong surrogate models. Our results further show a positive association between calibration error and regret, but interestingly, this association disappears when we control for the type of surrogate model in the analysis. We also study the effect of recalibration and demonstrate that it generally does not lead to improved regret. Finally, we provide theoretical justification for why uncertainty calibration might be difficult to combine with BO due to the small sample sizes commonly used.

Temporal synchrony between facial motion and acoustic modulations is a hallmark feature of audiovisual speech. The moving face and mouth during natural speech is known to be correlated with low-frequency acoustic envelope fluctuations (below 10 Hz), but the precise rates at which envelope information is synchronized with motion in different parts of the face are less clear. Here, we used regularized canonical correlation analysis (rCCA) to learn speech envelope filters whose outputs correlate with motion in different parts of the speakers face. We leveraged recent advances in video-based 3D facial landmark estimation allowing us to examine statistical envelope-face correlations across a large number of speakers (∼4000). Specifically, rCCA was used to learn modulation transfer functions (MTFs) for the speech envelope that significantly predict correlation with facial motion across different speakers. The AV analysis revealed bandpass speech envelope filters at distinct temporal scales. A first set of MTFs showed peaks around 3-4 Hz and were correlated with mouth movements. A second set of MTFs captured envelope fluctuations in the 1-2 Hz range correlated with more global face and head motion. These two distinctive timescales emerged only as a property of natural AV speech statistics across many speakers. A similar analysis of fewer speakers performing a controlled speech task highlighted only the well-known temporal modulations around 4 Hz correlated with orofacial motion. The different bandpass ranges of AV correlation align notably with the average rates at which syllables (3-4 Hz) and phrases (1-2 Hz) are produced in natural speech. Whereas periodicities at the syllable rate are evident in the envelope spectrum of the speech signal itself, slower 1-2 Hz regularities thus only become prominent when considering crossmodal signal statistics. This may indicate a motor origin of temporal regularities at the timescales of syllables and phrases in natural speech.

Our global population contributes visual content on platforms like Instagram, attempting to express themselves and engage their audiences, at an unprecedented and increasing rate. In this paper, we revisit the popularity prediction on Instagram. We present a robust, efficient, and explainable baseline for population-based popularity prediction, achieving strong ranking performance. We employ the latest methods in computer vision to maximize the information extracted from the visual modality. We use transfer learning to extract visual semantics such as concepts, scenes, and objects, allowing a new level of scrutiny in an extensive, explainable ablation study. We inform feature selection towards a robust and scalable model, but also illustrate feature interactions, offering new directions for further inquiry in computational social science. Our strongest models inform a lower limit to population-based predictability of popularity on Instagram. The models are immediately applicable to social media monitoring and influencer identification.

Sustained attention is a cognitive ability to maintain task focus over extended periods of time (Mackworth, 1948; Chun, Golomb, & Turk-Browne, 2011). In this study, scalp electroencephalography (EEG) signals were processed in real time using a 32 dry-electrode system during a sustained visual attention task. An attention training paradigm was implemented, as designed in DeBettencourt, Cohen, Lee, Norman, and Turk-Browne (2015) in which the composition of a sequence of blended images is updated based on the participant's decoded attentional level to a primed image category. It was hypothesized that a single neurofeedback training session would improve sustained attention abilities. Twenty-two participants were trained on a single neurofeedback session with behavioral pretraining and posttraining sessions within three consecutive days. Half of the participants functioned as controls in a double-blinded design and received sham neurofeedback. During the neurofeedback session, attentional states to primed categories were decoded in real time and used to provide a continuous feedback signal customized to each participant in a closed-loop approach. We report a mean classifier decoding error rate of 34.3% (chance = 50%). Within the neurofeedback group, there was a greater level of task-relevant attentional information decoded in the participant's brain before making a correct behavioral response than before an incorrect response. This effect was not visible in the control group (interaction p=7.23e-4), which strongly indicates that we were able to achieve a meaningful measure of subjective attentional state in real time and control participants' behavior during the neurofeedback session. We do not provide conclusive evidence whether the single neurofeedback session per se provided lasting effects in sustained attention abilities. We developed a portable EEG neurofeedback system capable of decoding attentional states and predicting behavioral choices in the attention task at hand. The neurofeedback code framework is Python based and open source, and it allows users to actively engage in the development of neurofeedback tools for scientific and translational use.

Many years after online social networks exceeded our collective attention, social influence is still built on attention capital. Quality is not a prerequisite for viral spreading, yet large diffusion cascades remain the hallmark of a social influencer. Consequently, our exposure to low-quality content and questionable influence is expected to increase. Since the conception of influence maximization frameworks, multiple content performance metrics became available, albeit raising the complexity of influence analysis. In this paper, we examine and consolidate a diverse set of content engagement metrics. The correlations discovered lead us to propose a new, more holistic, one-dimensional engagement signal. We then show it is more predictable than any individual influence predictors previously investigated. Our proposed model achieves strong engagement ranking performance and is the first to explain half of the variance with features available early. We share the detailed numerical workflow to compute the new compound engagement signal. The model is immediately applicable to social media monitoring, influencer identification, campaign engagement forecasting, and curating user feeds.

How does missing data affect our ability to learn signal structures? It has been shown that learning signal structure in terms of principal components is dependent on the ratio of sample size and dimensionality and that a critical number of observations is needed before learning starts (Biehl and Mietzner, 1993). Here we generalize this analysis to include missing data. Probabilistic principal component analysis is regularly used for estimating signal structures in datasets with missing data. Our analytic result suggest that the effect of missing data is to effectively reduce signal-to-noise ratio rather than - as generally believed - to reduce sample size. The theory predicts a phase transition in the learning curves and this is indeed found both in simulation data and in real datasets.

Human brain activity as measured by fMRI exhibits strong correlations between brain regions which are believed to vary over time. Importantly, dynamic connectivity has been linked to individual differences in physiology, psychology and behavior, and has shown promise as a biomarker for disease. The state of the art in computational neuroimaging is to estimate the brain networks as relatively short sliding window covariance matrices, which leads to high variance estimates, thereby resulting in high overall error. This manuscript proposes a novel Bayesian model for dynamic brain connectivity. Motivated by the underlying neuroscience, the model estimates covariances which vary smoothly over time, with an instantaneous decomposition into a collection of spatially sparse components – resulting in parsimonious and highly interpretable estimates of dynamic brain connectivity. Simulated results are presented to illustrate the performance of the model even when it is mis-specified. For real brain imaging data with unknown ground truth, in addition to qualitative evaluation, we devise a simple classification task which suggests that the estimated brain networks better capture the underlying structure.

The field of automatic sleep stage classification based on EEG has enjoyed substantial attention during the last decade, which has resulted in several supervised classification algorithms with highly encouraging performance. Such supervised machine learning algorithms require large training sets that have been manually labelled, and are time- and resource-consuming to acquire. Here we present a semi-supervised approach that can learn to distinguish the sleep stages from a one-night data set where only a fraction has been manually labelled. We show that for fractions larger than 50%, our semi-supervised approach performs as good as a similar, fully-supervised model.

Dynamic functional connectivity has become a prominent approach for tracking the changes of macroscale statistical dependencies between regions in the brain. Effective parametrization of these statistical dependencies, referred to as brain states, is however still an open problem. We investigate different emission models in the hidden Markov model framework, each representing certain assumptions about dynamic changes in the brain. We evaluate each model by how well they can discriminate between schizophrenic patients and healthy controls based on a group independent component analysis of resting-state functional magnetic resonance imaging data. We find that simple emission models without full covariance matrices can achieve similar classification results as the models with more parameters. This raises questions about the predictability of dynamic functional connectivity in comparison to simpler dynamic features when used as biomarkers. However, we must stress that there is a distinction between characterization and classification, which has to be investigated further.

Model-based classification of sequence data using a set of hidden Markov models is a well-known technique. The involved score function, which is often based on the class-conditional likelihood, can, however, be computationally demanding, especially for long data sequences. Inspired by recent theoretical advances in spectral learning of hidden Markov models, we propose a score function based on third-order moments. In particular, we propose to use the Kullback-Leibler divergence between theoretical and empirical third-order moments for classification of sequence data with discrete observations. The proposed method provides lower computational complexity at classification time than the usual likelihood-based methods. In order to demonstrate the properties of the proposed method, we perform classification of both simulated data and empirical data from a human activity recognition study.

Trends, they are not only for the fashion industry after all. Within the engineering and computer science research communities as well, we periodically observe the phenomenon, see how certain methods suddenly start receiving particular attention, and sometimes, even though they emerge as an attractive solution for a given set of problems, they tend to become a hammer looking for new nails. At fi rst, using a new method on old problems is the natural and reasonable way to proceed. There have been remarkable successes achieved through the adoption of a tool from another fi eld or a new way of looking at old problems that brings new insights and solutions. There have been a number of such trends throughout the years in every field. In signal processing, a few notable ones include maximum entropy, wavelets, kernel methods, and the multiple up and down cycles of neural nets. A current tool that is on the rise is sparsity, more specifically solutions that promote sparsity, including coding, compressive sensing, sparse learning/estimation, sparse factorizations, and of course deep nets, which, without careful use of sparsity, would be useless. We will use sparsity as the major example in this Point of View article because it is current and illustrates the points we are making very well. While we agree that sparsity is very useful and has led to some excellent results in the past decade or so, it also allows us to address the dangers and pitfalls of blindly following trends in research. These problems are reflected in our publications and help define the overall research climate. In the following, we provide an overview on use of sparsity and then discuss a couple of specific problems.

EEG based brain state decoding has numerous applications. State of the art decoding is based on processing of the multivariate sensor space signal, however evidence is mounting that EEG source reconstruction can assist decoding. EEG source imaging leads to high-dimensional representations and rather strong a priori information must be invoked. Recent work by Edelman et al. (2016) has demonstrated that introduction of a spatially focal source space representation can improve decoding of motor imagery. In this work we explore the generality of Edelman et al. hypothesis by considering decoding of face recognition. This task concerns the differentiation of brain responses to images of faces and scrambled faces and poses a rather difficult decoding problem at the single trial level. We implement the pipeline using spatially focused features and show that this approach is challenged and source imaging does not lead to an improved decoding. We design a distributed pipeline in which the classifier has access to brain wide features which in turn does lead to a 15% reduction in the error rate using source space features. Hence, our work presents supporting evidence for the hypothesis that source imaging improves decoding.

In this work, we address the problem of solving a series of underdetermined linear inverse problemblems subject to a sparsity constraint. We generalize the spike-and-slab prior distribution to encode a priori correlation of the support of the solution in both space and time by imposing a transformed Gaussian process on the spike-and-slab probabilities. An expectation propagation (EP) algorithm for posterior inference under the proposed model is derived. For large scale problems, the standard EP algorithm can be prohibitively slow. We therefore introduce three different approximation schemes to reduce the computational complexity. Finally, we demonstrate the proposed model using numerical experiments based on both synthetic and real data sets.

Data augmentation is a key element in training high-dimensional models. In this approach, one synthesizes new observations by applying pre-specified transformations to the original training data; e.g. new images are formed by rotating old ones. Current augmentation schemes, however, rely on manual specification of the applied transformations, making data augmentation an implicit form of feature engineering. With an eye towards true end-to-end learning, we suggest learning the applied transformations on a per-class basis. Particularly, we align image pairs within each class under the assumption that the spatial transformation between images belongs to a large class of diffeomorphisms. We then learn a class-specific probabilistic generative models of the transformations in a Riemannian submanifold of the Lie group of diffeomorphisms. We demonstrate significant performance improvements in training deep neural nets over manually-specified augmentation schemes. Our code and augmented datasets are available online.

The multivariate normal density is a monotonic function of the distance to the mean, and its ellipsoidal shape is due to the underlying Euclidean metric. We suggest to replace this metric with a locally adaptive, smoothly changing (Riemannian) metric that favors regions of high local density. The resulting locally adaptive normal distribution (LAND) is a generalization of the normal distribution to the "manifold" setting, where data is assumed to lie near a potentially low-dimensional manifold embedded in R^D. The LAND is parametric, depending only on a mean and a covariance, and is the maximum entropy distribution under the given metric. The underlying metric is, however, non-parametric. We develop a maximum likelihood algorithm to infer the distribution parameters that relies on a combination of gradient descent and Monte Carlo integration. We further extend the LAND to mixture models, and provide the corresponding EM algorithm. We demonstrate the efficiency of the LAND to fit non-trivial probability distributions over both synthetic data, and EEG measurements of human sleep.

Through simulated EEG we investigate the effect of the forward model's applied skull:scalp conductivity ratio on the source reconstruction performance. We show that having a higher conductivity ratio generally leads to improvement of the solution. Additionally we see a clear connection between higher conductivity ratios and lower coherence, thus a reduction of the ill-posedness of the EEG inverse problem. Finally we show on real EEG data the stability of the strongest source recovered across conductivity ratios.

Correlated component analysis as proposed by Dmochowski, Sajda, Dias, and Parra (2012) is a tool for investigating brain process similarity in the responses to multiple views of a given stimulus. Correlated components are identified under the assumption that the involved spatial networks are identical. Here we propose a hierarchical probabilistic model that can infer the level of universality in such multiview data, from completely unrelated representations, corresponding to canonical correlation analysis, to identical representations as in correlated component analysis. This new model, which we denote Bayesian correlated component analysis, evaluates favorably against three relevant algorithms in simulated data. A well-established benchmark EEG data set is used to further validate the new model and infer the variability of spatial representations across multiple subjects.

The articles in this special issue focus on multimodal data fusion. These papers provide a review of current approaches and results on multimodal data fusion from different disciplines, and by including the required background in each domain, aims to provide a common forum for the exchange of ideas in this very active field of research.

A fundamental and general representation of audio and music which integrates multi-modal data sources is important for both application and basic research purposes. In this paper we address this challenge by proposing a multi-modal version of the Latent Dirichlet Allocation model which provides a joint latent representation. We evaluate this representation on the Million Song Dataset by integrating three fundamentally different modalities, namely tags, lyrics, and audio features. We show how the resulting representation is aligned with common 'cognitive' variables such as tags, and provide some evidence for the common assumption that genres form an acceptable categorization when evaluating latent representations of music. We furthermore quantify the model by its predictive performance in terms of genre and style, providing benchmark results for the Million Song Dataset.

We propose kernel parallel analysis (kPA) for automatic kernel scale and model order selection in Gaussian kernel principal component analysis (KPCA). Parallel analysis is based on a permutation test for covariance and has previously been applied for model order selection in linear PCA, we here augment the procedure to also tune the Gaussian kernel scale of radial basis function based KPCA. We evaluate kPA for denoising of simulated data and the U.S. postal data set of handwritten digits. We find that kPA outperforms other heuristics to choose the model order and kernel scale in terms of signal-to-noise ratio of the denoised data.

Recent neuroscience studies have shown that it is possible to predict how concrete objects are represented in the brain based on the semantic relations of words defining the corresponding concepts. Whether we read the word `smile' or recognize the same expression in a face, the mental processes captured as event related potentials in EEG brain imaging appear indistinguishable. As both low-level semantics and our affective responses can be encoded in words, we propose a simplified cognitive approach to model how we emotionally perceive media. Representing song texts in a vector space of reduced dimensionality using LSA, we define distances between lines of lyrics and frequently used emotional last.fm tags, that constrain the latent semantics according to the psychological dimensions of valence and arousal. We compare the LSA derived emotions from texts with the user annotated tag clouds describing the corresponding songs at last.fm, and suggest the retrieved patterns may provide a sparse representation of how we perceive the emotional content in media.

In this paper, we present a new evaluation approach for missing data techniques (MDTs) where the efficiency of those are investigated using listwise deletion method as reference. We experiment on classification problems and calculate misclassification rates (MR) for different missing data percentages (MDP) using a missing completely at random (MCAR) scheme. We compare three MDTs: pairwise deletion (PW), mean imputation (MI) and a maximum likelihood method that we call complete expectation maximization (CEM). We use a synthetic dataset, the Iris dataset and the Pima Indians Diabetes dataset. We train a Gaussian mixture model (GMM). We test the trained GMM for two cases, in which test dataset is missing or complete. The results show that CEM is the most efficient method in both cases while MI is the worst performer of the three. PW and CEM proves to be more stable, in particular for higher MDP values than MI.

Functional magnetic resonance imaging (fMRI) can be applied to study the functional connectivity of the neural elements which form complex network at a whole brain level. Most analyses of functional resting state networks (RSN) have been based on the analysis of correlation between the temporal dynamics of various regions of the brain. While these models can identify coherently behaving groups in terms of correlation they give little insight into how these groups interact. In this paper we take a different view on the analysis of functional resting state networks. Starting from the definition of resting state as functional coherent groups we search for functional units of the brain that communicate with other parts of the brain in a coherent manner as measured by mutual information. We use the infinite relational model (IRM) to quantify functional coherent groups of resting state networks and demonstrate how the extracted component interactions can be used to discriminate between functional resting state activity in multiple sclerosis and normal subjects.

Non-negative matrix factorization (NMF), i.e. V ap WH where both V, W and H are non-negative has become a widely used blind source separation technique due to its part based representation. The NMF decomposition is not in general unique and a part based representation not guaranteed. However, imposing sparseness both improves the uniqueness of the decomposition and favors part based representation. Sparseness in the form of attaining as many zero elements in the solution as possible is appealing from a conceptional point of view and corresponds to minimizing reconstruction error with an L0norm constraint. In general, solving for a given L0norm is an NP hard problem thus convex relaxation to regularization by the L1norm is often considered, i.e., minimizing (1/2||V - WH||F2+ lambda||H||1).An open problem is to control the degree of sparsity lambda imposed. We here demonstrate that a full regularization path for the L1norm regularized least squares NMF for fixed W can be calculated at the cost of an ordinary least squares solution based on a modification of the least angle regression and selection (LARS) algorithm forming a non-negativity constrained LARS (NLARS). With the full regularization path, the L1regularization strength lambda that best approximates a given L0can be directly accessed and in effect used to control the sparsity of H. The MATLAB code for the NLARS algorithm is available for download.

Nonlinear hemodynamic models express the BOLD (blood oxygenation level dependent) signal as a nonlinear, parametric functional of the temporal sequence of local neural activity. Several models have been proposed for both the neural activity and the hemodynamics. We compare two such combined models: the original balloon model with a square-pulse neural model (Friston, Mechelli, Turner, & Price, 2000) and an extended balloon model with a more sophisticated neural model (Buxton, Uludag, Dubowitz, & Liu, 2004). We learn the parameters of both models using a Bayesian approach, where the distribution of the parameters conditioned on the data is estimated using Markov chain Monte Carlo techniques. Using a split-half resampling procedure (Strother, Anderson, & Hansen, 2002), we compare the generalization abilities of the models as well as their reproducibility, for both synthetic and real data, recorded from two different visual stimulation paradigms. The results show that the simple model is the better one for these data.

There is a increasing interest in analysis of large-scale multiway data. The concept of multiway data refers to arrays of data with more than two dimensions, that is, taking the form of tensors. To analyze such data, decomposition techniques are widely used. The two most common decompositions for tensors are the Tucker model and the more restricted PARAFAC model. Both models can be viewed as generalizations of the regular factor analysis to data of more than two modalities. Nonnegative matrix factorization (NMF), in conjunction with sparse coding, has recently been given much attention due to its part-based and easy interpretable representation. While NMF has been extended to the PARAFAC model, no such attempt has been done to extend NMF to the Tucker model. However, if the tensor data analyzed are nonnegative, it may well be relevant to consider purely additive (i.e., nonnegative) Tucker decompositions). To reduce ambiguities of this type of decomposition, we develop updates that can impose sparseness in any combination of modalities, hence, proposed algorithms for sparse nonnegative Tucker decompositions (SN-TUCKER). We demonstrate how the proposed algorithms are superior to existing algorithms for Tucker decompositions when the data and interactions can be considered nonnegative. We further illustrate how sparse coding can help identify what model (PARAFAC or Tucker) is more appropriate for the data as well as to select the number of components by turning off excess components. The algorithms for SN-TUCKER can be downloaded from Mørup (2007).

An integral representation for the minimum mean-square error (MMSE) estimator for a random variable in an observation model consisting of a linear combination of two random variables is derived. The derivation is based on the moment-generating functions for the random variables in the observation model. The method generalizes so that integral representations for higher-order moments of the posterior of interest can be easily obtained. Two examples are presented that demonstrate how saddle-point approximation can be used to obtain accurate approximations for a MMSE estimator using the derived integral representation. However, the examples also demonstrate that when two saddle points are close or coalesce, then saddle-point approximation based on isolated saddle points is not valid. A saddle-point approximation based on two close or coalesced saddle points is derived and in the examples, the validity and accuracy of the derivation is demonstrated.

In positron emission tomography, kinetic modelling of brain tracer uptake, metabolism or binding requires knowledge of the cerebral input function. Traditionally, this is achieved with arterial blood sampling in the arm or as shown in (Liptrot, M, et al., 2004) by non-invasive K-means clustering. We propose another method to estimate time-activity curves (TAC) extracted directly from dynamic positron emission tomography (PET) scans by non-negative matrix factorization (NMF). Since the scaling of the basis curves is lost in the NMF the estimated TAC is scaled by a vector alpha which is calculated from the NMF solution. The method is tested on a [18F]-Altanserin tracer ligand data set consisting of 5 healthy subjects. The results from using K-means clustering and NMF are compared to a sampled arterial TAC. The comparison is done by calculating the correlation with the arterial sampled TAC.

We generalize the non-negative matrix factorization (NMF) generative model to incorporate an explicit offset. Multiplicative estimation algorithms are provided for the resulting sparse affine NMF model. We show that the affine model has improved uniqueness properties and leads to more accurate identification of mixing and sources.

The paper describes our work on the development of a system for retrieval of relevant stories from broadcast news. The system utilizes a combination of audio processing and text mining. The audio processing consists of a segmentation step that partitions the audio into speech and music. The speech is further segmented into speaker segments and then transcribed using an automatic speech recognition system, to yield text input for clustering using non-negative matrix factorization (NMF). We find semantic topics that are used to evaluate the performance for topic detection. Based on these topics we show that a novel query expansion can be performed to return more intelligent search results. We also show that the query expansion helps overcome errors of the automatic transcription.

Automatic knowledge extraction from music signals is a key component for most music organization and music information retrieval systems. In this paper, we consider the problem of instrument modelling and instrument classification from the rough audio data. Existing systems for automatic instrument classification operate normally on a relatively large number of features, from which those related to the spectrum of the audio signal are particularly relevant. In this paper, we confront two different models about the spectral characterization of musical instruments. The first assumes a constant envelope of the spectrum (i.e., independent from the pitch), whereas the second assumes a constant relation among the amplitude of the harmonics. The first model is related to the Mel frequency cepstrum coefficients (MFCCs), while the second leads to what we will refer to as harmonic representation (HR). Experiments on a large database of real instrument recordings show that the first model offers a more satisfactory characterization, and therefore MFCCs should be preferred to HR for instrument modelling/classification.

We present a new algorithm for maximum likelihood convolutive independent component analysis (ICA) in which components are unmixed using stable autoregressive filters determined implicitly by estimating a convolutive model of the mixing process. By introducing a convolutive mixing model for the components, we show how the order of the filters in the model can be correctly detected using Bayesian model selection. We demonstrate a framework for deconvolving a subspace of independent components in electroencephalography (EEG). Initial results suggest that in some cases, convolutive mixing may be a more realistic model for EEG signals than the instantaneous ICA model.

Temporal feature integration is the process of combining all the feature vectors in a time window into a single feature vector in order to capture the relevant temporal information in the window. The mean and variance along the temporal dimension are often used for temporal feature integration, but they capture neither the temporal dynamics nor dependencies among the individual feature dimensions. Here, a multivariate autoregressive feature model is proposed to solve this problem for music genre classification. This model gives two different feature sets, the diagonal autoregressive (DAR) and multivariate autoregressive (MAR) features which are compared against the baseline mean-variance as well as two other temporal feature integration techniques. Reproducibility in performance ranking of temporal feature integration methods were demonstrated using two data sets with five and eleven music genres, and by using four different classification schemes. The methods were further compared to human performance. The proposed MAR features perform better than the other features at the cost of increased computational complexity.

It has been previously observed that independent component analysis (ICA), if applied to data pooled in a particular way, may lessen the need for spatial alignment of scans in a functional neuroimaging study. In this paper, we seek to determine analytically the conditions under which this observation is true, not only for spatial ICA, but also for temporal ICA and for principal component analysis (PCA). In each case, we find conditions that the spatial alignment operator must satisfy to ensure invariance of the results. We illustrate our findings using functional magnetic-resonance imaging (fMRI) data. Our analysis is applicable to both intersubject and intrasubject spatial normalization.

Cognitive component analysis (COCA) is defined as the process of unsupervised grouping of data such that the resulting group structure is well-aligned with that resulting from human cognitive activity (L.K. Hansen et al., 2005). In this paper we address COCA in the context short time sound features, finding phonemes which are the smallest contrastive units in the sound system of a language. Generalizable components were found deriving from phonemes based on homomorphic filtering features with basic time scale (20 msec). We sparsified the features based on energy as a preprocessing means to eliminate the intrinsic noise. Independent component analysis was compared with latent semantic indexing, and was demonstrated to be a more appropriate model in COCA

A sound classification model is presented that can classify signals into music, noise and speech. The model extracts the pitch of the signal using the harmonic product spectrum. Based on the pitch estimate and a pitch error measure, features are created and used in a probabilistic model with soft-max output function. Both linear and quadratic inputs are used. The model is trained on 2 hours of sound and tested on publicly available data. A test classification error below 0.05 with 1 s classification windows is achieved. Further more it is shown that linear input performs as well as a quadratic, and that even though classification gets marginally better, not much is achieved by increasing the window size beyond 1 s

We demonstrate that blind separation of more sources than sensors can be performed based solely on the second order statistics of the observed mixtures. This generalization of well-known robust algorithms are suited for equal number of sources and sensors. It is assumed that the sources are non-stationary and sparsely distributed in the time-frequency plane. The mixture model is convolutive, i.e. acoustic setups such as the cocktail party problem are contained. The limits of identifiability are determined in the framework of the PARAFAC model. In the experimental section, it is demonstrated that real room recordings of 3 speakers by 2 microphones can be separated using the method

A method for detecting clutter in weather radar images by information fusion is presented. Radar data, satellite images, and output from a numerical weather prediction model are combined and the radar echoes are classified using supervised classification. The presented method uses indirect information on precipitation in the atmosphere from Meteosat-8 multispectral images and near-surface temperature estimates from the DMIHIRLAM-S05 numerical weather prediction model. Alternatively, an operational now casting product called 'Precipitating Clouds' based on Meteosat-8 input is used. A scale-space ensemble method is used for classification and the clutter detection method is illustrated on a case of severe sea clutter contaminated radar data. Detection accuracies above 90 % are achieved and using an ensemble classification method the error rate is reduced by 40 %.

In this paper we are presenting a novel multivariate analysis method. Our scheme is based on a novel kernel orthonormalized partial least squares (PLS) variant for feature extraction, imposing sparsity constrains in the solution to improve scalability. The algorithm is tested on a benchmark of UCI data sets, and on the analysis of integrated short-time music features for genre prediction. The upshot is that the method has strong expressive power even with rather few features, is clearly outperforming the ordinary kernel PLS, and therefore is an appealing method for feature extraction of labelled data.

MOTIVATION: Hierarchical and relocation clustering (e.g. K-means and self-organizing maps) have been successful tools in the display and analysis of whole genome DNA microarray expression data. However, the results of hierarchical clustering are sensitive to outliers, and most relocation methods give results which are dependent on the initialization of the algorithm. Therefore, it is difficult to assess the significance of the results. We have developed a consensus clustering algorithm, where the final result is averaged over multiple clustering runs, giving a robust and reproducible clustering, capable of capturing small signal variations. The algorithm preserves valuable properties of hierarchical clustering, which is useful for visualization and interpretation of the results. RESULTS: We show for the first time that one can take advantage of multiple clustering runs in DNA microarray analysis by collecting re-occurring clustering patterns in a co-occurrence matrix. The results show that consensus clustering obtained from clustering multiple times with Variational Bayes Mixtures of Gaussians or K-means significantly reduces the classification error rate for a simulated dataset. The method is flexible and it is possible to find consensus clusters from different clustering algorithms. Thus, the algorithm can be used as a framework to test in a quantitative manner the homogeneity of different clustering algorithms. We compare the method with a number of state-of-the-art clustering methods. It is shown that the method is robust and gives low classification error rates for a realistic, simulated dataset. The algorithm is also demonstrated for real datasets. It is shown that more biological meaningful transcriptional patterns can be found without conservative statistical or fold-change exclusion of data. AVAILABILITY: Matlab source code for the clustering algorithm ClusterLustre, and the simulated dataset for testing are available upon request from T.G. and O.W.

We apply a type of generative modelling to the problem of blind source separation in which prior knowledge about the latent source signals, such as time-varying auto-correlation and quasi-periodicity, are incorporated into a linear state-space model. In simulations, we show that in terms of signal-to-error ratio, the sources are inferred more accurately as a result of the inclusion of strong prior knowledge. We explore different schemes of maximum-likelihood optimization for the purpose of learning the model parameters. The Expectation Maximization algorithm, which is often considered the standard optimization method in this context, results in slow convergence when the noise variance is small. In such scenarios, quasi-Newton optimization yields substantial improvements in a range of signal to noise ratios. We analyze the performance of the methods on convolutive mixtures of speech signals.

We analyze convergence of the expectation maximization (EM) and variational Bayes EM (VBEM) schemes for parameter estimation in noisy linear models. The analysis shows that both schemes are inefficient in the low-noise limit. The linear model with additive noise includes as special cases independent component analysis, probabilistic principal component analysis, factor analysis, and Kalman filtering. Hence, the results are relevant for many practical applications.

An online blind source separation algorithm, which is a special case of the geometric algorithm by Parra and Fancourt (Book chapt. in: Noise Reduction in Speech App., Ed. Gillian Davis, CRC Press LLC, 2002), has been implemented for the purpose of separating sounds recorded by microphones placed at each side of the head. By using the assumption that the position of the two sounds are known, the source separation algorithm has been geometrically constrained. Since the separation takes place in a non free-field, a head-related transfer function (HRTF) is used to simulate the response between microphones placed at the two ears. The use of an HRTF, instead of assuming free-field, improves the separation by approximately 1 dB, compared to when free-field is assumed. This indicates that the permutation ambiguity is solved more accurately compared to when free-field is assumed.

Language independent 'bag-of-words' representations are surprisingly effective for text classification. The representation is high dimensional though, containing many non-consistent words for text categorization. These non-consistent words result in reduced generalization performance of subsequent classifiers, e.g., from ill-posed principal component transformations. In this communication, our aim is to study the effect of reducing the least relevant words from the bag-of-words representation. We consider a new approach, using neural network based sensitivity maps and information gain for determination of term relevancy, when pruning the vocabularies. With reduced vocabularies, documents are classified using a latent semantic indexing representation and a probabilistic neural network classifier. Reducing the bag-of-words vocabularies with 90%-98%, we find consistent classification improvement using two mid size data-sets. We also study the applicability of information gain and sensitivity maps for automated keyword generation.

We discuss an identification framework for noisy speech mixtures. A block-based generative model is formulated that explicitly incorporates the time-varying harmonic plus noise (H+N) model for a number of latent sources observed through noisy convolutive mixtures. All parameters including the pitches of the source signals, the amplitudes and phases of the sources, the mixing filters and the noise statistics are estimated by maximum likelihood, using an EM-algorithm. Exact averaging over the hidden sources is obtained using the Kalman smoother. We show that pitch estimation and source separation can be performed simultaneously. The pitch estimates are compared to laryngograph (EGG) measurements. Artificial and real room mixtures are used to demonstrate the viability of the approach. Intelligible speech signals are re-synthesized from the estimated H+N models.

The Relevance Vector Machine (RVM) introduced by Tipping is a probabilistic model similar to the widespread Support Vector Machines (SVM), but where the training takes place in a Bayesian framework, and where predictive distributions of the outputs instead of point estimates are obtained. In this paper we focus on the use of RVM's for regression. We modify this method for training generalized linear models by adapting automatically the width of the basis functions to the optimal for the data at hand. Our Adaptive RVM is tried for prediction on the chaotic Mackey-Glass time series. Much superior performance than with the standard RVM and than with other methods like neural networks and local linear models is obtained.

We extend MacKays Bayesian approach to neural classifiers to include an outlier detector mechanism. We show that the outlier detector can locate misclassified samples.

We derive an independent-component analysis (ICA) method for analyzing two or more data sets simultaneously. Our model permits there to be components individual to the various data sets, and others that are common to all the sets. We explore the assumed time autocorrelation of independent signal components and base our algorithm on prediction analysis. We illustrate the algorithm using a simple image separation example. Our aim is to apply this method to functional brain mapping using functional magnetic resonance imaging (fMRI).

We develop mean-field approaches for probabilistic independent component analysis (ICA). The sources are estimated from the mean of their posterior distribution and the mixing matrix (and noise level) is estimated by maximum a posteriori (MAP). The latter requires the computation of (a good approximation to) the correlations between sources. For this purpose, we investigate three increasingly advanced mean-field methods: the variational (also known as naive mean field) approach, linear response corrections, and an adaptive version of the Thouless, Anderson and Palmer (1977) (TAP) mean-field approach, which is due to Opper and Winther (2001). The resulting algorithms are tested on a number of problems. On synthetic data, the advanced mean-field approaches are able to recover the correct mixing matrix in cases where the variational meanfield theory fails. For handwritten digits, sparse encoding is achieved using nonnegative source and mixing priors. For speech, the mean-field method is able to separate in the underdetermined (overcomplete) case of two sensors and three sources. One major advantage of the proposed method is its generality and algorithmic simplicity. Finally, we point out several possible extensions of the approaches developed here.

The second moment-based independent component analysis scheme of Molgedey and Schuster (1994) is generalized to fractional low-order moments, relevant for linear mixtures of heavy tail stable processes. The Molgedey-Schuster algorithm stands out by allowing explicitly construction of the independent components. Surprisingly, this turns out to be possible also for decorrelation based on fractional low-order moments.

In certain applications of independent component analysis (ICA) it is of interest to test hypotheses concerning the number of components or simply to test whether a given number of components is significant relative to a "white noise" null hypothesis. We estimate probabilities of such competing hypotheses for ICA based on dynamic decorrelation. The probabilities are evaluated in the so-called Bayesian information criterion approximation, however, they are able to detect the content of dynamic components as efficiently as an unbiased test set estimator.

We apply and discuss generalizable Gaussian mixture (GGM) models for text mining. The model automatically adapts model complexity for a given text representation. We show that the generalizability of these models depends on the dimensionality of the representation and the sample size. We discuss the relation between supervised and unsupervised learning in the test data. Finally, we implement a novelty detector based on the density model.

We propose a general Bayesian framework for performing independent component analysis (leA) which relies on ensemble learning and lin(cid:173) ear response theory known from statistical physics. We apply it to both discrete and continuous sources. For the continuous source the underde(cid:173) termined (overcomplete) case is studied. The naive mean-field approach fails in this case whereas linear response theory-which gives an improved estimate of covariances-is very efficient. The examples given are for sources without temporal correlations. However, this derivation can eas(cid:173) ily be extended to treat temporal correlations. Finally, the framework offers a simple way of generating new leA algorithms without needing to define the prior distribution of the sources explicitly.

We demonstrate that statistical analysis of ill-posed data sets is subject to a bias, which can be observed when projecting indepen(cid:173) dent test set examples onto a basis defined by the training exam(cid:173) ples. Because the training examples in an ill-posed data set do not fully span the signal space the observed training set variances in each basis vector will be too high compared to the average vari(cid:173) ance of the test set projections onto the same basis vectors. On basis of this understanding we introduce the Generalizable Singu(cid:173) lar Value Decomposition (GenSVD) as a means to reduce this bias by re-estimation of the singular values obtained in a conventional Singular Value Decomposition, allowing for a generalization perfor(cid:173) mance increase of a subsequent statistical model. We demonstrate that the algorithm succesfully corrects bias in a data set from a functional PET activation study of the human brain. 1 Ill-posed Data Sets An ill-posed data set has more dimensions in each example than there are examples. Such data sets occur in many fields of research typically in connection with image measurements. The associated statistical problem is that of extracting structure from the observed high-dimensional vectors in the presence of noise. The statistical analysis can be done either supervised (Le. modelling with target values: classifi(cid:173) cation, regresssion) or unsupervised (modelling with no target values: clustering, PCA, ICA). In both types of analysis the ill-posedness may lead to immediate prob(cid:173) lems if one tries to apply conventional statistical methods of analysis, for example the empirical covariance matrix is prohibitively large and will be rank-deficient. A common approach is to use Singular Value Decomposition (SVD) or the analogue Principal Component Analysis (PCA) to reduce the dimensionality of the data. Let the N observed i-dimensional samples Xj, j = L .N, collected in the data matrix X = [Xl ... XN] of size I x N, I> N . The SVD-theorem states that such a matrix can be decomposed as

Modeling the haemodynamic response in functional magnetic resonance (fMRI) experiments is an important aspect of the analysis of functional neuroimages. This has been done in the past using parametric response function, from a limited family. In this contribution, we adopt a semi-parametric approach based on finite impulse response (FIR) filters. In order to cope with the increase in the number of degrees of freedom, we introduce a Gaussian process prior on the filter parameters. We show how to carry on the analysis by incorporating prior knowledge on the filters, optimizing hyper-parameters using the evidence framework, or sampling using a Markov Chain Monte Carlo (MCMC) approach. We present a comparison of our model with standard haemodynamic response kernels on simulated data, and perform a full analysis of data acquired during an experiment involving visual stimulation.

This paper describes an underwater object discrimination system applied to recognize cylinders of various compositions from different angles. The system is based on a new combination of simulated dolphin clicks, simulated auditory filters and artificial neural networks. The model demonstrates its potential on real data collected from four different cylinders in an environment where the angles were controlled in order to evaluate the models capabilities to recognize cylinders independent of angles.

This paper addresses the problem of neuro-anatomical registration across individuals for functional [15O] water PET activation studies. A new algorithm for three-dimensional (3-D) nonlinear structural registration (warping) of MR scans is presented. The method performs a hierarchically scaled search for a displacement field, maximizing one of several voxel similarity measures derived from the two-dimensional (2-D) histogram of matched image intensities, subject to a regularizer that ensures smoothness of the displacement field. The effect of the nonlinear structural registration is studied when it is computed on anatomical MR scans and applied to coregistered [15O] water PET scans from the same subjects: in this experiment, a study of visually guided saccadic eye movements. The performance of the nonlinear warp is evaluated using multivariate functional signal and noise measures. These measures prove to be useful for comparing different intersubject registration approaches, e.g., affine versus nonlinear. A comparison of 12-parameter affine registration versus non-linear registration demonstrates that the proposed nonlinear method increases the number of voxels retained in the cross-subject mask. We demonstrate that improved structural registration may result in an improved multivariate functional signal-to-noise ratio (SNR). Furthermore, registration of PET scans using the 12-parameter affine transformations that align the coregistered MR images does not improve registration, compared to 12-parameter affine alignment of the PET images directly.

This paper addresses a new framework for designing robust neural network classifiers. The network is optimized using the maximum a posteriori technique, i.e., the cost function is the sum of the log-likelihood and a regularization term (prior). In order to perform robust classification, we present a modified likelihood function which incorporates the potential risk of outliers in the data. This leads to the introduction of a new parameter, the outlier probability. Designing the neural classifier involves optimization of network weights as well as outlier probability and regularization parameters. We suggest to adapt the outlier probability and regularisation parameters by minimizing the error on a validation set, and a simple gradient descent scheme is derived. In addition, the framework allows for constructing a simple outlier detector. Experiments with artificial data demonstrate the potential of the suggested framework.

This paper presents a quantitative comparison of three types of information available for 52 patients scheduled for carotid endarterectomy: subjective classification of the ultrasound images obtained during scanning before operation, first-and second-order statistical features extracted from regions of the plaque in still ultrasound images from three orthogonal scan planes and finally a histological analysis of the surgically removed plaque. The quantitative comparison was made with the linear model and with separation of the available data into training and test sets. The comparison of subjective classification with features from still ultrasound images revealed an overall agreement of 60% for classification of echogenicity and 70% for classification of structure. Comparison of the histologically determined relative volume of soft materials with features from the still images revealed a correlation coefficient of r = -0.42(p = 0.002). for mean echogenicity of the plaque region. The best performing feature was of second order and denoted Contrast (r = -0.5). Though significant, the latter correlation is probably not strong enough to be useful for clinical prediction of relative volume of soft materials for individual patients. Reasons for this is discussed in the paper, together with suggestions for improvements.

This paper addresses the problem of generalization error estimation in neural networks. A new early stop criterion based on a Bootstrap estimate of the generalization error is suggested. The estimate does not require the network to be trained to the minimum of the cost function, as required by other methods based on asymptotic theory. Moreover, in contrast to methods based on cross-validation which require data left out for testing, and thus biasing the estimate, the Bootstrap technique does not have this disadvantage. The potential of the suggested technique is demonstrated on various time-series problems.

A better understanding of pruning methods based on a ranking of weights according to their saliency in a trained network requires further information on the statistical properties of such saliencies. We focus on two-layer networks with either a linear or nonlinear output unit, and obtain analytic expressions for the distribution of saliencies and their logarithms. Our results reveal unexpected universal properties of the log-saliency distribution and suggest a novel algorithm for saliency-based weight ranking that avoids the numerical cost of second derivative evaluations.

This article presents a generalization of the Boltzmann machine that allows us to use the learning rule for a much wider class of maximum likelihood and maximum a posteriori problems, including both supervised and unsupervised learning. Furthermore, the approach allows us to discuss regularization and generalization in the context of Boltzmann machines. We provide an illustrative example concerning parameter estimation in an inhomogeneous Markov field. The regularized adaptation produces a parameter set that closely resembles the "teacher" parameters, hence, will produce segmentations that closely reproduce those of the inhomogeneous teacher network.

Second order properties of cost functions for recurrent networks are investigated. We analyze a layered fully recurrent architecture, the virtue of this architecture is that it features the conventional feedforward architecture as a special case. A detailed description of recursive computation of the full Hessian of the network cost func(cid:173) tion is provided. We discuss the possibility of invoking simplifying approximations of the Hessian and show how weight decays iron the cost function and thereby greatly assist training. We present tenta(cid:173) tive pruning results, using Hassibi et al.'s Optimal Brain Surgeon, demonstrating that recurrent networks can construct an efficient internal memory. 1 LEARNING IN RECURRENT NETWORKS Time series processing is an important application area for neural networks and numerous architectures have been suggested, see e.g. (Weigend and Gershenfeld, 94). The most general structure is a fully recurrent network and it may be adapted using Real Time Recurrent Learning (RTRL) suggested by (Williams and Zipser, 89). By invoking a recurrent network, the length of the network memory can be adapted to the given time series, while it is fixed for the conventional lag-space net (Weigend et al., 90). In forecasting, however, feedforward architectures remain the most popular structures; only few applications are reported based on the Williams&Zipser approach. The main difficulties experienced using RTRL are slow convergence and 674 Morten With Pedersen, Lars Kai Hansen lack of generalization. Analogous problems in feedforward nets are solved using second order methods for training and pruning (LeCun et al., 90; Hassibi et al., 92; Svarer et al., 93). Also, regularization by weight decay significantly improves training and generalization. In this work we initiate the investigation of second order properties for RTRL; a detailed calculation scheme for the cost function Hessian is presented, the importance of weight decay is demonstrated, and preliminary pruning results using Hassibi et al.'s Optimal Brain Surgeon (OBS) are presented. We find that the recurrent network discards the available lag space and constructs its own efficient internal memory. 1.1 REAL TIME RECURRENT LEARNING The fully connected feedback nets studied by Williams&Zipser operate like a state machine, computing the outputs from the internal units according to a state vector z(t) containing previous external inputs and internal unit outputs. Let x(t) denote a vector containing the external inputs to the net at time t, and let y(t) denote a vector containing the outputs of the units in the net. We now arrange the indices on x and y so that the elements of z(t) can be defined as , k E I , k E U where I denotes the set of indices for which Zk is an input, and U denotes the set of indices for which Zk is the output of a unit in the net. Thresholds are implemented using an input permanently clamped to unity. The k'th unit in the net is now updated according to where Wkj denotes the weight to unit k from input/unit j and "'0 is the activation function of the k'th unit. When used for time series prediction, the input vector (excluding threshold) is usually defined as x(t) = [x(t), . .. , x(t - L + 1)] where L denotes the dimension of the lag space. One of the units in the net is designated to be the output unit Yo, and its activating function 10 is often chosen to be linear in order to allow for arbitrary dynamical range. The prediction of x(t + 1) is x(t + 1) = lo[so(t»). Also, if the first prediction is at t = 1, the first example is presented at t = 0 and we 'set y(O) = O. We analyse here a modification of the standard Williams&Zipser construction that is appropriate for forecasting purposes. The studied architecture is layered. Firstly, we remove the external inputs from the linear output unit in order to prevent the network from getting trapped in a linear mode. The output then reads x(t + 1) = Yo(t + 1) = L WojYj(t) + Wthres,o

Inspired by the recent upsurge of interest in Bayesian methods we consider adaptive regularization. A generalization based scheme for adaptation of regularization parameters is introduced and compared to Bayesian regularization. We show that pruning arises naturally within both adaptive regularization schemes. As model example we have chosen the simplest possible: estimating the mean of a random variable with known variance. Marked similarities are found between the two methods in that they both involve a “noise limit,” below which they regularize with infinite weight decay, i.e., they prune. However, pruning is not always beneficial. We show explicitly that both methods in some cases may increase the generalization error. This corresponds to situations where the underlying assumptions of the regularizer are poorly matched to the environment.

The optimal brain damage (OBD) scheme of Le Cun, Denker and Solla for pruning of feedforward networks has been implemented and applied to the contiguity classification problem. It is shown that OBD improves the learning curve (the test error as a function of the number of examples). By inspecting the architectures obtained through pruning, it is found that the networks with less parameters have the smallest test error in agreement with "Ockhams Razor". Based on this, we propose a heuristic which selects the smallest successful architecture among a group of pruned networks and we show that it leads to very efficient optimization of the architecture. The validity of the approximations involved in OBD are discussed and it is found that they are surprisingly accurate for the problem studied.

Several means for improving the performance and training of neural networks for classification are proposed. Crossvalidation is used as a tool for optimizing network parameters and architecture. It is shown that the remaining residual generalization error can be reduced by invoking ensembles of similar networks.>