In this paper we have studied the use of cascade-correlation (CC) trained multilayer perceptron (MLP) networks with a maximum covariance (MC) initialization scheme for adaptive equalization of binary data bursts in a baseband digital communication system. Conventional MLP networks have been found to perform well in various adaptive equalization tasks, although they often require quite a lot of computation. Here, we have used the CC training method for MLP networks in order to be able to create a nonlinear neural network equalizer with an adaptive structure. The CC training method finds a suitable sized network for each channel response and thus decreases the amount of computation needed. To further decrease the computational load of the system, we have applied the MC initialization scheme for weight initialization, which results in faster convergence and thus decreases the amount of training needed in the network.

A hybrid neural network architecture is investigated for classification purposes. The proposed hybrid is based on the multilayer perceptron (MLP) network. In addition to the usual hidden layers the first hidden layer is selected to be an adaptive centroid layer. Each unit in this new layer incorporates a centroid vector that is located somewhere in the space spanned by the input variables. The output of these units is the Euclidean distance between the centroid vector and the inputs. The centroid layer has some resemblance to the hidden layer of the radial basis function (RBF) networks. Therefore the proposed design can be regarded as a sort of hybrid of the MLP and RBF networks. The presented benchmark experiments demonstrate that the proposed hybrid can provide significant advantages over standard MLPs in terms of fast and efficient learning, and compact network structure.

The main advantages of cascade-correlation learning are the abilities to learn quickly and to determine the network size. However, recent studies have shown that in many problems the generalization performance of a cascade-correlation trained network may not be quite optimal. Moreover, to reach a certain performance level, a larger network may be required than with other training methods. Recent advances in statistical learning theory emphasize the importance of a learning method to be able to learn optimal hyperplanes. This has led to advanced learning methods, which have demonstrated substantial performance improvements. Based on these recent advances in statistical learning theory, we introduce modifications to the standard cascade-correlation learning that take into account the optimal hyperplane constraints. Experimental results demonstrate that with modified cascade correlation, considerable performance gains are obtained compared to the standard cascade-correlation learning. This includes better generalization, smaller network size, and faster learning.

We have developed a method for finding the minimum or near minimum number of centroid units for the centroid based multilayer perceptron (CMLP) network. When near minimum number of units is used the problem of weight initialization becomes a more dominant factor in the training process. Even a single poorly initialized weight can cause one of the units to become useless in the network operation. However, in a near minimum sized network all the units must function properly. The purpose of this study is to address the problem of weight initialization in a CMLP network where minimum number of centroid units is used. Based on the inverse principal component analysis, we propose an efficient initialization method for the centroid units. In addition, we describe an initialization method for the MLP part of the CMLP. The benchmark simulations demonstrate that the proposed initialization scheme can significantly improve the convergence. The cost of initialization is relatively small compared to the actual training process.

The concept of optimal hyperplane has been recently investigated in the context of statistical learning theory. The important property of an optimal hyperplane is that it provides maximum margins to each class to be separated. Obviously, such a decision boundary is expected to yield good generalization. In neural network learning techniques, the majority of them do not make use of the optimal hyperplane concept. As a result, in many cases extensive tuning is required to reach good generalization. In this study we consider adaptive training schemes for optimal margin classification with neural networks. We describe some novel schemes and compare them with the conventional schemes. Simple experiments are presented to demonstrate the performance of each scheme.

The so called candidate training is commonly used to deal with the initialization problem in the cascade-correlation learning. There several candidate hidden units are first trained, and then the one yielding the best value for the covariance criterion is installed in the network. In the case where there are many candidate units to be trained, the total computational cost of the training can become very large. Here we consider an approach for weight initialization in the cascade-correlation learning. The proposed method is based on the concept of stepwise regression. Empirical simulations demonstrate that the proposed method can substantially speed-up the cascade-correlation learning compared to the case where the candidate training is used. Moreover the overall performance remained the same or was even better than with the candidate training.

In our recent studies we have investigated a centroid based multilayer perceptron (CMLP) network architecture for nonlinear modelling purposes. In CMLP network the first hidden layer is a centroid layer which performs a circular transformation for the input data. The output of this centroid layer is then fed to a usual multilayer perceptron network. We have found that this hybrid can provide significant advantages over standard multilayer perceptron networks in terms of fast and efficient learning, and compact network structure in complex classification problems. Previously the number of units for the centroid layer has been determined empirically. Here we extend our work by introducing a method for determining the minimum or near minimum number of units for the centroid layer in a given problem. The proposed scheme is based on the usage of principal component analysis. The presented benchmark simulations demonstrate that the proposed method can effectively discover the minimum number of centroid units. However such a minimum structure is shown to be relatively sensitive for the weight initialization.

Choosing a network size is a difficult problem in neural network modelling. In many recent studies constructive or destructive methods that add or delete connections, neurons, layers have been studied for solving this problem In this work we consider the constructive approach. In particular we address the construction of feedforward networks by the use of improved constructive backpropagation that utilizes output bias decomposition scheme. The proposed improved scheme is shown to be beneficial especially in regression type problems like time series modelling. Namely, our time series prediction experiments demonstrate that the improved method is competitive in terms of modelling performance and training time compared to the well known cascade-correlation method.

We address the construction of recurrent neural networks by the use of constructive backpropagation (CBP). The benefits of the proposed scheme include: 1) fully recurrent networks with arbitrary number of layers can be constructed efficiently; and 2) after the network has been constructed one can continue the adaptation of the network weights as well as continue structure adaptation. This includes both addition and deletion of neurons/layers in a computationally efficient manner. Thus the investigated method is very flexible compared to many previous methods. In addition, according to our time series prediction experiments, the proposed method is competitive compared to the well known recurrent cascade-correlation method.

This paper compares two noise compensation techniques for increasing recognition performance on tone corrupted speech from payphones. An analysis of payphone speech is made which identifies two harmful processes signalling tones and increased background noise. The techniques of spectral subtraction and parallel model combination are compared on this task and are shown to give some improvement. Further robustness is demonstrated using RASTA-CTM speech features.

In recent work, the all-pass transform (APT) was proposed as the basis of a speaker adaptation scheme intended for use with a large vocabulary speech recognition system. It was shown that APT-based adaptation reduces to a linear transformation of cepstral means, much like the better known maximum likelihood linear regression (MLLR), but is specified by far fewer free parameters. Due to its linearity, APT-based adaptation can be used in conjunction with speaker-adapted training (SAT), an algorithm for performing maximum likelihood estimation of the parameters of an HMM when speaker adaptation is to be employed during both training and test. In this work, we propose a refinement of SAT called single-pass adapted training (SPAT) which achieves the same improvement in system performance as SAT but requires much less computation for HMM training. In a set of speech recognition experiments conducted on the Switchboard Corpus, we report a word error rate reduction of 5.3% absolute using a single, global APT.

A hybrid neural network architecture is investigated for modeling purposes. The proposed hybrid is based on the multilayer perceptron (MLP) network. In addition to the usual hidden layers, the first hidden layer is selected to be an adaptive reference pattern layer. Each unit in this new layer incorporates a reference pattern that is located somewhere in the space spanned by the input variables. The outputs of these units are the component wise-squared differences between the elements of a reference pattern and the inputs. The reference pattern layer has some resemblance to the hidden layer of the radial basis function (RBF) networks. Therefore the proposed design can be regarded as a sort of hybrid of MLP and RBF networks. The presented benchmark experiments show that the proposed hybrid can provide significant advantages over standard MLPs and RBFs in terms of fast and efficient learning, and compact network structure.

Weight initialization in the cascade-correlation learning is considered. Most of the previous studies use the so called candidate training to deal with the initialization problem in the cascade-correlation learning. There several candidate hidden units are first trained, and then the one yielding the best value for the covariance criterion is installed to the network. In case there are many candidate units to be trained, the total computational cost of the training can become very large. Here we consider a new approach for weight initialization in the cascade-correlation learning. The proposed method is based on the concept of stepwise regression. Empirical simulations show that the new method can significantly speed-up the cascade-correlation learning compared to the case where the candidate training is used. Moreover, the overall performance remained similar or was even better than with the candidate training.

Although neural networks have already found many successful applications in speech signal processing, it is difficult to compare their performance with that of the traditional methods objectively. We address this problem by using the predictive minimum description length (PMDL) principle to select the optimal network size in modeling speech signals.

Nonlinear time series modeling with a multilayer perceptron network is presented. An important aspect of this modeling is the model selection, i.e., the problem of determining the size as well as the complexity of the model. To overcome this problem we apply the predictive minimum description length (PMDL) principle as a minimization criterion. In the neural network scheme it means minimizing the number of input and hidden units. Three time series modeling experiments are used to examine the usefulness of the PMDL model selection scheme. A comparison with the widely used cross-validation technique is also presented. In our experiments the PMDL scheme and the cross-validation scheme yield similar results in terms of model complexity. However, the PMDL method was found to be two times faster to compute. This is significant improvement since model selection in general is very time consuming.

Usually the training of a multilayer perceptron network starts by initializing the network weights with small random values, and then the weight adjustment is carried out by using an iterative gradient descent-based optimization routine called backpropagation training. If the random initial weights happen to be far from a good solution or they are near a poor local optimum, the training will take a lot of time since many iteration steps are required. Furthermore, it is very possible that the network will not converge to an adequate solution at all. On the other hand, if the initial weights are close to a good solution the training will be much faster and the possibility of obtaining adequate convergence increases. In this paper a new method for initializing the weights is presented. The method is based on the orthogonal least squares algorithm. The simulation results obtained with the proposed initialization method show a considerable improvement in training compared to the randomly initialized networks. In light of practical experiments, the proposed method has proven to be fast and useful for initializing the network weights.

An optimization tool for neural network architecture selection is presented. The main aim of the optimization tool is to reduce the size and complexity of the network and use the least number of weights and nodes for modeling and predictions on nonlinear time series. The problem of selecting the number of input and hidden nodes for modeling nodes is studied by the predictive minimum description length (MDL) principle. The authors discuss comparatively the performance of neural networks and conventional methods in predicting nonlinear time series. The neural network is found to yield better predictions than an optimum ARMA (autoregressive moving average) model.