Abstract Class imbalance occurs when the class distribution is not equal. Namely, one class is under-represented (minority class), and the other class has significantly more samples in the data (majority class). The class imbalance problem is prevalent in many real world applications. Generally, the under-represented minority class is the class of interest. The synthetic minority over-sampling technique (SMOTE) method is considered the most prominent method for handling unbalanced data. The SMOTE method generates new synthetic data patterns by performing linear interpolation between minority class samples and their K nearest neighbors. However, the SMOTE generated patterns do not necessarily conform to the original minority class distribution. This paper develops a novel theoretical analysis of the SMOTE method by deriving the probability distribution of the SMOTE generated samples. To the best of our knowledge, this is the first work deriving a mathematical formulation for the SMOTE patterns’ probability distribution. This allows us to compare the density of the generated samples with the true underlying class-conditional density, in order to assess how representative the generated samples are. The derived formula is verified by computing it on a number of densities versus densities computed and estimated empirically.

The usefulness of the oversampling approach to class-imbalanced structured medical datasets is discussed in this paper. In this regard, we basically look into the oversampling approach's prevailing assumption that synthesized instances do belong to the minority class. We used an off-the-shelf oversampling validation system to test this assumption. According to the experimental results from the validation system, at least one of the three medical datasets used had newly generated samples that were not belonging to the minority class as a result of the oversampling methods validated. Additionally, the error rate varied based on the dataset and oversampling method tested. Therefore, we claim that synthesizing new instances without first confirming that they are aligned with the minority class is a risky approach, especially in medical fields where misdiagnosis can have serious repercussions. As alternatives to oversampling, ensemble, data partitioning, and method-level approaches are advised since they do not make false assumptions.

The price demand relation is a fundamental concept that models how price affects the sale of a product. It is critical to have an accurate estimate of its parameters, as it will impact the company's revenue. The learning has to be performed very efficiently using a small window of a few test points, because of the rapid changes in price demand parameters due to seasonality and fluctuations. However, there are conflicting goals when seeking the two objectives of revenue maximization and demand learning, known as the learn/earn trade-off. This is akin to the exploration/exploitation trade-off that we encounter in machine learning and optimization algorithms. In this paper, we consider the problem of price demand function estimation, taking into account its exploration-exploitation characteristic. We design a new objective function that combines both aspects. This objective function is essentially the revenue minus a term that measures the error in parameter estimates. Recursive algorithms that optimize this objective function are derived. The proposed method outperforms other existing approaches.

Multistep-ahead forecasts can either be produced recursively by iterating a one-step-ahead time series model or directly by estimating a separate model for each forecast horizon. In addition, there are other strategies; some of them combine aspects of both aforementioned concepts. In this paper, we present a comprehensive investigation into the bias and variance behavior of multistep-ahead forecasting strategies. We provide a detailed review of the different multistep-ahead strategies. Subsequently, we perform a theoretical study that derives the bias and variance for a number of forecasting strategies. Finally, we conduct a Monte Carlo experimental study that compares and evaluates the bias and variance performance of the different strategies. From the theoretical and the simulation studies, we analyze the effect of different factors, such as the forecast horizon and the time series length, on the bias and variance components, and on the different multistep-ahead strategies. Several lessons are learned, and recommendations are given concerning the advantages, disadvantages, and best conditions of use of each strategy.

This paper introduces ASTD, an Arabic social sentiment analysis dataset gathered from Twitter.It consists of about 10,000 tweets which are classified as objective, subjective positive, subjective negative, and subjective mixed.We present the properties and the statistics of the dataset, and run experiments using standard partitioning of the dataset.Our experiments provide benchmark results for 4 way sentiment classification on the dataset.

Prediction intervals (PIs) have been proposed in the literature to provide more information by quantifying the level of uncertainty associated to the point forecasts. Traditional methods for construction of neural network (NN) based PIs suffer from restrictive assumptions about data distribution and massive computational loads. In this paper, we propose a new, fast, yet reliable method for the construction of PIs for NN predictions. The proposed lower upper bound estimation (LUBE) method constructs an NN with two outputs for estimating the prediction interval bounds. NN training is achieved through the minimization of a proposed PI-based objective function, which covers both interval width and coverage probability. The method does not require any information about the upper and lower bounds of PIs for training the NN. The simulated annealing method is applied for minimization of the cost function and adjustment of NN parameters. The demonstrated results for 10 benchmark regression case studies clearly show the LUBE method to be capable of generating high-quality PIs in a short time. Also, the quantitative comparison with three traditional techniques for prediction interval construction reveals that the LUBE method is simpler, faster, and more reliable.

This paper evaluates the four leading techniques proposed in the literature for construction of prediction intervals (PIs) for neural network point forecasts. The delta, Bayesian, bootstrap, and mean-variance estimation (MVE) methods are reviewed and their performance for generating high-quality PIs is compared. PI-based measures are proposed and applied for the objective and quantitative assessment of each method's performance. A selection of 12 synthetic and real-world case studies is used to examine each method's performance for PI construction. The comparison is performed on the basis of the quality of generated PIs, the repeatability of the results, the computational requirements and the PIs variability with regard to the data uncertainty. The obtained results in this paper indicate that: 1) the delta and Bayesian methods are the best in terms of quality and repeatability, and 2) the MVE and bootstrap methods are the best in terms of low computational load and the width variability of PIs. This paper also introduces the concept of combinations of PIs, and proposes a new method for generating combined PIs using the traditional PIs. Genetic algorithm is applied for adjusting the combiner parameters through minimization of a PI-based cost function subject to two sets of restrictions. It is shown that the quality of PIs produced by the combiners is dramatically better than the quality of PIs obtained from each individual method.

A correct video segmentation, namely the detection of moving objects within a scene plays a very important role in many application in safety, surveillance, traffic monitoring and object detection. The main objective of this paper is to implement an effective background segmentation algorithm for corner sets extracted from video sequences. A dynamic prototype of the structure of background corners is produced and incoming corners are classified using a fuzzy ARTMAP neural network and labeled as pertaining to the background or foreground using a spatial clustering method. Finally the accuracy of the proposed algorithm is evaluated using PETS2006 benchmark data.

The nearest neighbor method is one of the most widely used pattern classification methods. However its major drawback in practice is the curse of dimensionality. In this paper, we propose a new method to alleviate this problem significantly. In this method, we attempt to cover the training patterns of each class with a number of hyperspheres. The method attempts to design hyperspheres as compact as possible, and we pose this as a quadratic optimization problem. We performed several simulation experiments, and found that the proposed approach results in considerable speed-up over the k-nearest-neighbor method while maintaining the same level of accuray. It also significantly beats other prototype classification methods (Like LVQ, RCE and CCCD) in most performance aspects.

The K-nearest neighbor (KNN) rule is one of the most widely used pattern classification algorithms. For large data sets, the computational demands for classifying patterns using KNN can be prohibitive. A way to alleviate this problem is through the condensing approach. This means we remove patterns that are more of a computational burden but do not contribute to better classification accuracy. In this brief, we propose a new condensing algorithm. The proposed idea is based on defining the so-called chain. This is a sequence of nearest neighbors from alternating classes. We make the point that patterns further down the chain are close to the classification boundary and based on that we set a cutoff for the patterns we keep in the training set. Experiments show that the proposed approach effectively reduces the number of prototypes while maintaining the same level of classification accuracy as the traditional KNN. Moreover, it is a simple and a fast condensing algorithm.

Gaussian processes is a very promising novel technology that has been applied for both the regression problem and the classification problem. While for the regression problem it yields simple exact solutions, this is not the case for the classification case. The reason is that we encounter intractable integrals. In this paper we propose a new approximate solution for the Gaussian process classification problem. The approximating formula is based on certain transformations of the variables and manipulations that lead to orthant multivariate Gaussian integrals. An approximation is then applied that leads to a very simple formula. In spite of its simplicity, the formula gives better results in terms of classification accuracy and speed compared to some of the well-known competing methods.

Penalized likelihood regression is a concept whereby the log-likelihood of the observations is combined with a term measuring the smoothness of the fit, and the resulting expression is then optimized. This concept vies for achieving a compromise between goodness of fit (as typified by the likelihood function) and smoothness of the data. Penalized likelihood regression, which has been developed in the statistics literature since the seventies, has focused mostly on the one-dimensional case. Attempts to consider the general multidimensional case have been limited. In this paper we propose a new multidimensional penalized likelihood regression method. The approach is based on proposing a roughness term based on the discrepancy between the function values among the K-nearest-neighbors. The proposed formulation yields a simple solution in terms of a system of linear equations. We also derive an iterative solution to the problem that sheds light on its basic functionality. The iteration consists of repeatedly taking the weighted average of the target output value and the estimated function values of the K-nearest-neighbors. We show that the proposed model is fairly versatile in that it exhibits nice features in handling user-defined function constraints and data imperfections. Experimental results confirm that it is competitive with the Gaussian process regression method (one of the best methods out there), and exhibits significant speed advantage.

Structured diagrams are very prevalent in many document types. Most people who need to create such diagrams use structured graphics editors such as Microsoft Visio. Structured graphics editors are extremely powerful and expressive but they can be cumbersome to use. We have shown through extensive timing experiments that structured diagrams drawn by hand will take only about 10% of the time it takes to draw one using a tool like Visio. This indicates the value of automated recognition of hand-written diagrams. Recently, applications have been developed that use online systems running on pen-input PCs that allow users to create structured diagrams by drawing the diagram on the PC tablet. The progress of offline diagram recognition is still minimal. The objective of this paper is to propose a context-independent off-line diagram recognition system. Our approach utilizes support vector machines for recognition and line primitive extraction by interpretation of line continuation for segmentation.

The majority of work on forecasting the stock market has focused on individual stocks or stock indexes. In this study we consider the problem of forecasting stock sectors (or industries). We have found no study that considers this problem. Stock sectors are indexes that group several stocks covering a specific sector in the economy, for example the banking sector, the retail sector, etc. It is important for investment allocation purposes to know where each sector is going. In this study we apply linear models, such as Box-Jenkins methodology and multiple regression, as well as neural networks on the sector forecasting problem. As it turns out neural networks yielded the best forecasting performance.

We consider the problem of missing data, and develop ensemble-network models for handling the missing data. The proposed method is based on utilizing the inherent uncertainty of the missing records in generating diverse training sets for the ensemble's networks. The proposed method is based on generating the missing values using their probability density. We repeat this procedure many time thereby creating several complete data sets. A network is trained for each of these data sets, therefore obtaining an ensemble of networks. Several variants are proposed, including the univariate approach and the multivariate approach, which differ in the way missing values are generated. Simulation results confirm the general superiority of the proposed methods compared to the conventional approaches.

The quality of multimedia communicated through the Internet is highly sensitive to packet loss. In this letter, we develop a time-series prediction model for the end-to-end packet loss rate (PLR). The estimate of the PLR is needed in several transmission control mechanisms such as the TCP-friendly congestion control mechanism for UDP traffic. In addition, it is needed to estimate the amount of redundancy for the forward error correction (FEC) mechanism. An accurate prediction would therefore be very valuable. We used a relatively novel prediction model called sparse basis prediction model. It is an adaptive nonlinear prediction approach, whereby a very large dictionary of possible inputs are extracted from the time series (for example, through moving averages, some nonlinear transformations, etc.). Only few of the very best inputs among the dictionary are selected and are combined linearly. An algorithm adaptively updates the input selection (as well as updates the weights) each time a new time sample arrives in a computationally efficient way. Simulation experiments indicate significantly better prediction performance for the sparse basis approach, as compared to other traditional nonlinear approaches.

In many pattern classification problems, an estimate of the posterior probabilities (rather than only a classification) is required. This is usually the case when some confidence measure in the classification is needed. In this article, we propose a new posterior probability estimator. The proposed estimator considers the K-nearest neighbors. It attaches a weight to each neighbor that contributes in an additive fashion to the posterior probability estimate. The weights corresponding to the K-nearest-neighbors (which add to 1) are estimated from the data using a maximum likelihood approach. Simulation studies confirm the effectiveness of the proposed estimator.

Real-time prediction of video source traffic is an important step in many network management tasks such as dynamic bandwidth allocation and end-to-end quality-of-service (QoS) control strategies. In this paper, an adaptive prediction model for MPEG-coded traffic is developed. A novel technology is used, first developed in the signal processing community, called sparse basis selection. It is based on selecting a small subset of inputs (basis) from among a large dictionary of possible inputs. A new sparse basis selection algorithm is developed that is based on efficiently updating the input selection adaptively. When a new measurement is received, the proposed algorithm updates the selected inputs in a recursive manner. Thus, adaptability is not only in the weight adjustment, but also in the dynamic update of the inputs. The algorithm is applied to the problem of single-step-ahead prediction of MPEG-coded video source traffic, and the developed method achieves improved results, as compared to the published results in the literature. The present analysis indicates that the adaptive feature of the developed algorithm seems to add significant overall value.

The MDD is defined as the maximum loss incurred from peak to bottom during a specified period of time. It is often preferred over some of the other risk measures because of the tight relationship between large drawdowns and fund redemptions. Also, a large drawdown can even indicate the start of a deterioration of an otherwise successful trading system, for example due to a market regime switch. Overall, the MDD is a very important risk measure. To be able to use it more insightfully, its analytical properties have to be understood. As a step towards this direction, we have presented in this article some analytic results that we have developed. We hope more and more results will come out from the research community analyzing this important measure.

We present a fast and unbiased Monte Carlo approach to pricing barrier options when the underlying security follows a simple jump diffusion process with constant parameters and a continuously monitored barrier. We develop an algorithm that is based on the Brownian bridge concept and on a sampling approach to successively evaluate an integral that results from the application of the Brownian bridge. This method achieves zero bias. In addition, it is about 100 times faster than the conventional Monte Carlo method for which the discretization interval is chosen so as to achieve acceptable bias.

In this paper we consider two important topics: density estimation and random variate generation. We present a framework that is easily implemented using the familiar multilayer neural network. First, we develop two new methods for density estimation, a stochastic method and a related deterministic method. Both methods are based on approximating the distribution function, the density being obtained by differentiation. In the second part of the paper, we develop new random number generation methods. Our methods do not suffer from some of the restrictions of existing methods in that they can be used to generate numbers from any density provided that certain smoothness conditions are satisfied. One of our methods is based on an observed inverse relationship between the density estimation process and random number generation. We present two variants of this method, a stochastic, and a deterministic version. We propose a second method that is based on a novel control formulation of the problem, where a "controller network" is trained to shape a given density into the desired density. We justify the use of all the methods that we propose by providing theoretical convergence results. In particular, we prove that the L(infinity) convergence to the true density for both the density estimation and random variate generation techniques occurs at a rate O((log log N/N)((1-epsilon)/2)) where N is the number of data points and epsilon can be made arbitrarily small for sufficiently smooth target densities. This bound is very close to the optimally achievable convergence rate under similar smoothness conditions. Also, for comparison, the (2) root mean square (rms) convergence rate of a positive kernel density estimator is O(N(-2/5)) when the optimal kernel width is used. We present numerical simulations to illustrate the performance of the proposed density estimation and random variate generation methods. In addition, we present an extended introduction and bibliography that serves as an overview and reference for the practitioner.

There are several phases that an emerging field goes through before it reaches maturity, and computational finance is no exception. There is usually a trigger for the birth of the field. In our case, new techniques such as neural networks, significant progress in computing technology, and the need for results that rely on more realistic assumptions inspired new researchers to revisit the traditional problems of finance, problems that have often been tackled by introducing simplifying assumptions in the past. The result has been a wealth of new approaches to these time-honored problems, with significant improvements in many cases.

The prediction of corporate bankruptcies is an important and widely studied topic since it can have significant impact on bank lending decisions and profitability. This work presents two contributions. First we review the topic of bankruptcy prediction, with emphasis on neural-network (NN) models. Second, we develop an NN bankruptcy prediction model. Inspired by one of the traditional credit risk models developed by Merton (1974), we propose novel indicators for the NN system. We show that the use of these indicators in addition to traditional financial ratio indicators provides a significant improvement in the (out-of-sample) prediction accuracy (from 81.46% to 85.5% for a three-year-ahead forecast).

Practical algorithms are presented for adaptive state filtering in nonlinear dynamic systems when the state equations are unknown. The state equations are constructively approximated using neural networks. The algorithms presented are based on the two-step prediction-update approach of the Kalman filter. The proposed algorithms make minimal assumptions regarding the underlying nonlinear dynamics and their noise statistics. Non-adaptive and adaptive state filtering algorithms are presented with both off-line and online learning stages. The algorithms are implemented using feedforward and recurrent neural network and comparisons are presented. Furthermore, extended Kalman filters (EKFs) are developed and compared to the filter algorithms proposed. For one of the case studies, the EKF converges but results in higher state estimation errors that the equivalent neural filters. For another, more complex case study with unknown system dynamics and noise statistics, the developed EKFs do not converge. The off-line trained neural state filters converge quite rapidly and exhibit acceptable performance. Online training further enhances the estimation accuracy of the developed adaptive filters, effectively decoupling the eventual filter accuracy from the accuracy of the process model.

Consider an algorithm whose time to convergence is unknown (because of some random element in the algorithm, such as a random initial weight choice for neural network training). Consider the following strategy. Run the algorithm for a specific time T. If it has not converged by time T, cut the run short and rerun it from the start (repeat the same strategy for every run). This so-called restart mechanism has been proposed by Fahlman (1988) in the context of backpropagation training. It is advantageous in problems that are prone to local minima or when there is a large variability in convergence time from run to run, and may lead to a speed-up in such cases. In this article, we analyze theoretically the restart mechanism, and obtain conditions on the probability density of the convergence time for which restart will improve the expected convergence time. We also derive the optimal restart time. We apply the derived formulas to several cases, including steepest-descent algorithms.

How to efficiently train recurrent networks remains a challenging and active research topic. Most of the proposed training approaches are based on computational ways to efficiently obtain the gradient of the error function, and can be generally grouped into five major groups. In this study we present a derivation that unifies these approaches. We demonstrate that the approaches are only five different ways of solving a particular matrix equation. The second goal of this paper is develop a new algorithm based on the insights gained from the novel formulation. The new algorithm, which is based on approximating the error gradient, has lower computational complexity in computing the weight update than the competing techniques for most typical problems. In addition, it reaches the error minimum in a much smaller number of iterations. A desirable characteristic of recurrent network training algorithms is to be able to update the weights in an on-line fashion. We have also developed an on-line version of the proposed algorithm, that is based on updating the error gradient approximation in a recursive manner.

In this paper, we present a 3-D wavelet compression technique for image sequences in video conferencing applications. One of the main requirements for such applications is that the delay has to be within some acceptable limit. When applying wavelet decomposition in the temporal direction, we must store a large number of frames so that such a decomposition can be effective. But this translates into a correspondingly large algorithmic delay. A technique to avoid this obstacle will be proposed. The basic idea is to overlap the decomposition with previous frames that have already been transmitted and selectively transmitting only the new wavelet coefficients.

The estimation of states from input and output measurements using linear state-space models has been widely studied. In particular, the Kalman filtering algorithm has found many applications, such as time-series forecasting, control, parameter estimation, and fault diagnosis. In this paper we propose a new method for adaptive state estimation using feedforward and recurrent neural networks and, in particular, for state filtering that is applicable to general nonlinear systems. The developed method has been applied to the problem of estimating the parameters of an electromechanical system consisting of a DC motor and a centrifugal pump with the associated pumping system. Finally, the proposed algorithm is used in estimating the states and a critical parameter of a complex process system, namely a heat exchanger.

In numerous problems, such as in process control utilizing predictive control algorithms, it is required that a variable of interest be predicted multiple time-steps ahead into the future without having measurements of that variable in the horizon of interest. Additionally, in applications involving forecasting and fault diagnosis the availability of multistep-ahead predictors (MSP) is desired. MSPs are difficult to design because lack of measurements in the prediction horizon necessitates the recursive use of single-step-ahead predictors for reaching the final point in the horizon. Even small prediction errors resulting from noise at each point in the horizon accumulate and propagate, often resulting in poor prediction accuracy. We present a method for designing MSP using dynamic recurrent neural networks. The method is based on a dynamic gradient descent learning algorithm and its effectiveness is demonstrated through applications to an open-loop unstable process system, namely a heat-exchanger.

Three-dimensional (3-D) video compression using wavelets decomposition along the temporal axis dictates that a number of video frames must be buffered to allow for the temporal decomposition. Buffering of frames allows for the temporal correlation to be made use of, and the larger the buffer the more effective the decomposition. One problem inherent in such a set up in interactive applications such as video conferencing, is that buffering translates into a corresponding time delay. We show that 3-D coding of such image sequences can be achieved in the true sense of temporal direction decomposition but with much less buffering requirements. For a practical coder, this can be achieved by introducing an approximation to the way the transform coefficients are encoded. Applying wavelet decomposition using some types of filters may introduce edge errors, which become more prominent in short signal segments. We also present a solution to this problem for the Daubechies (1988) family of filters.

Estimating the flows of rivers can have significant economic impact, as this can help in agricultural water management and in protection from water shortages and possible flood damage. The first goal of this paper is to apply neural networks to the problem of forecasting the flow of the River Nile in Egypt. The second goal of the paper is to utilize the time series as a benchmark to compare between several neural-network forecasting methods.We compare between four different methods to preprocess the inputs and outputs, including a novel method proposed here based on the discrete Fourier series. We also compare between three different methods for the multistep ahead forecast problem: the direct method, the recursive method, and the recursive method trained using a backpropagation through time scheme. We also include a theoretical comparison between these three methods. The final comparison is between different methods to perform longer horizon forecast, and that includes ways to partition the problem into the several subproblems of forecasting K steps ahead.

This paper presents a new model for securing an intranet, connected to the Internet, based on a hybrid technique. The model integrates two security modules; the prevention module and the detection module. The proposed model provides a dynamic binding between the two modules. A comparison between the proposed model and the classical security techniques proved the effectiveness of the new model.

This paper presents a new method for solving the problem of designing dynamic interconnection network using Genetic algorithms. Three parameters are considered: network latency, hardware complexity and degree of reliability. Two types of faults are simulated, the link fault and the switch fault. A GA based package was developed to study the effect of the various parameters affecting the design problem. Results proved the effectiveness of the proposed technique.

Generalization is one of the most important problems in neural-network research. It is influenced by several factors in the network design, such as network size, weight decay factor, and others. We show here that the initial weight distribution (for gradient decent training algorithms) is one other factor that influences generalization. The initial conditions guide the training algorithm to search particular places of the weight space. For instance small initial weights tend to result in low complexity networks, and therefore can effectively act as a regularization factor. We propose a novel network complexity measure, which is helpful in shedding insight into the phenomenon, as well as in studying other aspects of generalization.

A system is developed for the compression of video images, with particular emphasis on video conferencing applications. It combines the well-known quadtree decomposition approach and the discrete cosine transform in an adaptively spatial subsampling and motion compensated scheme. A comparison with the ITU-T H.261 standard for low bit rate video coding reveals that it has better SNR, better perceptual quality (less blocking artifacts), less computational load (about four times less), as well as ease of hardware implementation.

We investigate the effects of delays on the dynamics and, in particular, on the oscillatory properties of simple neural network models. We extend previously known results regarding the effects of delays on stability and convergence properties. We treat in detail the case of ring networks for which we derive simple conditions for oscillating behavior and several formulas to predict the regions of bifurcation, the periods of the limit cycles and the phases of the different neurons. These results in turn can readily be applied to more complex and more biologically motivated architectures, such as layered networks. In general, the main result is that delays tend to increase the period of oscillations and broaden the spectrum of possible frequencies, in a quantifiable way. Simulations show that the theoretically predicted values are in excellent agreement with the numerically observed behavior. Adaptable delays are then proposed as one additional mechanism through which neural systems could tailor their own dynamics. Accordingly, we derive recurrent backpropagation learning formulas for the adjustment of delays and other parameters in networks with delayed interactions and discuss some possible applications.

A nonlinear dynamic model is developed for a process system, namely a heat exchanger, using the recurrent multilayer perceptron network as the underlying model structure. The perceptron is a dynamic neural network, which appears effective in the input-output modeling of complex process systems. Dynamic gradient descent learning is used to train the recurrent multilayer perceptron, resulting in an order of magnitude improvement in convergence speed over a static learning algorithm used to train the same network. In developing the empirical process model the effects of actuator, process, and sensor noise on the training and testing sets are investigated. Learning and prediction both appear very effective, despite the presence of training and testing set noise, respectively. The recurrent multilayer perceptron appears to learn the deterministic part of a stochastic training set, and it predicts approximately a moving average response of various testing sets. Extensive model validation studies with signals that are encountered in the operation of the process system modeled, that is steps and ramps, indicate that the empirical model can substantially generalize operational transients, including accurate prediction of instabilities not in the training set. However, the accuracy of the model beyond these operational transients has not been investigated. Furthermore, online learning is necessary during some transients and for tracking slowly varying process dynamics. Neural networks based empirical models in some cases appear to provide a serious alternative to first principles models.

An accelerated learning algorithm (ABP-adaptive back propagation) is proposed for the supervised training of multilayer perceptron networks. The learning algorithm is inspired from the principle of "forced dynamics" for the total error functional. The algorithm updates the weights in the direction of steepest descent, but with a learning rate a specific function of the error and of the error gradient norm. This specific form of this function is chosen such as to accelerate convergence. Furthermore, ABP introduces no additional "tuning" parameters found in variants of the backpropagation algorithm. Simulation results indicate a superior convergence speed for analog problems only, as compared to other competing methods, as well as reduced sensitivity to algorithm step size parameter variations.

A method for the storage of analog vectors, i.e., vectors whose components are real-valued, is developed for the Hopfield continuous-time network. An important requirement is that each memory vector has to be an asymptotically stable (i.e. attractive) equilibrium of the network. Some of the limitations imposed by the continuous Hopfield model on the set of vectors that can be stored are pointed out. These limitations can be relieved by choosing a network containing visible as well as hidden units. An architecture consisting of several hidden layers and a visible layer, connected in a circular fashion, is considered. It is proved that the two-layer case is guaranteed to store any number of given analog vectors provided their number does not exceed 1 + the number of neurons in the hidden layer. A learning algorithm that correctly adjusts the locations of the equilibria and guarantees their asymptotic stability is developed. Simulation results confirm the effectiveness of the approach.

The problem of finding the optimal single feature for the three-class case, where the covariance matrices associated with the three classes are equal, is considered. It is shown, after performing certain preliminary transformations, that the optimal linear feature can be obtained by searching for the zero of a certain monotone function of one variable in a certain specified narrow interval.