In recent years, bidirectional convolutional recurrent neural networks (RNNs) have made significant breakthroughs in addressing a wide range of challenging problems related to time series and prediction applications. However, the performance of the models is highly dependent on the hyperparameters chosen. Hence, we propose an automatic method for hyperparameter optimization and apply a bidirectional convolutional RNN based on the improved swarm intelligence optimization (sparrow search) to solve regression prediction problems. Specifically, a parallel multiscale dilated convolution (PMDC) module was designed to capture both local and global spatial correlations. This method utilizes convolution with different dilation rates to expand the receptive field without increasing the complexity of the model. Meanwhile, it integrates parallel multiscale structures to extract features at different scales and enhance the model's understanding of the input data. Then, the bidirectional gated recurrent units (BGRUs) learn temporal information from the convolutional features. To address the limitations of empirical hyperparameter selection, such as slow training and low efficiency, a novel PMDC-BGRU model integrated with a pretrained sparrow search algorithm (SSA) was proposed for hyperparameter optimization. Finally, experiments on multiple datasets verified the superiority of the algorithm and explained the flexibility of intelligent optimization algorithms in solving model parameter optimization.

Recurrent neural networks (RNNs) have been found to be a promising field of research for time series prediction, continuous-time system modeling, and discrete-time sequence data processing. The Elman network can be viewed as an RNN with a single layer of feedback connections and local storage units. However, the Elman network has two limitations that restrict its application. One is the overfitting phenomenon, which limits its generalization ability. The other is the complexity of the network structure, which makes its theoretical results incomplete and limits its practicality. The main contribution of this article is to propose a new Elman network model with a regularization term. First, we demonstrate the convergence of the backpropagation algorithm with weight decay regularization under some reasonable conditions. Second, contrary to the usual requirement for bounded weights, we show that such boundedness is no longer a necessary condition for convergence analysis. Third, we show that the requirements for learning rate and the stable set of error functions can also be relaxed. Finally, we provide simulation results to demonstrate the excellent performance of our proposed algorithm.

In this article, we investigate the boundedness and convergence of the online gradient method with the smoothing group regularization for the sigma-pi-sigma neural network (SPSNN). This enhances the sparseness of the network and improves its generalization ability. For the original group regularization, the error function is nonconvex and nonsmooth, which can cause oscillation of the error function. To ameliorate this drawback, we propose a simple and effective smoothing technique, which can effectively eliminate the deficiency of the original group regularization. The group regularization effectively optimizes the network structure from two aspects redundant hidden nodes tending to zero and redundant weights of surviving hidden nodes in the network tending to zero. This article shows the strong and weak convergence results for the proposed method and proves the boundedness of weights. Experiment results clearly demonstrate the capability of the proposed method and the effectiveness of redundancy control. The simulation results are observed to support the theoretical results.

Group regularization is commonly employed in network pruning to achieve structured model compression. However, the rationale behind existing studies on group regularization predominantly hinges on the sparsity capabilities of$L_{p}$regularizers. This singular focus may lead to erroneous interpretations. In response to these limitations, this article proposes a novel framework for evaluating the penalization efficacy of group regularization methods by analyzing the impact of$L_{p}$regularizers on weight magnitudes and weight group magnitudes. Within this framework, we demonstrate that$L_{1,2}$regularization, contrary to prevailing literature, indeed exhibits favorable performance in structured pruning tasks. Motivated by this insight, we introduce a hybrid group regularization approach that integrates$L_{1,2}$regularization and group$L_{1/2}$regularization (denoted as HGL1,2&$L_{1/2}$). This novel method addresses the challenge of selecting appropriate$L_{p}$regularizers for penalizing weight groups by leveraging$L_{1,2}$regularization for penalizing groups with magnitudes exceeding a critical threshold while employing group$L_{1/2}$regularization for other groups. Experimental evaluations are conducted to verify the efficiency of the proposed hybrid group regularization method and the viability of the introduced framework.

This article is concerned with the problem of the number and dynamical properties of equilibria for a class of connected recurrent networks with two switching subnetworks. In this network model, parameters serve as switches that allow two subnetworks to be turned ON or OFF among different dynamic states. The two subnetworks are described by a nonlinear coupled equation with a complicated relation among network parameters. Thus, the number and dynamical properties of equilibria have been very hard to investigate. By using Sturm's theorem, together with the geometrical properties of the network equation, we give a complete analysis of equilibria, including the existence, number, and dynamical properties. Necessary and sufficient conditions for the existence and exact number of equilibria are established. Moreover, the dynamical property of each equilibrium point is discussed without prior assumption of their locations. Finally, simulation examples are given to illustrate the theoretical results in this article.

We propose an embedded/integrated feature selection method based on neural networks with Group Lasso penalty. Group Lasso regularization is considered to produce sparsity on the inputs to the network, i.e., for selection of useful features. Lasso based feature selection using a multi-layer perceptron usually requires an additional set of weights, while our Group Lasso formulation does not require that. However, Group Lasso penalty is non-differentiable at the origin. This may lead to oscillations in numerical simulations and make it difficult to analyze theoretically. To address this issue, four smoothing Group Lasso penalties are introduced. A rigorous proof for the convergence of the proposed algorithm is presented under suitable assumptions. To verify the effectiveness, a three-step algorithmic architecture is adopted in implementation. Experimental results on several datasets validate the theoretical results and demonstrate the competitive performance of the proposed method.

Redundancy in deep neural network (DNN) models has always been one of their most intriguing and important properties. DNNs have been shown to overparameterize, or extract a lot of redundant features. In this work, we explore the impact of size (both width and depth), activation function, and weight initialization on the susceptibility of deep neural network models to extract redundant features. To estimate the number of redundant features in each layer, all the features of a given layer are hierarchically clustered according to their relative cosine distances in feature space and a set threshold. It is shown that both network size and activation function are the two most important components that foster the tendency of DNNs to extract redundant features. The concept is illustrated using deep multilayer perceptron and convolutional neural networks on MNIST digits recognition and CIFAR-10 dataset, respectively.

This paper proposes a new and efficient technique to regularize the neural network in the context of deep learning using correlations among features. Previous studies have shown that oversized deep neural network models tend to produce a lot of redundant features that are either the shifted version of one another or are very similar and show little or no variations, thus resulting in redundant filtering. We propose a way to address this problem and show that such redundancy can be avoided using regularization and adaptive feature dropout mechanism. We show that regularizing both negative and positive correlated features according to their differentiation and based on their relative cosine distances yields network extracting dissimilar features with less overfitting and better generalization. This concept is illustrated with deep multilayer perceptron, convolutional neural network, sparse autoencoder, gated recurrent unit, and long short-term memory on MNIST digits recognition, CIFAR-10, ImageNet, and Stanford Natural Language Inference data sets.

This paper proposes a new approach based on the unknown input method to synthesize the observer for polynomial Takagi–Sugeno fuzzy system with uncertainties. In this paper, the upper bounds of uncertainties are not given and the effect of uncertainties is eliminated without designing an extra controller. With the aids of the noncommon Lyapunov theory and Matlab's tools of the sum-of-square, a new observer is synthesized in which the observer form is completely different from the traditional observer forms reported in previous papers. The conditions for the observer synthesis are much relaxed and the complexity of the design process is reduced. Finally, two illustrative examples are presented to demonstrate the effectiveness of the proposed method.

In this paper, we propose a novel approach based on an unknown input method to synthesize the observer for a discrete time uncertain Takagi-Sugeno fuzzy system. The uncertainty bounds of the system can be unknown and the premise variables can be measurable or unmeasurable. Two main theorems are derived based on the noncommon quadratic Lyapunov function and feasible parameters are solved by the linear matrix inequality technique. Finally, a practical example is given to prove the effectiveness of the proposed method.

Unsupervised feature extractors are known to perform an efficient and discriminative representation of data. Insight into the mappings they perform and human ability to understand them, however, remain very limited. This is especially prominent when multilayer deep learning architectures are used. This paper demonstrates how to remove these bottlenecks within the architecture of non-negativity constrained autoencoder. It is shown that using both L1 and L2 regularizations that induce non-negativity of weights, most of the weights in the network become constrained to be non-negative, thereby resulting into a more understandable structure with minute deterioration in classification accuracy. Also, this proposed approach extracts features that are more sparse and produces additional output layer sparsification. The method is analyzed for accuracy and feature interpretation on the MNIST data, the NORB normalized uniform object data, and the Reuters text categorization data set.

In this paper, we propose four new variants of the backpropagation algorithm to improve the generalization ability for feedforward neural networks. The basic idea of these methods stems from the Group Lasso concept which deals with the variable selection problem at the group level. There are two main drawbacks when the Group Lasso penalty has been directly employed during network training. They are numerical oscillations and theoretical challenges in computing the gradients at the origin. To overcome these obstacles, smoothing functions have then been introduced by approximating the Group Lasso penalty. Numerical experiments for classification and regression problems demonstrate that the proposed algorithms perform better than the other three classical penalization methods, Weight Decay, Weight Elimination, and Approximate Smoother, on both generalization and pruning efficiency. In addition, detailed simulations based on a specific data set have been performed to compare with some other common pruning strategies, which verify the advantages of the proposed algorithm. The pruning abilities of the proposed strategy have been investigated in detail for a relatively large data set, MNIST, in terms of various smoothing approximation cases.

In this paper, we present an evolutionary transfer reinforcement learning framework (eTL) for developing intelligent agents capable of adapting to the dynamic environment of multiagent systems (MASs). Specifically, we take inspiration from Darwin's theory of natural selection and Universal Darwinism as the principal driving forces that govern the evolutionary knowledge transfer process. The essential backbone of our proposed eTL comprises several meme-inspired evolutionary mechanisms, namely meme representation, meme expression, meme assimilation, meme internal evolution, and meme external evolution. Our proposed approach constructs social selection mechanisms that are modeled after the principles of human learning to identify appropriate interacting partners. eTL also models the intrinsic parallelism of natural evolution and errors that are introduced due to the physiological limits of the agents' ability to perceive differences, so as to generate “growth” and “variation” of knowledge that agents have of the world, thus exhibiting higher adaptivity capabilities on solving complex problems. To verify the efficacy of the proposed paradigm, comprehensive investigations of the proposed eTL against existing state-of-the-art TL methods in MAS, are conducted on the “minefield navigation tasks” platform and the “Unreal Tournament 2004” first person shooter computer game, in which homogeneous and heterogeneous learning machines are considered.

In timeseries classification, shapelets are subsequences of timeseries with high discriminative power. Existing methods perform a combinatorial search for shapelet discovery. Even with speedup heuristics such as pruning, clustering, and dimensionality reduction, the search remains computationally expensive. In this paper, we take an entirely different approach and reformulate the shapelet discovery task as a numerical optimization problem. In particular, the shapelet positions are learned by combining the generalized eigenvector method and fused lasso regularizer to encourage a sparse and blocky solution. Extensive experimental results show that the proposed method is orders of magnitudes faster than the state-of-the-art shapelet-based methods, while achieving comparable or even better classification accuracy.

In this paper, we demonstrate new techniques for data representation in the context of deep learning using agglomerative clustering. The results from previous work show that a good number of encoding and decoding filters of layered autoencoders are duplicative thereby enforcing two or more processing filters to extract the same features due to filtering redundancy. We propose a new way to circumvent this problem and our results show that such redundancy is eliminated, yields smaller networks and filters are able to extract distinct features. The concept is illustrated with Sparse Autoenconders (SAE) using MNIST and NORB datasets.

In this paper, we propose a novel framework for the automated extraction of the brain from T1-weighted MR images. The proposed approach is primarily based on the integration of a stochastic model [a two-level Markov-Gibbs random field (MGRF)] that serves to learn the visual appearance of the brain texture, and a geometric model (the brain isosurfaces) that preserves the brain geometry during the extraction process. The proposed framework consists of three main steps: 1) Following bias correction of the brain, a new three-dimensional (3-D) MGRF having a 26-pairwise interaction model is applied to enhance the homogeneity of MR images and preserve the 3-D edges between different brain tissues. 2) The nonbrain tissue found in the MR images is initially removed using the brain extraction tool (BET), and then the brain is parceled to nested isosurfaces using a fast marching level set method. 3) Finally, a classification step is applied in order to accurately remove the remaining parts of the skull without distorting the brain geometry. The classification of each voxel found on the isosurfaces is made based on the first- and second-order visual appearance features. The first-order visual appearance is estimated using a linear combination of discrete Gaussians (LCDG) to model the intensity distribution of the brain signals. The second-order visual appearance is constructed using an MGRF model with analytically estimated parameters. The fusion of the LCDG and MGRF, along with their analytical estimation, allows the approach to be fast and accurate for use in clinical applications. The proposed approach was tested on in vivo data using 300 infant 3-D MR brain scans, which were qualitatively validated by an MR expert. In addition, it was quantitatively validated using 30 datasets based on three metrics: the Dice coefficient, the 95% modified Hausdorff distance, and absolute brain volume difference. Results showed the capability of the proposed approach, outperforming four widely used BETs: BET, BET2, brain surface extractor, and infant brain extraction and analysis toolbox. Experiments conducted also proved that the proposed framework can be generalized to adult brain extraction as well.

We demonstrate a new deep learning autoencoder network, trained by a nonnegativity constraint algorithm (nonnegativity-constrained autoencoder), that learns features that show part-based representation of data. The learning algorithm is based on constraining negative weights. The performance of the algorithm is assessed based on decomposing data into parts and its prediction performance is tested on three standard image data sets and one text data set. The results indicate that the nonnegativity constraint forces the autoencoder to learn features that amount to a part-based representation of data, while improving sparsity and reconstruction quality in comparison with the traditional sparse autoencoder and nonnegative matrix factorization. It is also shown that this newly acquired representation improves the prediction performance of a deep neural network.

Accurate segmentation of pathological lungs from large-size chest computed tomographic images is crucial for computer-assisted lung cancer diagnostics. In this paper, a new framework for automatic pathological lung segmentation is proposed. The proposed INMF-based segmentation approach has the ability to handle the in-homogeneities caused by the arteries, veins, bronchi, and possible pathologies that may exist in the lung tissues, and to detect the number of clusters in the image in an automated manner. The proposed INMF-based segmentation framework is quantitatively validated on simulated realistic lung phantoms that mimic different lung pathologies (7 datasets), in vivo data sets for 17 subjects, and for lung disease with severe pathologies. Three metrics are used: the Dice coefficient, modified Hausdorff distance, and absolute lung volume difference. Results show that the proposed approach outperforms existing lung segmentation techniques and can handle in-homogenities caused by different pathologies.

People can understand complex structures if they relate to more isolated yet understandable concepts. Despite this fact, popular pattern recognition tools, such as decision tree or production rule learners, produce only flat models which do not build intermediate data representations. On the other hand, neural networks typically learn hierarchical but opaque models. We show how constraining neurons' weights to be nonnegative improves the interpretability of a network's operation. We analyze the proposed method on large data sets: the MNIST digit recognition data and the Reuters text categorization data. The patterns learned by traditional and constrained network are contrasted to those learned with principal component analysis and nonnegative matrix factorization.

The use of domain knowledge in learning systems is expected to improve learning efficiency and reduce model complexity. However, due to the incompatibility with knowledge structure of the learning systems and real-time exploratory nature of reinforcement learning (RL), domain knowledge cannot be inserted directly. In this paper, we show how self-organizing neural networks designed for online and incremental adaptation can integrate domain knowledge and RL. Specifically, symbol-based domain knowledge is translated into numeric patterns before inserting into the self-organizing neural networks. To ensure effective use of domain knowledge, we present an analysis of how the inserted knowledge is used by the self-organizing neural networks during RL. To this end, we propose a vigilance adaptation and greedy exploitation strategy to maximize exploitation of the inserted domain knowledge while retaining the plasticity of learning and using new knowledge. Our experimental results based on the pursuit-evasion and minefield navigation problem domains show that such self-organizing neural network can make effective use of domain knowledge to improve learning efficiency and reduce model complexity.

In this paper, a new framework for 3D lung segmentation is proposed. The primary step of this framework is to model both the spatial interaction and first-order visual appearance of the lung tissue based on a new Nonnegative Matrix Factorization (NMF) approach that has the ability to handle the inhomogeneity in the lung regions caused by arteries, veins, bronchi, and possible pathological tissues. The performance of our framework is assessed on fourteen 3D CT images. Based on the Dice Similarity Coefficient (DSC), experimental results showed that the proposed approach outperforms other lung segmentation techniques.

This paper discusses the stability condition for discrete time multi-valued (MVN) recurrent neural networks in asynchronous update mode. In existing research literature, the MVN network in asynchronous update mode has been found convergent if its weight matrix is Hermitian with nonnegative diagonal entries. However, the new theorem and proof presented here show that weight matrix with zero diagonal entries can't guarantee the network stability. Simulation results are used to illustrate the theory.

This paper discusses the stability condition for discrete time recurrent neural networks (RNNs) with complex-valued linear threshold (CLT) neurons. The energy-function method is very useful for complex-valued RNNs study, especially for multi-stable RNNs. In addition to properties of CLT RNNs discussed in earlier work, a new stability condition is offered here by virtue of a lower-bounded energy function. Simulation results are presented to illustrate the theory.

The survey outlines and compares popular computational techniques for quantitative description of shapes of major structural parts of the human brain, including medial axis and skeletal analysis, geodesic distances, Procrustes analysis, deformable models, spherical harmonics, and deformation morphometry, as well as other less widely used techniques. Their advantages, drawbacks, and emerging trends, as well as results of applications, in particular, for computer-aided diagnostics, are discussed.

The problem of robust sparse coding is considered. It is defined as finding linear reconstruction coefficients that minimize the sum of absolute values of the errors, instead of the more typically used sum of squares of the errors. This change lowers the influence of large errors and enhances the robustness of the solution to noise in the data. Sparsity is enforced by limiting the sum of absolute values of the coefficients. We present an algorithm that finds the path traced by the coefficients when the sparsity-inducing constraint is varied. The optimality conditions are derived and included in the algorithm to speed its execution. The proposed method is validated on the problem of robust face recognition.

Occluded face recognition is one the most interesting problems of applied computer vision. Among many face recognition approaches, the Nonnegative Matrix Factorization (NMF) turns out to be one of the popular techniques especially for part-based learning. It aims to factorize a nonnegative data matrix into two nonnegative matrices and obtains a well approximated product using an objective function. In this paper we propose to maximize the correntropy similarity measure as an objective function for NMF. Correntropy has been recently defined as a nonlinear similarity measure using an entropy-based criterion. After the minimization process of the correntropy function, we use it to recognize occluded face data set and compare its recognition performance with the standard NMF and Principal Component Analysis (PCA). The experimental results are illustrated with ORL face data set. The results show that our correntropy-based NMF (NMF-Corr) has better recognition rate compared with PCA and NMF.

Weight-decay method as one of classical complexity regularizations is simple and appears to work well in some applications for multi-layer perceptron network (MPN). This paper shows results for the weak and strong convergence for cyclic and almost cyclic learning MPN with penalty term (weight-decay). The convergence is guaranteed under some relaxed conditions such as the activation functions, learning rate and the assumption for the stationary set of error function. Furthermore, the boundedness of the weights in the training procedure is obtained in a simple and clear way.

Although numerous computational techniques have been applied to predict protein secondary structure (PSS), only limited studies have dealt with discovery of logic rules underlying the prediction itself. Such rules offer interesting links between the prediction model and the underlying biology. In addition, they enhance interpretability of PSS prediction by providing a degree of transparency to the predicting model usually regarded as a black box. In this paper, we explore the generation and use of C4.5 decision trees to extract relevant rules from PSS predictions modeled with two-stage support vector machines (TS-SVM). The proposed rules were derived on the RS126 data set of 126 nonhomologous globular proteins and on the PSIPRED data set of 1,923 protein sequences. Our approach has produced sets of comprehensible, and often interpretable, rules underlying the PSS predictions. Moreover, many of the rules seem to be strongly supported by biological evidence. Further, our approach resulted in good prediction accuracy, few and usually compact rules, and rules that are generally of higher confidence levels than those generated by other rule extraction techniques.

The five papers in this special section focus on white-box nonlinear prediction models.

Rule extraction from neural networks (NNs) solves two fundamental problems: it gives insight into the logic behind the network and in many cases, it improves the network's ability to generalize the acquired knowledge. This paper presents a novel eclectic approach to rule extraction from NNs, named LOcal Rule Extraction (LORE), suited for multilayer perceptron networks with discrete (logical or categorical) inputs. The extracted rules mimic network behavior on the training set and relax this condition on the remaining input space. First, a multilayer perceptron network is trained under standard regime. It is then transformed into an equivalent form, returning the same numerical result as the original network, yet being able to produce rules generalizing the network output for cases similar to a given input. The partial rules extracted for every training set sample are then merged to form a decision diagram (DD) from which logic rules can be extracted. A rule format explicitly separating subsets of inputs for which an answer is known from those with an undetermined answer is presented. A special data structure, the decision diagram, allowing efficient partial rule merging is introduced. With regard to rules' complexity and generalization abilities, LORE gives results comparable to those reported previously. An algorithm transforming DDs into interpretable boolean expressions is described. Experimental running times of rule extraction are proportional to the network's training time.

Independent component analysis with reference (ICA-R), is a technique to incorporate prior information about the desired sources as reference signals into the contrast function of ICA so as to form an augmented Lagrangian function under the framework of constrained ICA (cICA). The ICA-R algorithm is constructed by solving the optimization problem via Newton-like learning style. Unfortunately, this algorithm does not find a global optimum once it reaches a local optimum resulting in misconvergence that hinders the capability of ICA-R. To overcome the optimization problems with the previous methods, this paper uses an evolutionary approach to ICA-R that brings the search out of local minima and finds a global optimal solution. Experiments with synthetic signals demonstrate the validity of the proposed method.

This letter discusses the competitive layer model (CLM) for a class of discrete-time recurrent neural networks with linear threshold (LT) neurons. It first addresses the boundedness, global attractivity, and complete stability of the networks. Two theorems are then presented for the networks to have CLM property. We also present the analysis for network dynamics, which performs a column winner-take-all behavior and grouping selection among different layers. Furthermore, we propose a novel synchronous CLM iteration method, which has similar performance and storage allocation but faster convergence compared with the previous asynchronous CLM iteration method (Wersing, Steil, & Ritter, 2001 ). Examples and simulation results are used to illustrate the developed theory, the comparison between two CLM iteration methods, and the application in image segmentation.

Multiclass cancer classification on microarray data has provided the feasibility of cancer diagnosis across all of the common malignancies in parallel. Using multiclass cancer feature selection approaches, it is now possible to identify genes relevant to a set of cancer types. However, besides identifying the relevant genes for the set of all cancer types, it is deemed to be more informative to biologists if the relevance of each gene to specific cancer or subset of cancer types could be revealed or pinpointed. In this paper, we introduce two new definitions of multiclass relevancy features, i.e., full class relevant (FCR) and partial class relevant (PCR) features. Particularly, FCR denotes genes that serve as candidate biomarkers for discriminating all cancer types. PCR, on the other hand, are genes that distinguish subsets of cancer types. Subsequently, a Markov blanket embedded memetic algorithm is proposed for the simultaneous identification of both FCR and PCR genes. Results obtained on commonly used synthetic and real-world microarray data sets show that the proposed approach converges to valid FCR and PCR genes that would assist biologists in their research work. The identification of both FCR and PCR genes is found to generate improvement in classification accuracy on many microarray data sets. Further comparison study to existing state-of-the-art feature selection algorithms also reveals the effectiveness and efficiency of the proposed approach.

This paper discusses a class of discrete time recurrent neural networks with multivalued neurons (MVN) with complex-valued weights and an activation function defined as a function of the argument of a weighted sum. Complementing state-of-the-art of such networks, this paper focuses on the convergence analysis of such networks in synchronous update mode. One theorem is presented and simulation results are used to illustrate the theory.

A filter method of feature selection based on mutual information, called normalized mutual information feature selection (NMIFS), is presented. NMIFS is an enhancement over Battiti's MIFS, MIFS-U, and mRMR methods. The average normalized mutual information is proposed as a measure of redundancy among features. NMIFS outperformed MIFS, MIFS-U, and mRMR on several artificial and benchmark data sets without requiring a user-defined parameter. In addition, NMIFS is combined with a genetic algorithm to form a hybrid filter/wrapper method called GAMIFS. This includes an initialization procedure and a mutation operator based on NMIFS to speed up the convergence of the genetic algorithm. GAMIFS overcomes the limitations of incremental search algorithms that are unable to find dependencies between groups of features.

Information on secondary structures of amino acid residues in proteins provides valuable clues for the prediction of their 3-D structure and function. Although numerous computational techniques have been applied to predict protein secondary structure (PSS), only limited studies have dealt with discovery of logic rules underlying the prediction itself. Such rules offer interesting links between the prediction model and the underlying biology. In addition, they enhance interpretability of PSS prediction by providing a degree of transparency to the predicting model usually regarded as a black-box. In this paper, we explore the generation and use of C 4.5 decision trees to extract relevant rules from PSS predictions modeled with two-stage support vector machines (TS-SVM). Our approach has produced sizable sets of comprehensible, and often interpretable, rules underlying the PSS predictions. Moreover, many of the rules seem to be strongly supported by biological evidence. Further, our approach resulted in good prediction accuracy, few and usually compact rules, and rules that are generally of higher confidence levels than those generated by other rule extraction techniques. The proposed rules were derived and tested on the RS126 dataset of 126 nonhomologous globular proteins.

This paper proposes a hierarchical architecture, HieNet, that utilizes the K-Iterations Fast Learning artificial Neural Network (KFLANN). Effective in its clustering capabilities, the KFLANN is capable of providing more stable and consistent clusters that are independent data presentation sequences (DPS). Leveraging on the ability to provide more consistent clusters, the KFLANN is initially used to identify the homogeneous Feature Spaces that prepare large dimensional networks for a hierarchical organization. We illustrate how this hierarchical structure can be constructed through the recurring use of the KFLANN and support our work with experimental results.

This paper discusses the framework for virtual collaborative environment for researchers, practitioners, users and learners in the areas of computational intelligence and machine learning (CIML) that is currently developed by our group. It also outlines main features of the community portal under construction that will support communication and sharing of computational resources. In particular, selected aspects of structure of the portal such as common formats of data, models, software, publications and software documentation are discussed.

A multilayer neural network based on multivalued neurons (MLMVN) is a neural network with a traditional feedforward architecture. At the same time, this network has a number of specific different features. Its backpropagation learning algorithm is derivative-free. The functionality of MLMVN is superior to that of the traditional feedforward neural networks and of a variety kernel-based networks. Its higher flexibility and faster adaptation to the target mapping enables to model complex problems using simpler networks. In this paper, the MLMVN is used to identify both type and parameters of the point spread function, whose precise identification is of crucial importance for the image deblurring. The simulation results show the high efficiency of the proposed approach. It is confirmed that the MLMVN is a powerful tool for solving classification problems, especially multiclass ones.

A knowledge-based computer assisted decision (KB-CAD) system is a case-based reasoning system previously proposed for breast cancer detection. Although it was demonstrated to be very effective for the diagnostic problem, it was also shown to be computationally expensive due to the use of mutual information between images as a similarity measure. Here, the authors propose to alleviate this drawback by reducing the case-base size. The problem is formalized and a genetic algorithm is utilized as an optimization tool. Appropriate for the problem representation and operators are presented and discussed. A clinically relevant index of the area under the receiver operator characteristic curve is used as a measure of the system performance during the optimization and testing stages. Experimental results show that application of the proposed method can significantly reduce the case-base size while the classification performance of the KB-CAD, in fact, increases.

In decentralized control of multi-agent systems each agent is making a decision regarding its action autonomously, based on its own observations. In the light of the formal models of decentralized environments presented in the last decade, finding an optimal solution to a decentralized control problem is computationally prohibitive, even for moderately complicated environments. The problem, however, is of great significance since many of the real world systems can be treated as multi-agent systems with decentralized control. In this article, the authors propose an approximate algorithm for the problem based on a genetic algorithm. First, the problem is formalized using decentralized partially observable Markov decision processes. Then a way of representing a solution (joint policy) in a chromosome is introduced and a genetic algorithm is proposed as a search mechanism. Finally, a multi-agent tiger problem is used as an experimental framework to show the effectiveness of the algorithm.

In this study, a Web based maze robot system has been designed and implemented for solving different maze algorithms with the help of machine learning approaches. The robot system has a map-based heuristic maze solving algorithm. The algorithm used for solving the maze is based on map creation and produces a control signal for robot direction. Robot motions were controlled by a fuzzy motion control system running on a chip. The control algorithm can be easily changed with the help of an algorithm via web interface controlled by the control center. The control center program powered by MATLAB functions and special libraries (image and control) in DELPHI manage all robotic activities. These activities are: command interpreter, image capturing, processing and serving, machine learning techniques, Web serving, database management, communication with robot, and compiling microcontroller programs. The results have shown that the proposed, designed and implemented system provides amazing new features to the applicants doing their real-time programming exercises on Web.

In this paper an extension of the merge neural gas (MNG) model for processing data sequences is presented. In the proposed Window MNG network, time windows are used for processing signal inputs and their contexts. The new strategy allows breaking complex patterns into sequences of simpler patterns. A recently proposed projection method is adapted for projecting the multidimensional vectors that result from the temporal vector quantization onto two-dimensional maps. In addition, a new training method for MNG networks is proposed that allows obtaining enhanced results. The proposed methods are applied to the visualization and clustering of the Mackey-Glass time series and control chart signals.

A neural network ensemble for adaptive prediction of patient response to anemia correction with erythropoietin is presented. The model combines individual neural networks whose outputs are aggregated using prediction error information. The purpose of this approach is to perform response modeling that can be used for individualized, patient-specific drug delivery.

This paper presents an experimental study on the impact of low class prevalence on the neural network based classifier performance as measured using receiver operator characteristic (ROC) analysis. Two methods of dealing with the problem are investigated: oversampling and undersampling in the context of varying the class prevalence and the size of training datasets with uncorrelated and correlated features. The results show that the class imbalance can significantly decrease the classifier performance especially in the case of small training datasets. Furthermore, the oversampling method is shown to be more effective than the undersampling method in compensating the class imbalance. Statistically significant differences, however, are observed only in the cases with large total number of samples and very low prevalence.

This paper outlines an approximate algorithm for finding an optimal decentralized control in multi-agent systems. Decentralized partially observable Markov decision processes and their extension to infinite state, observation and action spaces are utilized as a theoretical framework. In the presented algorithm, policies of each agent are represented by a feedforward neural network. Then, a search is performed in a joint weight space of all networks. Particle swarm optimization is applied as a search algorithm. Experimental results are provided showing that the algorithm finds good solutions for the classical Tiger problem extended to multi-agent systems, as well as for a multi-agent navigation task involving large state and action spaces

Identification of the significance of input variables is very important for complex systems with high-dimensional input space. In this paper, a method using fuzzy average with fuzzy cluster distribution is proposed. To avoid the interference of different distributions of the sampling data, the distribution of fuzzy clusters in the sampling data is considered, instead of the original data set. To discover the input-output relationship, the methods of fuzzy rules and fuzzy C-means are first used to partition the original sampling data set into fuzzy clusters. A new data set with the same distribution of the fuzzy clusters is produced. The fuzzy average method is then applied to the new data set. By doing so, the interference of distribution of the original sampling data is removed. This method is straightforward and computationally easy. The performance is tested on both benchmark data and real-world data.

Single-class minimax probability machines (MPMs) offer robust novelty detection with distribution-free worst case bounds on the probability that a pattern will fall inside the normal region. However, in practice, they are too cautious in labeling patterns as outlying and so have a high false negative rate (FNR). In this paper, we propose a more aggressive version of the single-class MPM that bounds the best case probability that a pattern will fall inside the normal region. These two MPMs can then be used together to delimit the solution space. By using the hyperplane lying in the middle of this pair of MPMs, a better compromise between false positives (FPs) and false negatives (FNs), and between recall and precision can be obtained. Experiments on the real-world data sets show encouraging results.

The hierarchical fast learning artificial neural network (HieFLANN) is a clustering NN that can be initialized using statistical properties of the data set. This provides the possibility of constructing the entire network autonomously with no manual intervention. This distinguishes it from many existing networks that, though hierarchically plausible, still require manual initialization processes. The unique system of hierarchical networks begins with a reduction of the high-dimensional feature space into smaller and manageable ones. This process involves using the K-iterations fast learning artificial neural network (KFLANN) to systematically cluster a square matrix containing the Mahalanobis distances (MDs) between data set features, into homogeneous feature subspaces (HFSs). The KFLANN is used for its heuristic network initialization capabilities on a given data set and requires no supervision. Through the recurring use of the KFLANN and a second stage involving canonical correlation analysis (CCA), the HieFLANN is developed. Experimental results on several standard benchmark data sets indicate that the autonomous determination of the HFS provides a viable avenue for feasible partitioning of feature subspaces. When coupled with the network transformation process, the HieFLANN yields results showing accuracies comparable with available methods. This provides a new platform by which data sets with high-dimensional feature spaces can be systematically resolved and trained autonomously, alleviating the effects of the curse of dimensionality.

Estimation of the dynamic spinal forces from kinematics data is very complicated because it involves the handling of the relationship between kinematic variables and electromyography (EMG) signals, as well as the relationship between EMG signals and the forces. A recurrent fuzzy neural network (RFNN) model is proposed to establish the kinematics-EMG-force relationship and model the dynamics of muscular activities. The EMG signals are used as an intermediate output and are fed back to the input layer. Since EMG is a direct reflection of muscular activities, the feedback of this model has a physical meaning. It expresses the dynamics of muscular activities in a straightforward way and takes advantage from the recurrent property. The trained model can then have the forces predicted directly from kinematic variables while bypassing the costly procedure of measuring EMG signals and avoiding the use of a biomechanics model. A learning algorithm is derived for the RFNN model.

we propose a novel neural network model for the identification of low back injury using electromyography (EMG) data. By connecting task condition variables to the second hidden-layer of the neural network, the importance of those variables can be improved. A 4-muscle method and a 10-muscle method are discussed. A higher classi cation accuracy was achieved by the 10-muscle method since it takes the correlation between muscle activities into account. We also found that identi cation accuracy decreases when the object weight or the lifting height increases. The obtained results improve our understanding of low back disorders and provide important guidance for future experimental studies.

Kernel methods, such as the support vector machine (SVM), are often formulated as quadratic programming (QP) problems. However, given m training patterns, a naive implementation of the QP solver takes O(m3) training time and at least O(m2) space. Hence, scaling up these QPs is a major stumbling block in applying kernel methods on very large data sets, and a replacement of the naive method for finding the QP solutions is highly desirable. Recently, by using approximation algorithms for the minimum enclosing ball (MEB) problem, we proposed the core vector machine (CVM) algorithm that is much faster and can handle much larger data sets than existing SVM implementations. However, the CVM can only be used with certain kernel functions and kernel methods. For example, the very popular support vector regression (SVR) cannot be used with the CVM. In this paper, we introduce the center-constrained MEB problem and subsequently extend the CVM algorithm. The generalized CVM algorithm can now be used with any linear/nonlinear kernel and can also be applied to kernel methods such as SVR and the ranking SVM. Moreover, like the original CVM, its asymptotic time complexity is again linear in m and its space complexity is independent of m. Experiments show that the generalized CVM has comparable performance with state-of-the-art SVM and SVR implementations, but is faster and produces fewer support vectors on very large data sets.

Individualization of drug delivery in treatment of chronic ailments is a challenge to the physician. Variability of response across patient population requires tailoring the dosing strategies to individual's needs. We have previously demonstrated the potential of reinforcement learning methods to support the physician in the management of anemia. In this paper, we propose the incorporation of prior knowledge into the learning mechanism to further improve the outcomes of the treatment.

Effective pharmacological therapy in chronic treatments poses many challenges to physicians. Individual response to treatment varies across patient populations. Furthermore, due to the prolonged character of the therapy, the response may change over time. A reinforcement learning-based framework is proposed for treatment individualization in the management of renal anemia. The approach is based on numerical simulation of the patient performed by Takagi-Sugeno fuzzy model and a radial basis function network implementation of an on-policy Q-learning critic. Simulation results demonstrate the potential of the proposed method to yield policies that achieve the therapeutic goal in individuals with different response characteristics.

Identification of influence of input variables is very important for complex nonlinear systems with high dimensional input space. In this paper we propose a method using fuzzy average with fuzzy cluster distribution (FAFCD). To avoid the interference of different distributions of the sampling data, we deal with the distribution of fuzzy clusters in the sampling data, instead of the original data set. To discover the input-output relationship, we first use method of fuzzy rules and fuzzy c-means to partition the original sampling data set into fuzzy clusters. We produce a new data set with the same distribution of the fuzzy clusters. Then the fuzzy average method is applied to the new data set. By doing this, the interference of distribution of the original sampling data is removed. This method is straightforward and computationally easy. The performance is tested on both benchmark data and the electromyographic (EMG) signal Evaluation System.

An energy function-based autoassociative memory design method to store a given set of unipolar binary memory vectors as attractive fixed points of an asynchronous discrete Hopfield network (DHN) is presented. The discrete quadratic energy function whose local minima correspond to the attractive fixed points of the network is constructed via solving a system of linear inequalities derived from the strict local minimality conditions. The weights and the thresholds are then calculated using this energy function. If the inequality system is infeasible, we conclude that no such asynchronous DHN exists, and extend the method to design a discrete piecewise quadratic energy function, which can be minimized by a generalized version of the conventional DHN, also proposed herein. In spite of its computational complexity, computer simulations indicate that the original method performs better than the conventional design methods in the sense that the memory can store, and provide the attractiveness for almost all memory sets whose cardinality is less than or equal to the dimension of its elements. The overall method, together with its extension, guarantees the storage of an arbitrary collection of memory vectors, which are mutually at least two Hamming distances away from each other, in the resulting network.

We propose a modified recurrent neural network model which establishes the relationship between kinematics and the dynamic forces on lumbar joint. By doing that we can have the forces predicted directly from kinematic variables while bypassing the costly procedure of measuring EMG (electromyography) signals and avoiding the use of biomechanics model. In the proposed model, we introduce the EMG signal as an intermediate output and loop it back to the input layer, instead of looping back the ultimate output, the forces. Since the EMG signal is a direct reflection of muscle activity, the most valuable point of this model is that the back-looping of the intermediate output has physical meaning. It solves the problem that the input and output of the system have no direct and explicit physical connection. At the same time, the advantages of recurrent neural network are utilized.

An EMG (electromyography) signal prediction model is built using artificial neural network. Kinematics variables and subject variables are selected as inputs of this model. A novel structure of feedforward neural network is proposed in This work to obtain better accuracy of prediction. By adding regional connections between the input and the output, the new architecture of the neural network can have both global features and regional features extracted from the input. The global connections put more emphasis on the whole picture and determine the global trend of the predicted curve, while the regional connections concentrate on each point and modify the prediction locally. Back-propagation algorithm is used in the modeling. A basic structure of neural network designed for this problem is discussed. Then to overcome its drawbacks, we propose a new structure.

A new radial basis function (RBF) network training procedure that employs a linear projection technique along parameter search is proposed. To be applied simultaneously with the conventional center and/or weight adjustment methods, a gradient descent iteration on the width parameters of RBF units is introduced. The projection mechanism used by the procedure avoids negative width parameters and enables detection of redundant units, which can then be pruned from the network. Proposed training approach is applied to design a feedback neuro-controller for a nonlinear plant to track a desired trajectory.

Presents the welcome message from the conference proceedings.

In many applications, black-box prediction is not satisfactory, and understanding the data is of critical importance. Typically, approaches useful for understanding of data involve logical rules, evaluate similarity to prototypes, or are based on visualization or graphical methods. This paper is focused on the extraction and use of logical rules for data understanding. All aspects of rule generation, optimization, and application are described, including the problem of finding good symbolic descriptors for continuous data, tradeoffs between accuracy and simplicity at the rule-extraction stage, and tradeoffs between rejection and error level at the rule optimization stage. Stability of rule-based description, calculation of probabilities from rules, and other related issues are also discussed. Major approaches to extraction of logical rules based on neural networks, decision trees, machine learning, and statistical methods are introduced. Optimization and application issues for sets of logical rules are described. Applications of such methods to benchmark and real-life problems are reported and illustrated with simple logical rules for many datasets. Challenges and new directions for research are outlined.

Biomedical image registration, or geometric alignment of two-dimensional and/or three-dimensional (3D) image data, is becoming increasingly important in diagnosis, treatment planning, functional studies, computer-guided therapies, and in biomedical research. Registration based on intensity values usually requires optimization of some similarity metric between the images. Local optimization techniques frequently fail because functions of these metrics with respect to transformation parameters are generally nonconvex and irregular and, therefore, global methods are often required. In this paper, a new evolutionary approach, particle swarm optimization, is adapted for single-slice 3D-to-3D biomedical image registration. A new hybrid particle swarm technique is proposed that incorporates initial user guidance. Multimodal registrations with initial orientations far from the ground truth were performed on three volumes from different modalities. Results of optimizing the normalized mutual information similarity metric were compared with various evolutionary strategies. The hybrid particle swarm technique produced more accurate registrations than the evolutionary strategies in many cases, with comparable convergence. These results demonstrate that particle swarm approaches, along with evolutionary techniques and local methods, are useful in image registration, and emphasize the need for hybrid approaches for difficult registration problems.

The olfactory system is a very efficient biological setup capable of odor information processing with neural signals. The nature of neural signals restricts the information representation to multidimensional temporal sequences of spikes. The information is contained in the interspike intervals within each individual neural signal and interspike intervals between multiple signals. A mechanism of interactions between random excitations evoked by odorants in the olfactory receptors of the epithelium and deterministic operation of the olfactory bulb is proposed in this paper. Inverse Frobenius-Perron models of the bulb's temporal sequences are fitted to the interspike distributions of temporally modulated receptor signals. Ultimately, such pattern matching results in ability to recognize odors and offer a hypothetic model for signal processing occurring in the primary stage of the olfactory system.

An artificial neural network (ANN)-based approach to pharmacodynamic population analysis presented. In pursuit of an optimal and cost-effective strategy for drug dosing in patients with renal failure, an ANN is applied to perform drug-dose-effect modeling. Such a model allows for versatile prediction of the response to the drug at the effect site and, subsequently, for adjusting the dosing regimen.

A method to store each element of an integer-valued memory set M /spl sub/ {1, 2, ..., K}/sup n/ as a fixed point into a complex-valued multistate Hopfield network is introduced. The method employs a set of inequalities to render each memory pattern as a strict local minimum of a quadratic energy landscape, and based on the solution of this system, gives a recurrent network of n multistate neurons with complex and symmetric synaptic weight, which operates on the finite state space {1, 2, ..., K}/sup n/ to minimize this quadratic functional. Maximum number of integer-valued vectors that can be embedded into the energy landscape of the network by this method is investigated by computer experiments. The paper also enlightens the performance of the proposed method in reconstructing noisy gray-scale images.

This paper presents a new approach to fuzzy rule-based modeling of nonlinear systems from numerical data. The novelty of the approach lies in the way of input partitioning and in the syntax of the rules. This paper introduces interpretable relational antecedents that incorporate local linear interactions between the input variables into the inference process. This modification improves the approximation quality and allows for limiting the number of rules. Additionally, the resulting linguistic description better captures the system characteristics by exposing the interactions between the input variables.

A method to store each element of an integral memory set M subset {1,2,...,K}/sup n/ as a fixed point into a complex-valued multistate Hopfield network is introduced. The method employs a set of inequalities to render each memory pattern as a strict local minimum of a quadratic energy landscape. Based on the solution of this system, it gives a recurrent network of n multistate neurons with complex and symmetric synaptic weights, which operates on the finite state space {1,2,...,K}/sup n/ to minimize this quadratic functional. Maximum number of integral vectors that can be embedded into the energy landscape of the network by this method is investigated by computer experiments. This paper also enlightens the performance of the proposed method in reconstructing noisy gray-scale images.

An algorithmic approach for efficient identification of a fuzzy relational rule network (FR/sup 2/N) from data is presented. FR/sup 2/N uses a relational input partition for human-understandable modeling of linear interactions between the input variables. Mutual subsethood has been used to estimate the optimal interaction structure. An analytical relationship between the mutual subsethood measure and one of the parameters of the membership functions is derived. The use of this relationship results in a dramatic speed-up of the identification process.

Neural networks (NNs) have been successfully applied to solve a variety of application problems including classification and function approximation. They are especially useful as function approximators because they do not require prior knowledge of the input data distribution and they have been shown to be universal approximators. In many applications, it is desirable to extract knowledge that can explain how Me problems are solved by the networks. Most existing approaches have focused on extracting symbolic rules for classification. Few methods have been devised to extract rules from trained NNs for regression. This article presents an approach for extracting rules from trained NNs for regression. Each rule in the extracted rule set corresponds to a subregion of the input space and a linear function involving the relevant input attributes of the data approximates the network output for all data samples in this subregion. Extensive experimental results on 32 benchmark data sets demonstrate the effectiveness of the proposed approach in generating accurate regression rules.

An approach to data-driven linguistic modeling is presented. The methodology is based on a fuzzy system with relational input partition that allows for transparent modeling of linear dependencies between the inputs. An identification algorithm for this type of fuzzy system is proposed. It automatically finds the strongest dependencies from numerical data. An application example illustrates the usefulness of the proposed approach.

An iterative backward selection method for determination of relevant input variables in data-driven fuzzy modeling is presented. The method utilizes parameters of the Takagi-Sugeno model as a factor to determine the significance of input variables. As a result, it is less computationally intensive than most of the existing methods for input variable selection.

This paper proposes a novel neural-network approach to blind source separation in nonlinear mixture. The approach utilizes a radial basis function (RBF) neural-network to approximate the inverse of the nonlinear mixing mapping which is assumed to exist and able to be approximated using an RBF network. A contrast function which consists of the mutual information and partial moments of the outputs of the separation system, is defined to separate the nonlinear mixture. The minimization of the contrast function results in the independence of the outputs with desirable moments such that the original sources are separated properly. Two learning algorithms for the parametric RBF network are developed by using the stochastic gradient descent method and an unsupervised clustering method. By virtue of the RBF neural network, this proposed approach takes advantage of high learning convergence rate of weights in the hidden layer and output layer, natural unsupervised learning characteristics, modular structure, and universal approximation capability. Simulation results are presented to demonstrate the feasibility, robustness, and computability of the proposed method.

Presents a neural-based approach to classifying and estimating the statistical parameters of speckle noise found in biomedical ultrasound images. Speckle noise, a very complex phenomenon, has been modeled in a variety of different ways: and there is currently no clear consensus as to its precise statistical characteristics. In this study, different neural network architectures are used to classify ultrasound images contaminated with three types of noise, based upon three one-parameter statistical distributions. At the same time: the parameter is estimated. It is expected that accurate characterization of ultrasound speckle noise will benefit existing post-processing methods, and may lead to new refinements in these techniques.

The paper presents a fuzzy neural network whose structure accounts for relations between input variables of the system under consideration. This modification results in a simple fuzzy model with improved approximation accuracy. An example of nonlinear time series prediction using the proposed fuzzy neural network is also included.

A number of neural network models and training procedures for time series prediction have been proposed in the technical literature. These models typically used uni-directional computation flow or its modifications. In this study a novel concept of bi-directional computation style is proposed and applied to prediction tasks. Since the coupling effects between the future prediction system and the past prediction system help the proposed model improve its performance, it is found that the prediction score is better than with the traditional uni-directional method. The bi-directional predicting architecture has been found to perform better than the conventional one when tested with standard benchmark sunspots data.

This paper introduces a novel technique for sequential blind extraction of singularly mixed sources. First, a neural-network model and an adaptive algorithm for single-source blind extraction are introduced. Next, extractability analysis is presented for singular mixing matrix, and two sets of necessary and sufficient extractability conditions are derived. The adaptive algorithm and neural-network model for sequential blind extraction are then presented. The stability of the algorithm is discussed. Simulation results are presented to illustrate the validity of the adaptive algorithm and the stability analysis. The proposed algorithm is suitable for the case of nonsingular mixing matrix as well as for singular mixing matrix.

A multi-stage algorithm for classification and fuzzy rule extraction from data based on a self-organizing network and a two-layer perceptron network is proposed. Self-organizing techniques are applied to find data prototypes, which are subsequently used to initialize the perceptron network and to produce membership functions from the learned mapping. Fuzzy rules created by the algorithm provide linguistic description of the relationship encoded in data.

Using process control approaches, pharmacokinetic models are developed. All models presented are based on the one-compartment model. These neural network-based and adaptive versions of linear, nonlinear, and time-dependent models are tested on data sets collected from hemodialysis patients receiving anti-coagulant heparin during treatment. Results show that increased model complexity ensures improved quality of identification, while it decreases the initial quality of estimation.

The existence of a Hamiltonian vector field in which trajectories of the invariant set /spl Lambda/ of a dissipative hyperbolic chaotic system are embedded, is proved. This evidence with an example concerning the Lorenz system is provided. Also, a constructive method of designing a Hamiltonian for the Lorenz attractor with a universal approximator is introduced. The present approach enables the use of universal approximator property of neural networks for modeling dynamics from the Hamiltonian perspective.

Structural learning with forgetting is an established method of using Laplace regularization to generate skeletal artificial neural networks. In this paper we develop a continuous dynamical system model of regularization in which the associated regularization parameter is generalized to be a time-varying function. Analytic results are obtained for a Laplace regularizer and a quadratic error surface by solving a different linear system in each region of the weight space. This model also enables a comparison of Laplace and Gaussian regularization. Both of these regularizers have a greater effect in weight space directions which are less important for minimization of a quadratic error function. However, for the Gaussian regularizer, the regularization parameter modifies the associated linear system eigenvalues, in contrast to its function as a control input in the Laplace case. This difference provides additional evidence for the superiority of the Laplace over the Gaussian regularizer.

A model of a multivalued associative memory is presented. This memory has the form of a fully connected attractor neural network composed of multistate complex-valued neurons. Such a network is able to perform the task of storing and recalling gray-scale images. It is also shown that the complex-valued fully connected neural network may be considered as a generalization of a Hopfield network containing real-valued neurons. A computational energy function is introduced and evaluated in order to prove network stability for asynchronous dynamics. Storage capacity as related to the number of accessible neuron states is also estimated.

A unique and accurate approach for modeling semiconductor device characteristics and estimating IC parametric yield is described. Multilayer perceptron neural networks (MLPNN) are trained using error back propagation to model DC device characteristics measured at the final fabrication stage. Measurements of material and/or device characteristics taken at earlier fabrication stages are used to develop neural network models of the final DC parameters. A very good agreement has been found between the actual measurements and the MLPNN modeled parameters, and the resulting yield estimations are in excellent agreement with the actual yield.

The model of a multivalued associative memory which has the form of a fully connected attractor neural network composed of multi-state complex-valued neurons is presented. The network is able to perform the task of storing and recalling grey-scale images. Storage capacity estimation is given and convergence of the network asynchronous dynamics is proven.

This paper proposes a multilevel logic approach to output coding using multilevel neurons in the output layer. Training convergence for a single multilevel perceptron is considered. It has been found that a multilevel neural network classifier with a reduced number of outputs is often able to learn faster and requires fewer weights. Concepts are illustrated with an example of a digit classifier.

A pruning algorithm is devised for multilayer multi-output feedforward perceptron networks. The algorithm efficiently reduces the total number of hidden units and the number of weights in the output layer. Test examples include network pruning for the IRIS classifier and for the bitmap digit classifier.

Current-mode network transformations are presented from which it is possible to generate new current-mode circuits by performing simple operations on the network. These transformations are developed by interchanging the ports of a current-mode circuit and exploiting the properties of current-conveyors and RC networks. The new circuits need at most one additional current-conveyor for buffering purposes.>

A new approach to the problem of n-dimensional continuous and sampled-data function approximation using a two-layer neural network is presented. The generalized Nyquist theorem is introduced to solve for the optimum number of training examples in n-dimensional input space. Choosing the smallest but still sufficient set of training vectors results in a reduced learning time for the network. Analytical formulas and algorithm for training set size reduction are developed and illustrated by two-dimensional data examples.>

Multilayer feedforward networks are often used for modeling complex relationships between the data sets. Deleting unimportant data components in the training sets could lead to smaller networks and reduced-size data vectors. This can be achieved by analyzing the total disturbance of network outputs due to perturbed inputs. The search for redundant data components is performed for networks with continuous outputs and is based on the concept in sensitivity of linearized neural networks. The formalized criteria and algorithm for pruning data vectors are formulated and illustrated with examples.>

Application of the contraction mapping theorem to single-layer feedback neural networks of a gradient-type is discussed. The sufficient condition for stability of a relaxation algorithm in actual continuous-time networks is derived and illustrated with an example. Results showing the stability of a numerical solution obtained with the relaxation algorithm are presented.