Most Convolutional Neural Networks (CNNs) consist of a number of stages of decreasing spatial resolution and increasing channel dimension between succeeding stages, each stage is composed of convolutional blocks that are repeated a number of times. Previous research on very simple CNNs consisting of a number of convolutional layers in each stage demonstrated that the introduction of feedback loops around convolutional layers can improve results. This paper studies the effectiveness of recursion on convolutional blocks in a more general setting and aims at explaining the results. Four recent models, namely ResNet, Inception, MobileNet and DenseNet are considered in this study. It is found that for all but DenseNet, the recursive version produces results that are similar or better than their feedforward counterpart when the number of convolutional blocks are preserved. To understand this finding and to discriminate the functional behaviors of the feedforward and recursive counterparts, we embark on three investigations: (1) measuring the evolution of the contextualization of the neurons of the last layer using the effective receptive field concept; (2) comparing the position and the size of the global coverage of the networks using class activation maps; and (3) analyzing the evolution of the organization of the feature space prior to the classifier using the Silhouette score. The investigations reveal that the recursion of a convolutional block shares many similarities with the behavior of a sequence of that block, indicating that a recursive alternative consisting of a single physical layer, can be regarded as a “faithful simulation” of its deeper‘feedforward counterpart. We conclude that except for densely connected models, the recursion of convolutional blocks is a safe and powerful alternative enhancing modern network architectures.

The Recursive Convolutional Layer (RCL) is a module that wraps a recursive feedback loop around a convolutional layer (CL). The RCL has been proposed to address some of the shortcomings of Convolutional Neural Networks (CNNs), as its unfolding increases the depth of a network without increasing the number of weights. We investigated the “naïve” substitution of CL with RCL on three base models: a 4-CL model, ResNet, DenseNet and their RCL-ized versions: C-FRPN, R-ResNet, and R-DenseNet using five image classification datasets. We find that this one-to-one replacement significantly improves the performances of the 4-CL model, but not those of ResNet or DenseNet. This led us to investigate the implication of the RCL substitution on the 4-CL model which reveals, among a number of properties, that RCLs are particularly efficient in shallow CNNs. We proceeded to re-visit the first set of experiments by gradually transforming the 4-CL model and the C-FRPN into respectively ResNet and R-ResNet, and find that the performance improvement is largely driven by the training regime whereas any depth increase negatively impacts the RCL-ized version. We conclude that the replacement of CLs by RCLs shows great potential in designing high-performance shallow CNNs.

Freezing of Gait (FoG) is a common symptom of Parkinson's disease (PD), manifesting as a brief, episodic absence, or marked reduction in walking, despite a patient's intention to move. Clinical assessment of FoG events from manual observations by experts is both time-consuming and highly subjective. Therefore, machine learning-based FoG identification methods would be desirable. In this article, we address this task as a fine-grained human action recognition problem based on vision inputs. A novel deep learning architecture, namely, higher order polynomial transformer (HP-Transformer), is proposed to incorporate pose and appearance feature sequences to formulate fine-grained FoG patterns. In particular, a higher order self-attention mechanism is proposed based on higher order polynomials. To this end, linear, bilinear, and trilinear transformers are formulated in pursuit of discriminative fine-grained representations. These representations are treated as multiple streams and further fused by a cross-order fusion strategy for FoG detection. Comprehensive experiments on a large in-house dataset collected during clinical assessments demonstrate the effectiveness of the proposed method, and an area under the receiver operating characteristic (ROC) curve (AUC) of 0.92 is achieved for detecting FoG.

The development of Convolutional Neural Networks (CNNs) trends towards models with an ever growing number of Convolutional Layers (CLs) and increases the number of trainable parameters significantly. Such models are sensitive to these structural parameters, which implies that large models have to be carefully tuned using hyperparameter optimisation, a process that can be very time consuming. In this paper, we study the usage of Recursive Convolutional Layers (RCLs), a module relying on an algebraic feedback loop wrapped around a CL, which can replace any CL in CNNs. Using three publicly available datasets, CIFAR10, CIFAR100 and SVHN, and a simple model comprised of 4 RCLs, we compare its performances with those obtained by its feedforward counterpart, and exhibit some core properties and use-cases of RCLs. In particular, we show that RCLs can lead to models of better performances, and that reducing the number of modules from four to one lead to a decrease in accuracy of 3.5% on average for models using RCLs, compared to 23% using CLs. Hence, the resulting architecture is much more robust to the addition or the removal of layers. We conclude by relating the effects obtained using additional CLs with those obtained using additional recursion on RCLs, which provides incentives that the latter can simulate an increase of depth but with no extra cost of parameters. Such results point to the potential benefits of either selectively or replacing all CLs by RCLs, in most recently introduced CNNs.

The popularity of neural networks that explicitly utilise the global correlation structure of their features have become vastly more popular ever since the Transformer architecture was developed. We propose to embed unsupervised Self-Organising Maps within neural networks as a means to model the global correlation structure. By enforcing topological preservation therein, such a neural network is able to represent more complex correlation structures and to produce interpretable visualisations as a byproduct. We validate this approach by comparing with the existing state-of-the-art attention substitute within its own ‘Hamburger’ framework and by illustrating maps learnt by the module. Overall, this paper serves as a proof of concept for integrating Self-Organising Maps within a supervised network.

Freezing of gait (FoG) is identified as a sudden and brief episode of movement cessation despite the intention to continue walking. It is one of the most disabling symptoms of Parkinson's disease (PD) and often leads to falls and injuries. Many computer-aided FoG detection methods have been proposed to use data collected from unimodal sources, such as motion sensors, pressure sensors, and video cameras. However, there are limited efforts of multimodal-based methods to maximize the value of all the information collected from different modalities in clinical assessments and improve the FoG detection performance. Therefore, in this study, a novel end-to-end deep architecture, namely graph fusion neural network (GFN), is proposed for multimodal learning-based FoG detection by combining footstep pressure maps and video recordings. GFN constructs multimodal graphs by treating the encoded features of each modality as vertex-level inputs and measures their adjacency patterns to construct complementary FoG representations, thus reducing the representation redundancy among different modalities. In addition, since GFN is devised to process multimodal graphs of arbitrary structures, it is expected to achieve superior performance with inputs containing missing modalities, compared to the alternative unimodal methods. A multimodal FoG dataset was collected, which included clinical assessment videos and footstep pressure sequences of 340 trials from 20 PD patients. Our proposed GFN demonstrates a great promise of multimodal FoG detection with an area under the curve (AUC) of 0.882. To the best of our knowledge, this is one of the first studies to utilize multimodal learning for automated FoG detection, which offers significant opportunities for better patient assessments and clinical trials in the future.

Sign language translation (SLT), which generates text in a spoken language from visual content in a sign language, is important to assist the hard-of-hearing community for their communications. Inspired by neural machine translation (NMT), most existing SLT studies adopted a general sequence to sequence learning strategy. However, SLT is significantly different from general NMT tasks since sign languages convey messages through multiple visual-manual aspects. Therefore, in this paper, these unique characteristics of sign languages are formulated as hierarchical spatio-temporal graph representations, including high-level and fine-level graphs of which a vertex characterizes a specified body part and an edge represents their interactions. Particularly, high-level graphs represent the patterns in the regions such as hands and face, and fine-level graphs consider the joints of hands and landmarks of facial regions. To learn these graph patterns, a novel deep learning architecture, namely hierarchical spatio-temporal graph neural network (HST-GNN), is proposed. Graph convolutions and graph self-attentions with neighborhood context are proposed to characterize both the local and the global graph properties. Experimental results on benchmark datasets demonstrated the effectiveness of the proposed method.

This study proposes a three‐stream model using two different types of deep convolutional neural networks (CNNs): (i) a spatial stream with a CNN on images; (ii) a ResNet (residual network) on optical flows; and, (iii) a ResNet on the concatenation of motion features. This model is applied to four datasets: (i) UCF Sports; (ii) Youtube Sports; (iii) SBU action interaction; and (iv) a subset of the UCF‐1M Sports. Using two optical flow estimation methods: (i) a motion boundary emphasised Epicflow (Edge Preserving Interpolation Correspondences for Optical Flow) method, (MBEpicflow); and (ii) the Flownet 2 method, a learning optical flow estimation method. It was found that (i) the proposed MBEpicflow outperforms the Flownet 2 method on the SBU dataset, while the Flownet 2 performs equally well or better than the MBEpicflow method on the other three datasets, and these results are the best when compared with those obtained using other approaches on all datasets evaluated. These results showed the importance of accurate optical flow plays in human action recognition, an aspect which has been seldom addressed. Moreover, it showed that if some measure of the global behaviours of motion is incorporated, the generalisation performance is often improved by 1–2%.

The recent explosive growth of multimedia data has posed a new set of challenges in computer vision, and video summarization (VS) techniques are increasingly important to automatically summarize a large amount of multimedia data in an effective and efficient manner. Recent years have witnessed the rise and developments of sparse representation based approaches for VS. While the existing methods select keyframes according to the information contained in the single frame, and such a selection based solely on single-frame information may not be robust. Therefore, in this paper, the information of the single frame's neighborhood is taken into consideration, and different weights are assigned to these neighbouring frames. We formulate the VS problem as a weighted neighborhood based representation model, and design a greedy pursuit algorithm to extract keyframes. Experimental results on a benchmark dataset demonstrate that the proposed method can outperform the state of the arts.

In this paper, we present multinomial latent logistic regression (MLLR), a new learning paradigm that introduces latent variables to logistic regression. By inheriting the advantages of logistic regression, MLLR is efficiently optimized using the second-order derivatives and provides effective probabilistic analysis on output predictions. MLLR is particularly effective in weakly supervised settings where the latent variable has an exponential number of possible values. The effectiveness of MLLR is demonstrated on four different image understanding applications, including a new challenging architectural style classification task. Furthermore, we show that MLLR can be generalized to general structured output prediction, and in doing so, we provide a thorough investigation of the connections and differences between MLLR and existing related algorithms, including latent structural SVMs and hidden conditional random fields.

People often take photographs at tourist sites and these pictures usually have two main elements: a person in the foreground and scenery in the background. This type of “souvenir photo” is one of the most common photos clicked by tourists. Although algorithms that aid a user-photographer in taking a well-composed picture of a scene exist [Ni et al. 2013], few studies have addressed the issue of properly positioning human subjects in photographs. In photography, the common guidelines of composing portrait images exist. However, these rules usually do not consider the background scene. Therefore, in this article, we investigate human-scenery positional relationships and construct a photographic assistance system to optimize the position of human subjects in a given background scene, thereby assisting the user in capturing high-quality souvenir photos. We collect thousands of well-composed portrait photographs to learn human-scenery aesthetic composition rules. In addition, we define a set of negative rules to exclude undesirable compositions. Recommendation results are achieved by combining the first learned positive rule with our proposed negative rules. We implement the proposed system on an Android platform in a smartphone. The system demonstrates its efficacy by producing well-composed souvenir photos.

This paper introduces a novel video representation based on superpixel segmentation and appearance and motion descriptors. Superpixel represents a very useful preprocessing step for a wide range of computer vision applications, as they group pixels into perceptually meaningful atomic regions which can be used for recognizing complex motion patterns. We construct a novel video representation in terms of superpixel-based histograms of oriented gradients (HOG), histograms of optical flow (HOF) and motion boundary histograms (MBH) descriptors, and integrate such representations with a bag-of-features (BoF) model for classification. The proposed approach is evaluated in the context of action classification on a challenging benchmark dataset: UCF Sports dataset and it achieves 87.9% generalization accuracy. The experimental results demonstrate the advantage of superpixel-based descriptors compared to other approaches for human action recognition.

In this paper, we apply Scale Invariant Feature Transform (SIFT) Flow, a recently developed method of video representation to human action recognition. SIFT Flow provides a convenient way to express the displacement between keypoints, points which are invariant to scale changes spatially, in two adjacent frames of a video, and it furnishes a compact way to describe the behaviour at keypoints and their neighborhoods as they move in time. A dense trajectory approach using keypoints is developed, and its shape descriptor can be obtained. Local appearance descriptor like histogram of oriented gradients (HOG) evaluated at keypoints, local motion descriptors, like histogram of oriented flows (HOF) and motion boundary histogram (MBH) can be evaluated using SIFT flows. The HOG, HOF MBH, evaluated using SIFT flow, and the keypoint trajectory shape descriptor, together can be used as a feature vector to represent the video. We compare the performance of a number of classifiers to classify the feature vectors, including a bag-of-words approach, support vector machines, linear and nonlinear. It is shown that the proposed novel approach based on keypoints, and SIFT flows produces competitive results when compared with other state-of-the-art results.

Many underlying relationships among data in several areas of science and engineering, e.g., computer vision, molecular chemistry, molecular biology, pattern recognition, and data mining, can be represented in terms of graphs. In this paper, we propose a new neural network model, called graph neural network (GNN) model, that extends existing neural network methods for processing the data represented in graph domains. This GNN model, which can directly process most of the practically useful types of graphs, e.g., acyclic, cyclic, directed, and undirected, implements a function tau(G,n) is an element of IR(m) that maps a graph G and one of its nodes n into an m-dimensional Euclidean space. A supervised learning algorithm is derived to estimate the parameters of the proposed GNN model. The computational cost of the proposed algorithm is also considered. Some experimental results are shown to validate the proposed learning algorithm, and to demonstrate its generalization capabilities.

In this paper, we will consider the approximation properties of a recently introduced neural network model called graph neural network (GNN), which can be used to process-structured data inputs, e.g., acyclic graphs, cyclic graphs, and directed or undirected graphs. This class of neural networks implements a function tau(G,n) is an element of IR(m) that maps a graph G and one of its nodes n onto an m-dimensional Euclidean space. We characterize the functions that can be approximated by GNNs, in probability, up to any prescribed degree of precision. This set contains the maps that satisfy a property called preservation of the unfolding equivalence, and includes most of the practically useful functions on graphs; the only known exception is when the input graph contains particular patterns of symmetries when unfolding equivalence may not be preserved. The result can be considered an extension of the universal approximation property established for the classic feedforward neural networks (FNNs). Some experimental examples are used to show the computational capabilities of the proposed model.

The World Wide Web is an unregulated communication medium which exhibits very limited means of quality control. Quality assurance has become a key issue for many information retrieval services on the Internet, e.g. web search engines. This paper introduces some quality evaluation and assessment methods to assess the quality of Web pages. The proposed quality evaluation mechanisms are based on a set of quality criteria which were extracted from a targeted user survey. A weighted algorithmic interpretation of the most significant user quoted quality criteria is proposed. In addition, the paper utilizes machine learning methods to produce a prediction of quality for Web pages before they are downloaded. The set of quality criteria allows us to implement a Web search engine with quality ranking schemes, leading to Web crawlers which can crawl directly quality Web pages. The proposed approaches produce some very promising results on a sizeable Web repository.

A novel 9 rules self-tuning PD+I fuzzy logic controller applicable for a class of nonlinear plants is proposed in this paper. The controller comprises of three separate fuzzy logic controllers with each uses minimum number of rules and the output scaling factor is tuned automatically depending on the tracking error dynamic conditions. The controller is applied to a two-link revolute robot for the tracking control. Simulation results show that the robustness and tracking performance of the proposed controller is comparable to standard PD+I fuzzy logic controller at low and medium speed motions. However, the performance of the proposed new design far exceeds the standard design at high speed motions.

In this article, we present a new approach to page ranking. The page rank of a collection of Web pages can be represented in a parameterized model, and the user requirements can be represented by a set of constraints. For a particular parameterization, namely, a linear combination of the page ranks produced by different forcing functions, and user requirements represented by a set of linear constraints, the problem can be solved using a quadratic programming method. The solution to this problem produces a set of parameters which can be used for ranking all pages in the Web. We show that the method is suitable for building customized versions of PageRank which can be readily adapted to the needs of a vertical search engine or that of a single user.

An artificial neural network model, capable of processing general types of graph structured data, has recently been proposed. This paper applies the new model to the computation of customised page ranks problem in the World Wide Web. The class of customised page ranks that can be implemented in this way is very general and easy because the neural network model is learned by examples. Some preliminary experimental findings show that the model generalizes well over unseen Web pages, and hence, may be suitable for the task of page rank computation on a large Web graph.

The national health insurance system in Australia records details on medical services and claims provided to its population. An effective method to the discovery of temporal behavioral patterns in the data set is proposed in this paper. The method consists of a two-step approach which is applied recursively to the data set. First, a clustering algorithm is used to segment the data into classes. Then, hidden Markov models are employed to find the underlying temporal behavioral patterns. These steps are applied recursively to features extracted from the data set until convergence. The main objective is to minimize the misclassification of patient profiles into various classes. This results in a hierarchical tree model consisting of a number of classes; each class groups similar patient temporal behavioral patterns together. The capabilities of the proposed method are demonstrated through the application to a subset of the Australian national health insurance data set. It is shown that the proposed method not only clusters data into various categories of interest, but it also automatically marks the periods in which similar temporal behavioral patterns occurred.

We propose a new algorithm for the incremental training of support vector machines (SVMs) that is suitable for problems of sequentially arriving data and fast constraint parameter variation. Our method involves using a "warm-start" algorithm for the training of SVMs, which allows us to take advantage of the natural incremental properties of the standard active set approach to linearly constrained optimization problems. Incremental training involves quickly retraining a support vector machine after adding a small number of additional training vectors to the training set of an existing (trained) support vector machine. Similarly, the problem of fast constraint parameter variation involves quickly retraining an existing support vector machine using the same training set but different constraint parameters. In both cases, we demonstrate the computational superiority of incremental training over the usual batch retraining method.

This work proposes an improvement of a supervised learning technique for self organizing maps. The ideas presented in This work differ from Kohonen's approach to supervision in that a.) a rejection term is used, and b.) rejection affects the training only locally. This approach produces superior results because it does not affect network weights globally, and hence, prevents the addition of noise to the learning process of remote neurons. We implemented the ideas into self-organizing maps for structured data (SOM-SD) which is a more general form of self-organizing maps capable of processing graphs. The capabilities of the proposed ideas are demonstrated by utilizing a relatively large real world learning problem from the area of image recognition. It is shown that the proposed method produces better classification performances while being more robust and flexible than other supervised approaches to SOM.

In this paper, the problem of blind deconvolution of dynamical systems is considered using a state space approach. A balanced parameterized canonical form is used as a model for the underlying dynamical system instead of the more common controller or observable canonical form. The results are compared with those obtained using a controller canonical form. It is shown experimentally that using the balanced parameterized canonical form is more robust than the ones using a controller canonical form.

In this paper, we consider the possibility of altering the PageRank of web pages, from an administrator's point of view, through the modification of the PageRank equation. It is shown that this problem can be solved using the traditional quadratic programming techniques. In addition, it is shown that the number of parameters can be reduced by clustering web pages together through simple clustering techniques. This problem can be formulated and solved using quadratic programming techniques. It is demonstrated experimentally on a relatively large web data set, viz., the WT10G, that it is possible to modify the PageRanks of the web pages through the proposed method using a set of linear constraints. It is also shown that the PageRank of other pages may be affected; and that the quality of the result depends on the clustering technique used. It is shown that our results compared well with those obtained by a HITS based method.

Many researchers have explored the use of neural-network representations for the adaptive processing of data structures. One of the most popular learning formulations of data structure processing is backpropagation through structure (BPTS). The BPTS algorithm has been successful applied to a number of learning tasks that involve structural patterns such as logo and natural scene classification. The main limitations of the BPTS algorithm are attributed to slow convergence speed and the long-term dependency problem for the adaptive processing of data structures. In this paper, an improved algorithm is proposed to solve these problems. The idea of this algorithm is to optimize the free learning parameters of the neural network in the node representation by using least-squares-based optimization methods in a layer-by-layer fashion. Not only can fast convergence speed be achieved, but the long-term dependency problem can also be overcome since the vanishing of gradient information is avoided when our approach is applied to very deep tree structures.

Recent developments in the area of neural networks produced models capable of dealing with structured data. Here, we propose the first fully unsupervised model, namely an extension of traditional self-organizing maps (SOMs), for the processing of labeled directed acyclic graphs (DAGs). The extension is obtained by using the unfolding procedure adopted in recurrent and recursive neural networks, with the replicated neurons in the unfolded network comprising of a full SOM. This approach enables the discovery of similarities among objects including vectors consisting of numerical data. The capabilities of the model are analyzed in detail by utilizing a relatively large data set taken from an artificial benchmark problem involving visual patterns encoded as labeled DAGs. The experimental results demonstrate clearly that the proposed model is capable of exploiting both information conveyed in the labels attached to each node of the input DAGs and information encoded in the DAG topology.

Recursive least squares (RLS)-based algorithms are a class of fast online training algorithms for feedforward multilayered neural networks (FMNNs). Though the standard RLS algorithm has an implicit weight decay term in its energy function, the weight decay effect decreases linearly as the number of learning epochs increases, thus rendering a diminishing weight decay effect as training progresses. In this paper, we derive two modified RLS algorithms to tackle this problem. In the first algorithm, namely, the true weight decay RLS (TWDRLS) algorithm, we consider a modified energy function whereby the weight decay effect remains constant, irrespective of the number of learning epochs. The second version, the input perturbation RLS (IPRLS) algorithm, is derived by requiring robustness in its prediction performance to input perturbations. Simulation results show that both algorithms improve the generalization capability of the trained network.

In using the B-spline network for nonlinear system modeling, owing to a lack of suitable theoretical results, it is quite difficult to choose an appropriate set of knot points to achieve a good network structure for minimizing, say, a minimum error criterion. In this paper, a novel knot-optimizing B-spline network is proposed to approximate the general nonlinear system behavior. The knot points are considered to be independent variables in the B-spline network and are optimized together with the B-spline expansion coefficients. The simulated annealing algorithm with an appropriate search strategy is used as an optimization algorithm for the training process in order to avoid any possible local minima. Examples involving dynamic systems up to six dimensions in the input space to the network are solved by the proposed method to illustrate the effectiveness of this approach.

We introduce two new models which are obtained through the modification of the well known methods MLP and cascade correlation. These two methods differ fundamentally as they employ learning techniques and produce network architectures that are not directly comparable. We extended the MLP architecture, and reduced the constructive method to obtain very comparable network architectures. The greatest benefit of these new models is that we can obtain an MLP-structured network through a constructive method based on the cascade correlation algorithm, and that we can train a cascade correlation structured network using the standard MLP learning technique. Additionally, we show that cascade correlation is a universal approximator, a fact that has not yet been discussed in literature.

The problem of high sensitivity in modeling is well known. Small perturbations in the model parameters may result in large, undesired changes in the model behavior. A number of authors have considered the issue of sensitivity in feedforward neural networks from a probabilistic perspective. Less attention has been given to such issues in recurrent neural networks. In this article, we present a new recurrent neural network architecture, that is capable of significantly improved parameter sensitivity properties compared to existing recurrent neural networks. The new recurrent neural network generalizes previous architectures by employing alternative discrete-time operators in place of the shift operator normally used. An analysis of the model demonstrates the existence of parameter sensitivity in recurrent neural networks and supports the proposed architecture. The new architecture performs significantly better than previous recurrent neural networks, as shown by a series of simple numerical experiments.

Given a multilayer perceptron (MLP) with a fixed architecture, there are functions that can be approximated up to any degree of accuracy, without having to increase the number of the hidden nodes. Those functions belong to the closure F of the set F of the maps realizable by the MLP. In this paper, we give a list of maps with this property. In particular, it is proven that 1) rational functions belongs to F for networks with inverse tangent activation function and 2) products of polynomials and exponentials belongs to F for networks with sigmoid activation function. Moreover, for a restricted class of MLP's, we prove that the list is complete and give an analytic definition of F.

In this letter we point out that multilayer neural networks (MLP's) with either sigmoidal units or radial basis functions can be given a canonical form with positive interunits weights, which does not restrict the well-known MLP universal computational capabilities. We give some results on the local minima of the error function using this canonical form. In particular, we prove that the local minima free conditions established in previous works can be relaxed significantly.

The paper considers the prediction of noisy time series data, specifically, the prediction of foreign exchange rate data. A novel hybrid neural network algorithm for noisy time series prediction is presented which exhibits excellent performance on the problem. The method is motivated by consideration of how neural networks work, and by fundamental difficulties with random correlations when dealing with small sample sizes and high noise data. The method permits the inference and extraction of rules. One of the greatest complaints against neural networks is that it is hard to figure out exactly what they are doing-this work provides one answer for the internal workings of the network. Furthermore, these rules can be used to gain insight into both the real world system and the predictor. The paper focuses on noisy time series prediction and rule inference-use of the system in trading would typically involve the utilization of other financial indicators and domain knowledge.

The performance of neural network simulations is often reported in terms of the mean and standard deviation of a number of simulations performed with different starting conditions. However, in many cases, the distribution of the individual results does not approximate a Gaussian distribution, may not be symmetric, and may be multimodal. We present the distribution of results for practical problems and show that assuming Gaussian distributions can significantly affect the interpretation of results, especially those of comparison studies. For a controlled task which we consider, we find that the distribution of performance is skewed toward better performance for smoother target functions and skewed toward worse performance for more complex target functions. We propose new guidelines for reporting performance which provide more information about the actual distribution.

We present a hybrid neural-network for human face recognition which compares favourably with other methods. The system combines local image sampling, a self-organizing map (SOM) neural network, and a convolutional neural network. The SOM provides a quantization of the image samples into a topological space where inputs that are nearby in the original space are also nearby in the output space, thereby providing dimensionality reduction and invariance to minor changes in the image sample, and the convolutional neural network provides partial invariance to translation, rotation, scale, and deformation. The convolutional network extracts successively larger features in a hierarchical set of layers. We present results using the Karhunen-Loeve transform in place of the SOM, and a multilayer perceptron (MLP) in place of the convolutional network for comparison. We use a database of 400 images of 40 individuals which contains quite a high degree of variability in expression, pose, and facial details. We analyze the computational complexity and discuss how new classes could be added to the trained recognizer.

We describe a sequence of experiments investigating the strengths and limitations of Fukushima's neocognitron as a handwritten digit classifier. Using the results of these experiments as a foundation, we propose and evaluate improvements to Fukushima's original network in an effort to obtain higher recognition performance. The neocognitron performance is shown to be strongly dependent on the choice of selectivity parameters and we present two methods to adjust these variables. Performance of the network under the more effective of the two new selectivity adjustment techniques suggests that the network fails to exploit the features that distinguish different classes of input data. To avoid this shortcoming, the network's final layer cells were replaced by a nonlinear classifier (a multilayer perceptron) to create a hybrid architecture. Tests of Fukushima's original system and the novel systems proposed in this paper suggest that it may be difficult for the neocognitron to achieve the performance of existing digit classifiers due to its reliance upon the supervisor's choice of selectivity parameters and training data.

The controller output error method (COEM) is introduced and applied to the design of adaptive fuzzy control systems. The method employs a gradient descent algorithm to minimize a cost function which is based on the error at the controller output. This contrasts with more conventional methods which use the error at the plant output. The cost function is minimized by adapting some or all of the parameters of the fuzzy controller. The proposed adaptive fuzzy controller is applied to the adaptive control of a nonlinear plant and is shown to be capable of providing good overall system performance.

Faces represent complex, multidimensional, meaningful visual stimuli and developing a computational model for face recognition is difficult. We present a hybrid neural network solution which compares favorably with other methods. The system combines local image sampling, a self-organizing map neural network, and a convolutional neural network. The self-organizing map provides a quantization of the image samples into a topological space where inputs that are nearby in the original space are also nearby in the output space, thereby providing dimensionality reduction and invariance to minor changes in the image sample, and the convolutional neural network provides for partial invariance to translation, rotation, scale, and deformation. The method is capable of rapid classification, requires only fast, approximate normalization and preprocessing, and consistently exhibits better classification performance than the eigenfaces approach on the database considered as the number of images per person in the training database is varied from 1 to 5. With 5 images per person the proposed method and eigenfaces result in 3.8% and 10.5% error respectively. The recognizer provides a measure of confidence in its output and classification error approaches zero when rejecting as few as 10% of the examples. We use a database of 400 images of 40 individuals which contains quite a high degree of variability in expression, pose, and facial details.

In this paper, a matrix formulation of fuzzy rule based systems is introduced. A gradient descent training algorithm for the determination of the unknown parameters can also be expressed in a matrix form for various adaptive fuzzy networks. When converting a rule-based system to the proposed matrix formulation, only three sets of linear/nonlinear equations are required instead of set of rules and an inference mechanism. There are a number of advantages which the matrix formulation has compared with the linguistic approach. Firstly, it obviates the differences among the various architectures; and secondly, it is much easier to organize data in the implementation or simulation of the fuzzy system. The formulation will be illustrated by a number of examples.

An adaptive membership function scheme for general additive fuzzy systems is proposed in this paper. The proposed scheme can adapt a proper membership function for any nonlinear input-output mapping, based upon a minimum number of rules and an initial approximate membership function. This parameter adjustment procedure is performed by computing the error between the actual and the desired decision surface. Using the proposed adaptive scheme for fuzzy system, the number of rules can be minimized. Nonlinear function approximation and truck backer-upper control system are employed to demonstrate the viability of the proposed method.

A wide range of modifications to the backpropagation (BP) algorithm, motivated by heuristic arguments and optimisation theory, has been examined on a real-world medical signal classification problem. The method of choice depends both upon the nature of the learning task and whether one wants to optimise learning for speed or generalisation. It was found that, comparatively, standard BP was sufficiently fast and provided good generalisation when the task was to learn the training set within a given error tolerance. However, if the task was to find the global minimum, then standard BP failed to do so within 100000 iterations, but first order methods which adapt the stepsize were as fast as, if not faster than, conjugate gradient and quasi-Newton methods. Second order methods required the same amount of fine tuning of line search and restart parameters as did the first order methods of their parameters in order to achieve optimum performance.

This paper presents a novel indirect learning architecture which uses a single neural network in implementation. The new architecture generates the error signal required in training the controller network by an innovative design using a memory element and few switches. The new controller needs only half the number of neurons and connection weights in comparison with the original indirect learning architecture. Also given are the simulation results in controlling a nonlinear plant.

In this paper, we will consider a number of local-recurrent-global-feedforward (LRGF) networks that have been introduced by a number of research groups in the past few years. We first analyze the various architectures, with a view to highlighting their differences. Then we introduce a general LRGF network structure that includes most of the network architectures that have been proposed to date. Finally we will indicate some open issues concerning these types of networks.

A network architecture with a global feedforward local recurrent construction was presented recently as a new means of modeling nonlinear dynamic time series (Back and Tsoi 1991a). The training rule used was based on minimizing the least mean square (LMS) error and performed well, although the amount of memory required for large networks may become significant if a large number of feedback connections are used. In this note, a modified training algorithm based on a technique for linear filters is presented, simplifying the gradient calculations significantly. The memory requirements are reduced from O[na(na + nb)Ns] to O[(2na + nb)Ns], where na is the number of feedback delays, and Ns is the total number of synapses. The new algorithm reduces the number of multiply-adds needed to train each synapse by na at each time step. Simulations indicate that the algorithm has almost identical performance to the previous one.

A difficulty with the application of the closed-form training algorithm for the neocognitron proposed by T.H. Hildebrandt (ibid., vol.2, p.557-88, Nov. 1991) is reported. In applying this algorithm the commenters have observed that S-cells frequently fail to respond to features that they have been trained to extract. Results which indicate that this training vector rejection in an important factor in the overall classification performance of the neocognitron trained using Hildebrandt's procedure are presented. In reply, Hildebrandt explains that the negative results obtained by the commenter are not specific to the proposed algorithm and are easily explained in terms of set theory.

Time-series modeling is a topic of growing interest in neural network research. Various methods have been proposed for extending the nonlinear approximation capabilities to time-series modeling problems. A multilayer perceptron (MLP) with a global-feedforward local-recurrent structure was recently introduced as a new approach to modeling dynamic systems. The network uses adaptive infinite impulse response (IIR) synapses (it is thus termed an IIR MLP), and was shown to have good modeling performance. One problem with linear IIR filters is that the rate of convergence depends on the covariance matrix of the input data. This extends to the IIR MLP: it learns well for white input signals, but converges more slowly with nonwhite inputs. To solve this problem, the adaptive lattice multilayer perceptron (AL MLP), is introduced. The network structure performs Gram-Schmidt orthogonalization on the input data to each synapse. The method is based on the same principles as the Gram-Schmidt neural net proposed by Orfanidis (1990b), but instead of using a network layer for the orthogonalization, each synapse comprises an adaptive lattice filter. A learning algorithm is derived for the network that minimizes a mean square error criterion. Simulations are presented to show that the network architecture significantly improves the learning rate when correlated input signals are present.

A new neural network architecture involving either local feedforward global feedforward, and/or local recurrent global feedforward structure is proposed. A learning rule minimizing a mean square error criterion is derived. The performance of this algorithm (local recurrent global feedforward architecture) is compared with a local-feedforward global-feedforward architecture. It is shown that the local-recurrent global-feedforward model performs better than the local-feedforward global-feedforward model.