Synchronization of chaotic systems and/or maps is a key step to implement secure communication schemes with chaos. If the process to synchronize chaotic systems is modeled stochastic, schemes based on extended Kalman filter (EKF) and unscented Kalman filter (UKF) have been studied in the past. However, such nonlinear filters are employed with assumptions of Gaussian noise processes and the Markov property. Further, EKF and UKF are suboptimal filtering methods, incurring unacceptable errors for high nonlinear systems. In this paper, neural filter (NF) is proposed for chaotic synchronization. This new approach requires no mentioned assumptions and achieves optimal filter. Numerical comparisons between the proposed approach and existing schemes are presented in this paper, showing the superiority of the proposed approach.

By the fundamental neural filtering theorem, a properly trained recursive neural filter with fixed weights that processes only the measurement process generates recursively the conditional expectation of the signal process with respect to the joint probability distributions of the signal and measurement processes and any uncertain environmental process involved. This means that a recursive neural filter with fixed weights has the ability to adapt to the uncertain environmental parameter. The neural filter with this ability is called an accommodative neural filter. In this paper, we show that if the uncertain environmental process is observable from the measurement process, the accommodative neural filter outputs virtually the estimate of the signal process that would be generated by a non-adaptive minimal-variance filter as if the precise value of the uncertain environmental process were given. Numerical results comparing the accommodative neural filter and the existing non-adaptive filters each designed for a precise value of the environmental process confirm our theorem and show the advantages of the accommodative neural filter in both accuracy and efficiency.

A new method of training deep neural networks including the convolutional network is proposed. The method deconvexifies the normalized risk-averting error (NRAE) gradually and switches to the risk-averting error (RAE) whenever RAE is computationally manageable. The method creates tunnels between the depressed regions around saddle points, tilts the plateaus, and eliminates nonglobal local minima. Numerical experiments show the effectiveness of gradual deconvexification as compared with unsupervised pretraining. After the minimization process, a statistical pruning method is used to enhance the generalization capability of the neural network under training. Numerical results show further reduction of the testing criterion.

Proper use of the normalized risk-averting error (NRAE) criterion has been shown to avoid nonglobal local minima effectively in the mean squared error (MSE) criterion. For training on large datasets, a pairwise algorithm for the NRAE criterion similar to the widely-used least mean square algorithm for the MSE criterion is proposed. The gradual deconvexification method employing this pairwise algorithm is tested on examples with built-in nonglobal local minima that are difficult to avoid and on recognition of handwritten numerals with the MNIST dataset. Numerical experiments show that the pairwise algorithm for the NRAE criterion is computationally more economical than the corresponding batch algorithm and delivers multilayer perceptrons with better performances than training methods based on the MSE criterion.

A method of training neural networks using the risk-averting error (RAE) criterion Jλ(w), which was presented in IJCNN 2001, has the capability to avoid nonglobal local minima, but suffers from a severe limitation on the magnitude of the risk-sensitivity index λ. To eliminating the limitation, an improved method using the normalized RAE (NRAE) Cλ(w) was proposed in ISNN 2012, but it requires a selection of a proper λ, whose range may be dependent on the application. A new training method called the gradual deconvexification (GDC) is proposed in this paper. It starts with a very large λ and gradually decreases it in the training process until a global minimum of Cλ(w) or a good generalization capability is achieved. GDC training method was tested on a large number of numerical examples and produced a very good result in each test.

A low-order model (LOM) of biological neural networks, which is biologically plausible, is herein reported. LOM is a recurrent hierarchical network composed of novel models of dendritic trees for encoding information, spiking neurons for computing subjective probability distributions and generating spikes, nonspiking neurons for transmitting inhibitory graded signals to modulate their neighboring spiking neurons, unsupervised and supervised covariance learning and accumulation learning mechanisms, synapses, a maximal generalization scheme, and feedback connections with different delay durations. An LOM with a main network that learns without supervision and clusters similar patterns, and offshoot structures that learn with supervision and assign labels to clusters formed in the main network is proposed as a learning machine that learns and retrieves easily, generalizes maximally on corrupted, distorted and occluded temporal and spatial patterns, and utilizes fully the spatially and temporally associated information.

A biologically plausible low-order model (LOM) of biological neural networks is proposed. LOM is a recurrent hierarchical network of models of dendritic nodes and trees; spiking and nonspiking neurons; unsupervised, supervised covariance and accumulative learning mechanisms; feedback connections; and a scheme for maximal generalization. These component models are motivated and necessitated by making LOM learn and retrieve easily without differentiation, optimization, or iteration, and cluster, detect, and recognize multiple and hierarchical corrupted, distorted, and occluded temporal and spatial patterns. Four models of dendritic nodes are given that are all described as a hyperbolic polynomial that acts like an exclusive-OR logic gate when the model dendritic nodes input two binary digits. A model dendritic encoder that is a network of model dendritic nodes encodes its inputs such that the resultant codes have an orthogonality property. Such codes are stored in synapses by unsupervised covariance learning, supervised covariance learning, or unsupervised accumulative learning, depending on the type of postsynaptic neuron. A masking matrix for a dendritic tree, whose upper part comprises model dendritic encoders, enables maximal generalization on corrupted, distorted, and occluded data. It is a mathematical organization and idealization of dendritic trees with overlapped and nested input vectors. A model nonspiking neuron transmits inhibitory graded signals to modulate its neighboring model spiking neurons. Model spiking neurons evaluate the subjective probability distribution (SPD) of the labels of the inputs to model dendritic encoders and generate spike trains with such SPDs as firing rates. Feedback connections from the same or higher layers with different numbers of unit-delay devices reflect different signal traveling times, enabling LOM to fully utilize temporally and spatially associated information. Biological plausibility of the component models is discussed. Numerical examples are given to demonstrate how LOM operates in retrieving, generalizing, and unsupervised and supervised learning.

By a fundamental neural filtering theorem, a recurrent neural network with fixed weights is known to be capable of adapting to an uncertain environment. This letter reports some mathematical results on the performance of such adaptation for series-parallel identification of a dynamical system as compared with the performance of the best series-parallel identifier possible under the assumption that the precise value of the uncertain environmental process is given. In short, if an uncertain environmental process is observable (not necessarily constant) from the output of a dynamical system or constant (not necessarily observable), then a recurrent neural network exists as a series-parallel identifier of the dynamical system whose output approaches the output of an optimal series-parallel identifier using the environmental process as an additional input.

Recurrent multilayer network structures and Hebbian learning are two of the research results on the brain that are widely accepted by neuroscientists. The former led to multilayer perceptrons (MLPs) and recurrent MLPs, and the latter to associative memories. This paper presents recurrent and/or multilayer networks of novel associative memories, each being a new functional model of the neuron with its dendritic weights. The recurrent and/or multilayer networks are called probabilistic associative memory (PAMs) and the functional model of the neuron is called processing element. Each processing element with its weights learns by the Hebbian rule and computes a subjective conditional probability as well as a point estimate of the class label of the cause(s) within its receptive field. Detected and recognized causes are integrated by the processing elements, aided by feedbacks, from layer to layer and from time to time into a spatial and/or temporal hierarchy of causes to facilitate under standing of the pattern or sequence of patterns presented to the PAM. Mainly due to multilayer and recurrent structures and Hebbian learning, PAMs have many such desirable properties of a pattern recognizer or learning machine as (1) fast learning and responding to large temporal and spatial patterns; (2) detecting and recognizing multiple causes associatively and hierarchically; (3) having good generalization capabilities; (4) representing and resolving ambiguity and uncertainty with conditional probabilities.

As opposed to the analytic approach used in the modern theory of optimal filtering, a synthetic approach is presented. The signal/sensor data, which are generated by either computer simulation or actual experiments, are synthesized into a filter by training a recurrent multilayer perceptron (RMLP) with at least one hidden layer of fully or partially interconnected neurons and with or without output feedbacks. The RMLP, after adequate training, is a recursive filter optimal for the given structure, with the lagged feedbacks carrying the optimal conditional statistics at each time point. Above all, it converges to the minimum variance filter as the number of hidden neurons increases. We call such an RMLP a neural filter. Simulation results show that the neural filters with only a few hidden neurons consistently outperform the extended Kalman filter and even the iterated extended Kalman filter for the simple nonlinear signal/sensor systems considered.

The training of a multilayer perceptron is formulated as a maximum likelihood estimation problem. Statistical techniques such as the EM algorithm and the linear regression are used to exploit the linearity and separability structures of the feedforward multilayer neural networks. The resulting algorithm iterates a few linear regressors and nonlinear data transformers. All its parameters are statistically determined. Initial numerical experiments show that the algorithm has outstanding performance and deserves to be fully developed.>

Two methods for integrating prior information about the signal directions of arrival and the sensor locations and intrinsic phases into the cyclic regression method for signal direction-finding as well as sensor self-localization and calibration are presented. Only coherent signals are considered. The methods demonstrate the versatility of the cyclic regression method.>

A novel approach is presented that greatly reduces the amount of computation for the maximum-likelihood estimation of the multiple signal arrival directions. The approach iterates a finite sequence of linear regression procedures and is therefore called cyclic regression. It is believed that cyclic regression requires much less computation than MUSIC. Being very versatile, the approach can effectively handle array sensor uncertainties.>

The problem of localizing the array sensors, relative to a reference sensor, using planar wave-front, is formulated and studied by analyzing the eigenvectors and eigenvalues of the sample covariance matrices. "Observability" of the sensor locations requires a minimum of two calibrating signals with known direction of arrival. We consider three types of calibrating signal in accordance with whether it can be timely turned on and off or whether it appears alone in a frequency bin. The technique of covariance differencing (or despoking) and an iterative method are used to treat separately or jointly these different types of signal. The ambiguity resulting from taking phase delay modulo 2π and the rank-2 ambiguity of the array manifold parametrized by the phase delay are characterized and eliminated in our algorithms.

A systematic method to develop approximate nonlinear estimators is presented, in the form of a functional series, for the signal that modulates the rate of a counting process. The estimators are optimal for the given structure and approach the minimum variance (MV) estimator when the approximation order increases. Two kinds of functional series, the iterated integral (II) series and the Fourier-Charlier (FC) series, are used. Product-to-sum formulas for the II and FC functionals are derived. By using the formulas, the MV estimate is projected onto the Hilbert subspaces of the II and the FC series driven by the counting observations with the given index set. The projection results in a Wiener-Hopf type equation for the II kernels and a system of linear algebraic equations for the FC coefficients. The FC series estimator consists of finitely many single Wiener integrals of the counting observations and a nonlinear postprocessor. The nonlinear postprocessor, however, is not memoryless.

Several models are proposed which are believed to be generic for the estimation of discrete-time axial processes. By introducing axial exponential Fourier densities and axial exponential trigonometric densities, finite-dimensional recursive schemes are obtained for updating the conditional density functions. The underlying idea is the closure properties under the Bayes rule of the various combinations of these exponential densities. An estimation error criterion is introduced which is compatible with a Riemannian metric. The corresponding Optimal axial estimates be easily computed from the conditional probability distributions.

A new representation, called an exponential Fourier density, of a probability density on a circle,S^{1}is introduced. It is shown that a density of bounded variation onS^{1}can be approximated as closely as desired by such a representation in the space of square-integrable functions onS^{1}. The exponential Fourier densities have the desirable feature of being closed under the operation of taking conditional distributions. Facilitated by the use of these densities, finite-dimensional, recursive, and optimal estimation and detection schemes are derived for some simple models including a PSK communication system. A deficiency of the exponential Fourier densities is that they are not closed under convolution. How to circumvent this deficiency is still an open question.

Two classes of probability densities, the exponential Fourier densities and the exponential trigonometric densities, are introduced on the unit sphere, as well as four kinds of displacements. In general, neither class is closed under the operation of taking conditional distributions with respect to any of the displacements. A combined usage of both classes is required to study the estimation and detection models obtained from various combinations of the displacements. The merits and disadvantages of each model are discussed. Recursive formulas for the conditional densities and the likelihood ratios are derived for many of the models. The additive measurement noise case is also considered in detail. An error criterion for direction estimation is presented with respect to which the optimal estimates can be easily computed from the probability distribution. A deficiency of the models and techniques developed in this paper is that random driving terms are disallowed in the signal processes.

The filtering problem of a system with linear dynamics and non-Gaussian a priori distribution is investigated. A closed-form exact solution to the problem is presented along with an approximation scheme. The approximation is made in the construction of a mathematical model. It reduces optimal estimation to a combination of linear estimations. The asymptotic behavior of the filter is examined. The limiting distributions of the conditional mean and the conditional-error covariance exist as the time interval of observation becomes infinite. In the autonomous case, the estimate for the Wiener problem satisfies a linear stochastic differential equation. A large class of nonlinear problems with more nonlinear features than the one discussed above can be reduced to it through the idea of finite-dimensional sensor orbits. The general idea and a number of examples are discussed briefly.