We describe a general technique for estimating the intractable quantities that occur in a wide variety of large-scale probabilistic models. The technique transforms intractable sums into integrals which are subsequently approximated via saddle point methods. When applied to sigmoid and noisy-OR networks, the technique yields a generic mean-field approximation as well as a second order Gaussian approximation that accounts for the pairwise correlations between random variables in the network. In two example models, we observe that our lowest order approximation is identical to expressions obtained using Plefka's approach for deriving the TAP equations.

The ability to discriminate a reentry vehicle (RV) from booster parts and other debris is critical to theater ballistic missile defense (TBMD). As it travels along its trajectory, a threat missile separates into a reentry vehicle (RV) and clutter. The latter consists of several tanks, separation debris and fragments of hot fuel. Interception of the RV requires discrimination of the RV from the clutter. The required discrimination must be performed no later than 30 seconds before intercept. A time-delay neural network (TDNN) is proposed for discrimination of the RV from other missile parts debris. The rate of change of the IR signature over several seconds is used as a discriminant. The performances of two different approaches are compared: 1) A TDNN that employs backpropagation weight updates is used to calculate activation levels for output nodes. The RV is selected via winner-take-all. A TDNN that updates weights using a cross-entropy error function with a softmax activation function is used to estimate assignment probabilities. The RV is subsequently selected via a probabilistic assignment algorithm that imposes the constraint that there can only be a single RV. We found that the TDNN employing backpropagated softmax learning performed better than the TDNN employing backpropagated least mean square learning.

A representation-independent mean-field dynamics is presented for batched TD(λ). The task is learning to predict the outcome of an indirectly observed absorbing Markov process. In the case of linear representations, the discrete-time deterministic iteration is an affine map whose fixed point can be expressed in closed form without the assumption of linearly independent observation vectors. Batched linear TD(λ) is proved to converge with probability 1 for all λ. Theory and simulation agree on a random walk example.

We consider the problem of decoding block coded data, using a physical dynamical system. We sketch out a decompression algorithm for fractal block codes and then show how to implement a recurrent neural network using physically simple but highly-nonlinear, analog circuit models of neurons and synapses. The nonlinear system has many fixed points, but we have at our disposal a procedure to choose the parameters in such a way that only one solution, the desired solution, is stable. As a partial proof of the concept, we present experimental data from a small system a 16-neuron analog CMOS chip fabricated in a 2m analog p-well process. This chip operates in the subthreshold regime and, for each choice of parameters, converges to a unique stable state. Each state exhibits a qualitatively fractal shape.

The concept of Contrastive Learning (CL) is developed as a family of possible learning algorithms for neural networks. CL is an extension of Deterministic Boltzmann Machines to more general dynamical systems. During learning, the network oscillates between two phases. One phase has a teacher signal and one phase has no teacher signal. The weights are updated using a learning rule that corresponds to gradient descent on a contrast function that measures the discrepancy between the free network and the network with a teacher signal. The CL approach provides a general unified framework for developing new learning algorithms. It also shows that many different types of clamping and teacher signals are possible. Several examples are given and an analysis of the landscape of the contrast function is proposed with some relevant predictions for the CL curves. An approach that may be suitable for collective analog implementations is described. Simulation results and possible extensions are briefly discussed together with a new conjecture regarding the function of certain oscillations in the brain. In the appendix, we also examine two extensions of contrastive learning to time-dependent trajectories.

Error backpropagation in feedforward neural network models is a popular learning algorithm that has its roots in nonlinear estimation and optimization. It is being used routinely to calculate error gradients in nonlinear systems with hundreds of thousands of parameters. However, the classical architecture for backpropagation has severe restrictions. The extension of backpropagation to networks with recurrent connections will be reviewed. It is now possible to efficiently compute the error gradients for networks that have temporal dynamics, which opens applications to a host of problems in systems identification and control.

Useful computation can be performed by systematically exploiting the phenomenology of nonlinear dynamical systems. Two dynamical phenomena are isolated into primitive architectural components which perform the operations of continuous nonlinear transformation and autoassociative recall. Backpropagation techniques for programming the architectural components are presented in a formalism appropriate for a collective nonlinear dynamical system. It is shown that conventional recurrent backpropagation is not capable of storing multiple patterns in an associative memory which starts out with an insufficient number of point attractors. It is shown that a modified algorithm can solve this problem by introducing new attractors near the to-be-stored patterns. Two primitive components are assembled into an elementary machine and trained to perform invariant pattern recognition with respect to small arbitrary transformations of the input pattern, provided the transformations are sufficiently small. The machine realizes modular learning since error signals do not propagate across the boundaries of the components.