Experimental constraints have traditionally implied separate studies of different cortical functions, such as memory and sensory-motor control. Yet certain cortical modalities, while repeatedly observed and reported, have not been clearly identified with one cortical function or another. Specifically, while neuronal membrane and synapse polarities with respect to a certain potential value have been attracting considerable interest in recent years, the purposes of such polarities have largely remained a subject for speculation and debate. Formally identifying these polarities as on-off neuronal polarity gates, we analytically show that cortical circuit structure, behavior, and memory are all governed by the combined potent effect of these gates, which we collectively term circuit polarity. Employing widely accepted and biologically validated firing rate and plasticity paradigms, we show that circuit polarity is mathematically embedded in the corresponding models. Moreover, we show that the firing rate dynamics implied by these models are driven by ongoing circuit polarity gating dynamics. Furthermore, circuit polarity is shown to segregate cortical circuits into internally synchronous, externally asynchronous subcircuits, defining their firing rate modes in accordance with different cortical tasks. In contrast to the Hebbian paradigm, which is shown to be susceptible to mutual neuronal interference in the face of asynchrony, circuit polarity is shown to block such interference. Noting convergence of synaptic weights, we show that circuit polarity holds the key to cortical memory, having a segregated capacity linear in the number of neurons. While memory concealment is implied by complete neuronal silencing, memory is restored by reactivating the original circuit polarity. Finally, we show that incomplete deterioration or restoration of circuit polarity results in memory modification, which may be associated with partial or false recall, or novel innovation.

Widely accepted neural firing and synaptic potentiation rules specify a cross-dependence of the two processes, which, evolving on different timescales, have been separated for analytic purposes, concealing essential dynamics. Here, the morphology of the firing rates process, modulated by synaptic potentiation, is shown to be described by a discrete iteration map in the form of a thresholded polynomial. Given initial synaptic weights, a firing activity is triggered by conductance. Elementary dynamic modes are defined by fixed points, cycles, and saddles of the map, building blocks of the underlying firing code. Showing parameter-dependent multiplicity of real polynomial roots, the map is proved to be noninvertible. The incidence of chaos is then implied by the parameter-dependent existence of snap-back repellers. The highly patterned geometric and statistical structures of the associated chaotic attractors suggest that these attractors are an integral part of the neural code. It further suggests the chaotic attractor as a natural mechanism for statistical encoding and temporal multiplexing of neural information. The analytic findings are supported by simulation.

Real-time human-computer interaction plays an important role in virtual-environment (VE) applications. Such interaction can be improved by detecting and reacting to the user's head motion. Today's VE systems use head-mounted inertial sensors to update and spatially stabilize the image displayed to a user through a head-mounted display. Since motion can only be detected after it has already occurred, latencies in the stabilization scheme can only be reduced but never eliminated. Such latencies slow down manual control, cause inaccuracies in matching real and virtual objects through a half-transparent display, and reduce the sense of presence. This paper presents novel methods for reducing VE latencies by anticipating future head motion based on electromyographic (EMG) signals originating from the major neck muscles and head kinematics; it also reports results for anticipation of 17.5 and 35 ms. Features extracted from the EMG signals are used to train a neural network in mapping EMG data, given present head kinematics, into future head motion. The trained network is then used in real time for head-motion anticipation, which gives the VE system the time advantage necessary to compensate for the inherent latencies. The main contribution of this work is the use of EMG energy and bounded head acceleration as the key input/output information, which results in improved performance compared to the previous work

Kernels are key components of pattern recognition mechanisms. We propose a universal kernel optimality criterion, which is independent of the classifier to be used. Defining data polarization as a process by which points of different classes are driven to geometrically opposite locations in a confined domain, we propose selecting the kernel parameter values that polarize the data in the associated feature space. Conversely, the kernel is said to be polarized by the data. Kernel polarization gives rise to an unconstrained optimization problem. We show that complete kernel polarization yields consistent classification by kernel-sum classifiers. Tested on real-life data, polarized kernels demonstrate a clear advantage over the Euclidean distance in proximity classifiers. Embedded in a support vectors classifier, kernel polarization is found to yield about the same performance as exhaustive parameter search.

Real classification problems involve structured data that can be essentially grouped into a relatively small number of clusters. It is shown that, under a local clustering condition, a set of points of a given class, embedded in binary space by a set of randomly parameterized surfaces, is linearly separable from other classes, with arbitrarily high probability. We call such a data set a local relative cluster. The size of the embedding set is shown to be inversely proportional to the squared local clustering degree. A simple parameterization by embedding hyperplanes, implementing a voting system, results in a random reduction of the nearest-neighbor method and leads to the separation of multicluster data by a network with two internal layers. This represents a considerable reduction of the learning problem with respect to known techniques, resolving a long-standing question on the complexity of random embedding. Numerical tests show that the proposed method performs as well as state-of the-art methods and in a small fraction of the time.

We propose a new Markov Chain Monte Carlo algorithm, which is a generalization of the stochastic dynamics method. The algorithm performs exploration of the state-space using its intrinsic geometric structure, which facilitates efficient sampling of complex distributions. Applied to Bayesian learning in neural networks, our algorithm was found to produce results comparable to the best state-of-the-art method while consuming considerably less time.

It is shown that, under a local clustering condition, a set of points of a given class, embedded in binary space by a set of randomly parameterized surfaces, is linearly separable from other classes, with arbitrarily high probability. We call such a data set a local relative cluster. The size of the embedding set is linear in the input dimension and inversely proportional to the squared local clustering degree. A simple parameterization by embedding hyperplanes leads to the separation of multi-cluster data by a network with two internal layers. The computational complexity is linear in the number of relative clusters in the data. This represents a considerable reduction of the learning problem with respect to known techniques, resolving a long-standing question on the complexity of random embedding. Numerical tests show that the proposed method performs as well as state-of the-art methods, in a small fraction of the time.

It is shown that partial classification which allows for indecision in certain regions of the data space, can increase a benefit function, defined as the difference between the probabilities of correct and incorrect decisions, joint with the event that a decision is made. This is particularly true for small data samples, which may cause a large deviation of the estimated separation surface from the intersection surface between the corresponding probability density functions. Employing a particular density estimation method, an indecision domain is naturally defined by a single parameter whose optimal size, maximizing the benefit function, is derived from the data. The benefit function is shown to translate into profit in stock trading. Employing medical and economic data, it is shown that partial classification produces, on average, higher benefit values than full classification, assigning each new object to a class, and that the marginal benefit of partial classification reduces as the data size increases.

In the author's recent paper (ibid., vol.20, no.8 (1998)), it was shown that partial classification, which allows for indecision in certain regions of the data space, can increase a benefit function. It was also claimed that in the case of ideal separation, corresponding to the maximum likelihood criterion, partial classification cannot be more beneficial than full classification. The latter claim is false, and a counterexample is given to support the argument. The error in the original statement was caused by the omission of certain terms in the calculation of the decision probability. The erroneous result may be omitted without changing the rest of the paper. In fact, the present observation strengthens the main claim of the paper, that partial classification can be beneficial.

An estimate of the probability density function of a random vector is obtained by maximizing the output entropy of a feedforward network of sigmoidal units with respect to the input weights. Classification problems can be solved by selecting the class associated with the maximal estimated density. Newton's optimization method, applied to the estimated density, yields a recursive estimator for a random variable or a random sequence. A constrained connectivity structure yields a linear estimator, which is particularly suitable for "real time" prediction. A Gaussian nonlinearity yields a closed-form solution for the network's parameters, which may also be used for initializing the optimization algorithm when other nonlinearities are employed. A triangular connectivity between the neurons and the input, which is naturally suggested by the statistical setting, reduces the number of parameters. Applications to classification and forecasting problems are demonstrated.

An estimate of the probability density function of a random vector is obtained by maximizing the output entropy of a feedforward network of sigmoidal units with respect to the input weights. A normalized version of the sigmoidal transfer function simplifies the algorithm considerably and leads to a maximum entropy estimate of the input density under a certain model. Newton's method, applied to the estimated density, yields a recursive estimator for a random variable or a random sequence. A constrained connectivity structure yields a linear estimator, which is particularly suitable for "real time" prediction.

The capacity of networks of ternary neurons, storing, by the so-called Hebbian rule, sparse vectors over {-1, 0, 1}(N), is shown to be at least of the order of N(2)/K log N, where K=Omega(log N) is the number of nonzero elements in each vector. The error correction capability of such networks is also analyzed. These results generalize previously known capacity bounds for binary networks storing vectors of equally probable {+/-1} bits and yield considerably higher capacities for small values of K.

A mechanism fur the visual detection of obstacles is presented. A new immediacy measure, representing the imminence of collision between an object and a moving observer, is defined. A diffusion process on the image domain, whose initial condition is determined by the motion field normal to the object's boundary, is shown to converge asymptotically to the immediacy measure. A network of locally connected cells, derived from a finite-difference approximation of the diffusion equation, estimates the immediacy measure from normal velocity and boundary information provided by a motion measurement and segmentation stage. The algorithm's performance on real image sequences is demonstrated.>

A neural network that retrieves stored binary vectors, when probed by possibly corrupted versions of them, is presented. It employs sparse ternary internal coding and autocorrelation (Hebbian) storage. It is symmetrically structured and, consequently, can be folded into a feedback configuration. Bounds on the network parameters are derived from probabilistic considerations. It is shown that when the input dimension is n, the proportional activation radius is /spl rho/ and the network size is 2/sup /spl nu/n/ with /spl nu/>1-h/sub 2/(/spl rho/), the equilibrium capacity is at least 2/sup /spl alpha/n//8n/spl rho/(1-/spl rho/) for any /spl alpha/>

We present a neural network employing Hebbian storage and sparse internal coding, which is capable of memorizing and correcting sequences of binary vectors by association. A ternary version of the Kanerva memory, folded into a feedback configuration, is shown to perform the basic sequence memorization and regeneration function. The inclusion of lateral connections between the internal cells increases the network capacity considerably and facilitates the correction of individual input patterns and the detection of large errors. The introduction of higher delays in the transmission lines between the external input-output layer and the internal memory layer is shown to further improve the network's error correction capability.

We present a method for analyzing the convergence properties of nonlinear dynamical systems yielding second-order bounds on the domain of attraction of an asymptotically stable equilibrium point and on the time of convergence in the estimated domain. We show that under certain conditions on the system, there exists an analytic solution to the corresponding optimization problem. The method is applied in analyzing the dynamics of a neural network model.

It is shown that the memory capacity of networks of binary neurons storing, by the Hebbian rule, sparse random vectors over the field (0, 1)/sup N/ is at least c(N/p log N), where c is a positive scalar involving input error probabilities probability of an element being nonzero. A similar bound is derived for networks of ternary neurons, storing sparse vectors over (-1,0,1)/sup N/. These results, pertaining to stability and error correction with probability tending to one as the number of neurons tends to infinity, generalize and extend previously known capacity bounds for binary networks storing vectors of equally probable (+or-1) bits. Lower bounds on the capacities of binary and ternary networks of finite sizes are also derived. These bounds suggest critical network sizes that guarantee high gains in capacity per neuron for given sparsites.>

Networks of ternary neurons storing random vectors over the set (-1,0,1) by the so-called Hebbian rule are considered. It is shown that the maximal number of stored patterns that are equilibrium states of the network with probability tending to one as N tends to infinity is at least on the order of N/sup 2-1// alpha /K, where N is the number of neurons, K is the number of nonzero elements in a pattern. and t= alpha K, 1/2>

Nested neural networks are constructed from outer products of patterns over (-1,0,1)/sup N/, whose nonzero bits define subnetworks and the subcodes stored in them. The set of permissible words, which are network-size binary patterns composed of subcode words that agree in their common bits, is characterized and their number is derived. It is shown that if the bitwise products of the subcode words are linearly independent, the permissible words are the unique global minima of the Hamiltonian associated with the network.>

Neural networks defined by outer products of vectors over (-1, 0, 1) are considered. Patterns over (-1, 0, 1) define by their outer products partially connected neural networks consisting of internally strongly connected externally weakly connected subnetworks. Subpatterns over (-1, 1) define subnetworks, and their combinations that agree in the common bits define permissible words. It is shown that the permissible words are locally stable states of the network, provided that each of the subnetworks stores mutually orthogonal subwords, or, at most, two subwords. It is also shown that when each of the subnetworks stores two mutually orthogonal binary subwords at most, the permissible words, defined as the combinations of the subwords (one corresponding to each subnetwork), that agree in their common bits are the unique ground states of the associated energy function.>

Neural networks consisting of small subnetworks interconnected in a layered hierarchy are described, and their performance as associative memories is analyzed. The networks are fractal when the subnetworks corresponding to different layers have the same geometric forms but different sizes and may be related to different spatial frequencies in the pattern field. Information is stored naturally in the form of subpatterns and retrieved in the form of their permutations. Storage of two subpatterns or of mutually orthogonal subpatterns in each of the subnetworks, which is shown to guarantee local stability at the stored subpatterns, can be readily accomplished by simple saturation and threshold mechanisms. The error-correction capability of the subnetworks in a fractal network is shown to be higher than that of disconnected subnetworks of the same sizes due to the interlayer connections.>

A lower bound on the mean square error of any reduced-order estimator for a given nonlinear process, in continuous or discrete time, is derived. The bound can be calculated from any pair of lower and upper bounds on the optimal error covariance.

It is shown that the intensity of a possibly nonstationary measurement-noise sequence associated with a linear system may be identified from the sample correlation matrix constructed from multiple data records.