This letter quantitiatively investigates the effect of a temporary refractory period or dead time in the ability of a stochastic Bernoulli processor to record subsequent pulse events, following the arrival of a pulse. These effects can arise in either the input detectors of a stochastic neural network or in subsequent processing. A transient period is observed, which increases with both the dead time and the Bernoulli probability of the dead-time free system, during which the system reaches equilibrium. Unless the Bernoulli probability is small compared to the inverse of the dead time, the mean and variance of the pulse count distributions are both appreciably reduced.

We extend, in two major ways, earlier work in which sigmoidal neural nonlinearities were implemented using stochastic counters. 1) We define the signal to noise limitations of unipolar and bipolar stochastic arithmetic and signal processing. 2) We generalize the use of stochastic counters to include neural transfer functions employed in Gaussian mixture models. The hardware advantages of (nonlinear) stochastic signal processing (SSP) may be offset by increased processing time; we quantify these issues. The ability to realize accurate Gaussian activation functions for neurons in pulsed digital networks using simple hardware with stochastic signals is also analyzed quantitatively.

Stochastic signal processing can implement gaussian activation functions for radial basis function networks, using stochastic counters. The statistics of neural inputs which control the increment and decrement operations of the counter are governed by Bernoulli distributions. The transfer functions relating the input and output pulse probabilities can closely approximate gaussian activation functions which improve with the number of states in the counter. The means and variances of these gaussian approximations can be controlled by varying the output combinational logic function of the binary counter variables.

This paper examines a number of stochastic computational elements employed in artificial neural networks, several of which are introduced for the first time, together with an analysis of their operation. We briefly include multiplication, squaring, addition, subtraction, and division circuits in both unipolar and bipolar formats, the principles of which are well-known, at least for unipolar signals. We have introduced several modifications to improve the speed of the division operation. The primary contribution of this paper, however, is in introducing several state machine-based computational elements for performing sigmoid nonlinearity mappings, linear gain, and exponentiation functions. We also describe an efficient method for the generation of, and conversion between, stochastic and deterministic binary signals. The validity of the present approach is demonstrated in a companion paper through a sample application, the recognition of noisy optical characters using soft competitive learning. Network generalization capabilities of the stochastic network maintain a squared error within 10 percent of that of a floating-point implementation for a wide range of noise levels. While the accuracy of stochastic computation may not compare favorably with more conventional binary radix-based computation, the low circuit area, power, and speed characteristics may, in certain situations, make them attractive for VLSI implementation of artificial neural networks.

For pt. I see ibid., p.891-905. An investigation has been made into the use of stochastic arithmetic to implement an artificial neural network solution to a typical pattern recognition application. Optical character recognition is performed on very noisy characters in the E-13B MICR font. The artificial neural network is composed of two layers, the first layer being a set of soft competitive learning subnetworks and the second a set of fully connected linear output neurons. The observed number of clock cycles in the stochastic case represents an order of magnitude improvement over the floating-point implementation assuming clock frequency parity. Network generalization capabilities were also compared based on the network squared error as a function of the amount of noise added to the input patterns. The stochastic network maintains a squared error within 10 percent of that of the floating-point implementation for a wide range of noise levels.

Explores some of the properties of stochastic digital signal processing in which the input signals are represented as sequences of Bernoulli events. The event statistics of the resulting stochastic process may be governed by compound binomial processes, depending upon how the individual input variables to a neural network are stochastically multiplexed. Similar doubly stochastic statistics can also result from datasets which are Bernoulli mixtures, depending upon the temporal persistence of the mixture components at the input terminals to the network. The principal contribution of these results is in determining the required integration period of the stochastic signals for a given precision in pulsed digital neural networks.

A modified adaptive resonance theory (ART2) learning algorithm, which we employ in this paper, belongs to the family of NN algorithms whose main goal is the discovery of input data clusters, without considering their actual size. This feature makes the modified ART2 algorithm very convenient for image compression tasks, particularly when dealing with images with large background areas containing few details. Moreover, due to the ability to produce hierarchical quantization (clustering), the modified ART2 algorithm is proved to significantly reduce the computation time required for coding, and therefore enhance the overall compression process. Examples of the results obtained are presented, suggesting the benefits of using this algorithm for the purpose of VQ, i.e., image compression, over the other NN learning algorithms.

The implementation of an intermediary-proxy is a common approach to the problem of network heterogeneity in the Internet infrastructure. Due to the hypertext nature of the most popular Internet application-the World Wide Web, image compression is considered to be one of the fundamental functions of such a proxy. It has been observed that most images embedded into Web documents are of 'information-delivery' type, so an algorithm intended for their compression has to satisfy some specific requirements. First, in order to support network (bandwidth) constraints for an arbitrary case, the algorithm should be inherently adaptive, i.e. able to provide a wide range of compression rates. Second, as dealing with images that are integral parts of an interactive application (such as a Web browser), the algorithm should be capable of preserving a sufficient level of image semantics according to the quality standards of human perception. The vector quantization (VQ) technique, in its general form, is proven to satisfy the first requirement. On the other hand, a modified adaptive resonance (modified ART2) learning algorithm (which we employ in this paper) more properly belongs to the family of NN algorithms whose main goal is the discovery of input data clusters, without considering their actual size. This feature makes the modified ART2 algorithm satisfy the second requirement. Thus, the discussion and results presented are intended to show that modified ART2 underlying the general VQ procedure is an appropriate techniques for image compression purposes in a bandwidth-constrained environment.

A genetic algorithm (GA) is used to search for a set of local feature detectors or hidden units. These are in turn employed as a representation of the input data for neural learning in the upper layer of a multilayer perceptron (MLP) which performs an image classification task. Three different methods of encoding hidden unit weights in the chromosome of the GA are presented, including one which coevolves all the feature detectors in a single chromosome, and two which promote the cooperation of feature detectors by encoding them in their own individual chromosomes. The fitness function measures the MLP classification accuracy together with the confidence of the networks.

Simulations indicate that the deterministic Boltzmann machine, unlike the stochastic Boltzmann machine from which it is derived, exhibits unstable behavior during contrastive Hebbian learning of nonlinear problems, including oscillation in the learning algorithm and extreme sensitivity to small weight perturbations. Although careful choice of the initial weight magnitudes, the learning rate, and the annealing schedule will produce convergence in most cases, the stability of the resulting solution depends on the parameters in a complex and generally indiscernible way. We show that this unstable behavior is the result of over parameterization (excessive freedom in the weights), which leads to continuous rather than isolated optimal weight solution sets. This allows the weights to drift without correction by the learning algorithm until the free energy landscape changes in such a way that the settling procedure employed finds a different minimum of the free energy function than it did previously and a gross output error occurs. Because all the weight sets in a continuous optimal solution set produce exactly the same network outputs, we define reliability, a measure of the robustness of the network, as a new performance criterion.

Examines the application of four competitive learning algorithms to the clustering of simple visual motion for use in the vision system of autonomous mobile robots. The arrangement and properties of the optical sensors used were loosely based on the visual apparatus of a jumping spider. It was found that competitive learning and specifically frequency sensitive competitive learning is able to learn to identify motion in an unsupervised manner. These learned visual representations can then be combined in subsequent processing stages for the development of active robotic vision systems. The unpredictability of a robot's operating environment and the inherent variations in the properties of physical sensors makes the use of adaptive clustering techniques essential. Both simulated and empirical results involving a modest robot demonstrate that novel motion and stationary position can be expressed as a combination of basic learned motion vectors.

The WWW is an online hypertextual collection, and a more sophisticated algorithm for Web page clustering may have to be based on combined term-similarity and hyperlink-similarity measures. It has been observed that nearly all currently employed techniques for document classification on the Web make use of textual information only. In addition, most of these techniques are incapable of discovering the real nature of the collection to which they are applied due to rather inefficient clustering algorithms employed. This paper describes a novel technique for hypertext clustering, called an adaptive hypertext clustering (AHC) algorithm. This algorithm has been derived from a modified neural network algorithm, and adjusted to the problem of combined term-similarity and hyperlink-similarity measures. The results presented in the paper show that AHC can be easily adapted to enable the most appropriate Web page classification within collections of various thematic and functional profiles, suggesting its main benefits over the traditional techniques.

The task of providing robust vision for autonom ous mobile robots is a complex signal processing problem which cannot be solved using traditional deterministic computing techniques.In this article we investigate four unsupervised neural lear ning algor ithms, known collectively as competitive learning, in order to assess both their theoretical operation and their ability to lear n to represent a basic robotic vision task.This task involves the ability of a modest robotic system to identify the components of basic motion and to generalize upon that lear ned knowledge to classify correctly novel visual experiences.This investigation shows that standard competitive lear ning and the DeSieno version of frequency-sensitive competitive lear ning (FSCL) are unsuitable for solving this problem.Soft competitive lear ning , while capable of producing an appropriate solution, is too computationally expensive in its present form to be used under the constraints of this application.However, the Krishnamur thy version of FSC L is found to be both computationally eý cient and capable of reliably lear ning a suitable solution to the motion identi® cation problem both in simulated tests and in actual hardware-based experiments.

This paper begins with an overview of several competitive learning algorithms in artificial neural networks, including self-organizing feature maps, focusing on properties of these algorithms important to hardware implementations. We then discuss previously reported digital implementations of these networks. Finally, we report a reconfigurable parallel neurocomputer architecture we have designed using digital signal processing chips and field-programmable gate array devices. Communications are based upon a broadcast network with FPGA-based message preprocessing and postprocessing. A small prototype of this system has been constructed and applied to competitive learning in self-organizing maps. This machine is able to model slowly-varying nonstationary data in real time.

This paper investigates neuron activation statistics in artificial neural networks employing stochastic arithmetic. It is shown that a doubly stochastic Poisson process is an appropriate model for the signals in these circuits.

The fabrication of silicon versions of artificial neural learning algorithms in existing VLSI processes introduces a variety of concerns which do not exist in a theoretical system. These include such well known circuit properties as noise, variations and nonlinearity of fabricated devices, arithmetic inaccuracy, and capacitive decay. The supervised learning algorithm-contrastive Hebbian learning, and unsupervised soft competitive learning have demonstrated their resiliency in the presence of these effects as observed in 1.2 /spl mu/m CMOS circuits employing Gilbert multipliers. It has been found that the learning circuits will operate correctly in the presence of offset errors in analog multipliers and adders, if thresholding is applied when performing weight updates.

In this paper we present results of simulations performed assuming both forward and backward computation are done on-chip using analog components. Aspects of analog hardware studied are component variability, limited voltage ranges, components (multipliers) that only approximate the computations in the backpropagation algorithm, and capacitive weight decay. It is shown that backpropagation networks can learn to compensate for all these shortcomings of analog circuits except for zero offsets, and the latter are correctable with minor circuit complications. Variability in multiplier gains is not a problem, and learning is still possible despite limited voltage ranges and function approximations. Fixed component variation from fabrication is shown to be less detrimental to learning than component variation due to noise. Weight decay is tolerable provided it is sufficiently small, which implies frequent refreshing by rehearsal on the training data or modest cooling of the circuits. The former approach allows for learning nonstationary problem sets.

Selected examples are presented of recent advances, primarily from the U.S. and Canada, in analog circuits for relaxation networks. Relaxation networks having feedback connections exhibit potentially greater computational power per neuron than feedforward networks. They are also more poorly understood especially with respect to learning algorithms. Examples are described of analog circuits for (i) supervised learning in deterministic Boltzmann machines, (ii) unsupervised competitive learning and feature maps and (iii) networks with resistive grids for vision and audition tasks. We also discuss recent progress on in-circuit learning and synaptic weight storage mechanisms.

This paper presents a tutorial review of artificial neurons and synapses which are implemented as analog CMOS circuits and which employ in situ learning of the synaptic weights. In situ learning implies that circuits local to the synapses perform computation of the weight updates according to built-in learning rules. This makes these synapses less flexible than those with external learning which do not commit to predetermined rules. On the other hand, in situ learning has potential advantages in increased learning rates in compensating for component inaccuracies and in adapting to nonstationary tasks. Both supervised and unsupervised learning rules and their implementations are described. The discussion focusses on synapses whose weights are stored on capacitors or in binary registers but also includes EEPROMs and CCDs. The emphasis is on synapses implemented using analog multipliers but other circuit techniques such as pulse streams are also briefly mentioned.

Relevant signature analysis properties for elementary one-dimensional cellular automata are presented. It is found that cellular automata with cyclic-group rules provide signature analysis properties comparable to the LFSR (linear feedback shift register). A technique of using CALBO (cellular automata-based logic block observation) for both test pattern generation and signature analysis, in a similar manner to a typical BILBO (built-in block observation) implementation, is presented.>

It has been stated by R.N. Gorgui-Naguib and R.A. King (1986) that the operations of addition and multiplication on Hensel codes originally defined by E.V. Krishnamurthy, T.M. Rao, and K. Subramanian (1975) are seriously in error in that it is possible to add/subtract or multiply Hensel codes and not get a valid Hensel code. It is shown that it is the presence of so-called invalid Farey fractions that results in the need to modify the original arithmetic operations. However, this also results in the Hensel codes becoming redundant. The authors show how to include the invalid Farey fractions such that it is possible to compute with their Hensel codings without the need to map back and forth between the rationals and their Hensel codings. This provides an alternative to the method of Gorgui-Naguib and King. Unfortunately, it turns out that Hensel codes of a large size will be needed in practice, even for relatively small problems.>

This paper provides a tutorial of various VLSI approaches to synthesizing artificial neural networks as microelectronic systems. The means by which the network learns and the synaptic weights become modified is a central theme in this study. The majority of the presentation is concerned with analog circuit approaches to neurons and synapses, employing CMOS circuits. Also included is recent work towards VLSI in situ learning circuits which implement qualitative approximations to Hebbian learning with economy of transistors. An attempt is also made to anticipate relevant developments in VLSI devices which would be suited to neural networks, just as conventional MOS transistors are well suited to traditional digital computer systems.

The authors demonstrate that one-dimensional (1-D) cellular automata (CA) form the basis of efficient VLSI architectures for computations involved in the Monte Carlo simulation of the two-dimensional (2-D) Ising model. It is shown that the time-intensive task of importance sampling the Ising configurations is expedited by the inherent parallelism in this approach. The CA architecture further provides a spatially distributed set of pseudorandom numbers that are required in the local nondeterministic decisions at the various sites in the array. The novel approach taken to random-number generation can also be applied to a variety of other highly nondeterministic algorithms from many fields, such as computational geometry, pattern recognition, and artificial intelligence.>

A novel random number generation (RNG) architecture of particular importance in VLSI for fine-grained parallel processing is proposed. It is demonstrated that efficient parallel pseudorandom sequence generation can be accomplished using certain elementary one-dimensional cellular automata (two binary states per site and only nearest-neighbor connections). The pseudorandom numbers appear in parallel from various cells in the cellular automaton on each clock cycle and pass standard empirical random number tests. Applications have been demonstrated in the design and analysis of special-purpose accelerators for Monte Carlo simulation of large intractable systems. In addition, significant advantages in pseudorandom built-in self-test of VLSI circuits using cellular automata based RNGs have been demonstrated.>

A variation on a built-in self-test technique is presented that is based on a distributed pseudorandom number generator derived from a one-dimensional cellular automata (CA) array. The cellular automata-logic-block-observation circuits presented are expected to improve upon conventional design for testability circuitry such as built-in logic-block operation as a direct consequence of reduced cross correlation between the bit streams that are used as inputs to the logic unit under test. Certain types of circuit faults are undetectable using the correlated bit streams produced by a conventional linear-feedback-shift-register (LFSR). It is also noted that CA implementations exhibit data compression properties similar to those of the LFSR and that they display locality and topological regularity, which are important attributes for a very large-scale integration implementation. It is noted that some CAs may be able to generate weighted pseudorandom test patterns. It is also possible that some of the analysis of pseudorandom testing may be more directly applicable to CA-based pseudorandom testing than to LFSR-based schemes.>

The computational complexity measures introduced here are motivated by the trend to higher VLSI integration levels (rather than increased chip area) to accomplish solutions to larger problem instances. It seems that the increase in the computational power of VLSI circuits can be mainly attributed to the reduction in the minimum feature size rather than to an increase in the chip area. In view of this, we present a constant area perspective and consider the discrete Fourier transform and related problems in a VLSI model that has λand T as its resources. Advantages of the mesh algorithm over the shuffle-exchange algorithm in the computation time for the DFT are shown to arise from an upper bound on current density in the wires, which we suggest must be considered in any VLSI grid model.

This correspondence presents a linear systolic array for the implementation pf digital signal processing systems based upon matrix- vector multiplication algorithms where the matrix elements can be computed from their row and column indexes. Haar, Walsh, and the discrete Fourier transforms are solved using this approach. The method presented enables the n2 matrix elements to be computed in situ directly from the 2n matrix indexes. Thus, performance comparable to known systolic matrix-vector multipliers is achieved using only constant I/O bandwidth, rather than O(n) bandwidth required in the more general case. A generalized method is given for the development of recursively formed matrices and specifically the VLSI implementation of the Haar and Walsh transforms.

The study of one-dimensional cellular automata exhibiting group properties is presented. The results show that only a certain class of cellular automata rules exhibit group characteristics based on rule multiplication. However, many other of these automata reveal groups based on permutations of their global states. It is further shown how these groups may be utilized in the design of modulo arithmetic units. The communication properties of cellular automata are observed to map favorably to optimal communication graphs for VLSI layouts. They exploit the implementation medium and properly address the physical limits on computational structures. Comparisons of cellular automata-based modulo arithmetic units with other VLSI algorithms are presented using area-time complexity measures.