This paper presents a framework to investigate the potential impact of time-dependent variability at future technology nodes. Both static statistical variability and NBTI-induced device degradation have been integrated to represent the time-dependent variability, and the impact on the performance of an ISCAS benchmark circuit in sub-35nm technologies has been studied. The BSIM4 compact models of MOSFET at 25, 18 and 13nm nodes are calibrated by a 3D atomistic device simulator with chip measurements of 35nm gate length devices. Synthesis results confirm that the variability of circuit performance will increase as device scaling continues, and can be more severe in new circuits at 18nm than in those stressed for a period of three years at 35nm. In addition, the results also reveal that increasing power consumption as adopted in adaptive supply voltage (ASV) and adaptive back bias (ABB) schemes is not a sustainable solution to compensate the drift in performance for future generations of CMOS circuits and systems.

Adaptation is one of the most important phenomena in biology. A young barn owl can adapt to imposed environmental changes, such as artificial visual distortion caused by wearing a prism. This adjustment process has been modeled mathematically and the model replicates the sensory map realignment of barn owl superior colliculus (SC) through axonogenesis and synaptogenesis. This allows the biological mechanism to be transferred to an artificial computing system and thereby imbue it with a new form of adaptability to the environment. The model is demonstrated in a real-time robot environment. Results of the experiments are compared with and without prism distortion of vision, and show improved adaptability for the robot. However, the computation speed of the embedded system in the robot is slow. A digital and analog mixed signal very-large-scale integration (VLSI) circuit has been fabricated to implement adaptive sensory pathway changes derived from the SC model at higher speed. VLSI experimental results are consistent with simulation results.

Animals such as stick insects can adaptively walk on complex terrains by dynamically adjusting their stepping motion patterns. Inspired by the coupled Matsuoka and resonate-and-fire neuron models, we present a nonlinear oscillation model as the neuromorphic central pattern generator (CPG) for rhythmic stepping pattern generation. This dynamic model can also be used to actuate the motoneurons on a leg joint with adjustable driving frequencies and duty cycles by changing a few of the model parameters while operating such that different stepping patterns can be generated. A novel mixed-signal integrated circuit design of this dynamic model is subsequently implemented, which, although simplified, shares the equivalent output performance in terms of the adjustable frequency and duty cycle. Three identical CPG models being used to drive three joints can make an arthropod leg of three degrees of freedom. With appropriate initial circuit parameter settings, and thus suitable phase lags among joints, the leg is expected to walk on a complex terrain with adaptive steps. The adaptation is associated with the circuit parameters mediated both by the higher level nervous system and the lower level sensory signals. The model is realized using a 0.3- complementary metal-oxide-semiconductor process and the results are reported.

Poisson distributed spike trains are often used as the input to VLSI implementations of spiking neural networks. However, it can be difficult to generate large truly random spike distributions which can be easily applied as input to a chip. This work presents results recorded from an avalanche photo diode which demonstrates that it can be used to create a Poisson distributed spike train and describes the circuitry which will allow it to interface with other neuromorphic chips using the Address Event Representation protocol. The chip is currently being fabricated using the AMS 0.35μm HV process.

Static statistical variability and time-dependent reliability are traditionally analyzed separately. This paper presents a new methodology which combines both types of variability within a single circuit analysis framework. A comprehensive Negative Bias Temperature Instability (NBTI) model was implemented. Effects of random discrete dopants, line edge roughness and poly-Silicon granularity were considered. Using a 74X-series benchmark circuit (4-bit fast-carry adder) as an example, the concept of integrating both static statistical variability and time-dependent reliability into circuit analysis is demonstrated.

A distributed and locally reprogrammable address-event receiver has been designed, in which incoming address-events are monitored simultaneously by all synapses, allowing for arbitrarily large axonal fan-out without reducing channel capacity. Synapses can change the address of their presynaptic neuron, allowing the distributed implementation of a biologically realistic learning rule, with both synapse formation and elimination (synaptic rewiring). Probabilistic synapse formation leads to topographic map development, made possible by a cross-chip current-mode calculation of Euclidean distance. As well as synaptic plasticity in rewiring, synapses change weights using a competitive Hebbian learning rule (spike-timing-dependent plasticity). The weight plasticity allows receptive fields to be modified based on spatio-temporal correlations in the inputs, and the rewiring plasticity allows these modifications to become embedded in the network topology.

Visual and auditory map alignment in the superior colliculus (SC) of the barn owl is important for its accurate localization of prey. The visual map, and hence the alignment, may be purposefully disturbed in a juvenile barn owl by fitting it with ocular prisms, and it is found that it can adapt its auditory map to this mismatch after several weeks training. In our previous SC model, the axon growing process is instructed by an inhibitory network, the strength of which is adjusted using the neural structures involved in spatial localization. Based on this model, a mixed signal integrated circuit of the SC has been designed, and simulation results are consistent with those found by biological experiment. This new model makes possible artificial networks capable of eliminating the disparity between the visual and auditory maps.

A new methodology to include random telegraph signals (RTS) noise in circuit analysis is proposed. The aim of this methodology is to allow integrated circuit designers to study the sensitivity of their circuits to RTS noise and thus minimise the impact of it. In this work, compact models extracted from three-dimensional dasiaatomisticpsila simulations based on random doping were used. These models define 35 nm CMOS technology devices with single charge trapping at the Si-SiO2interface, and therefore the amplitude of RTS noise. The timing parameters of RTS noise were predicted based on the Shockley-Reed-Hall statistics. The methodology was applied to a test circuit, four-quadrant Chible multiplier as an example, under both steady-state and time-varying bias conditions. Simulation results on variability in devices (based on Monte Carlo methods) and temperature sweep are also reported.

We present a simple neuromorphic central pattern generators (CPG) circuit module, which is essentially a pair of coupled oscillators, to actuate a joint on a leg. This novel, reconfigurable CPG module is able to generate different motor patterns of different frequencies or duty cycles, simply by changing a few of circuit parameters. Three CPG modules, corresponding to three joints, can make an arthropod leg of three degrees of freedom (DOFs). With appropriate circuit parameter settings, and thus suitable phase lags among joints, the leg is expected to walk on a complex terrain with adaptive steps. The adaptation is associated with the circuit parameters mediated by external commands or sensory signals. Simulation results for the circuitry, designed using a 0.35 mum process, are reported.

This paper compares three common evolutionary algorithms and our modified GA, a Distributed Adaptive Genetic Algorithm (DAGA). The optimal approach is sought to adapt, in near real-time, biological model behaviour to that of real biology within a laboratory. Near real-time adaptation is achieved with a Graphics Processing Unit (GPU). This, together with evolutionary computation, enables new forms of experimentation such as online testing, where biology and computational model are simultaneously stimulated and their responses compared. Rapid analysis and validation provide a platform that is required for rapid prototyping, and along with online testing, can provide new insight into the cause of biological behaviour. In this context, results demonstrate that our DAGA implementation is more efficient than the other three evolutionary algorithms due to its suitability to the adaptation environment, namely the large population sizes promoted by the GPU architecture.

We have designed a distributed and locally reprogrammable address event receiver. Incoming address-events are monitored simultaneously by all synapses, allowing for arbitrarily large axonal fan-out without reducing channel capacity. Synapses can change input address, allowing neurons to implement a biologically realistic learning rule locally, with both synapse formation and elimination.

To localize a seen object, the superior colliculus of the barn owl integrates the visual and auditory localization cues which are accessed from the sensory system of the brain. These cues are formed as visual and auditory maps, thus the alignment between visual and auditory maps is very important for accurate localization in prey behavior. Blindness or prism wearing may disturb this alignment. The juvenile barn owl could adapt its auditory map to this mismatch after several weeks training. Here we investigate this process by building a computational model of auditory and visual integration with map adjustment in the deep superior colliculus. The adaptation is based on activity dependent axon developing which is instructed by an inhibitory network. In the inhibitory network, the strength of the inhibition is adjusted by spike timing dependent plasticity(STDP). The simulation results are in line with the biological experiment and support the idea that the STDP is involved in the alignment of sensory maps. The system of the model provides a new mechanism capable of eliminating the disparity in visual and auditory map integration.

The visual and auditory map alignment in the Superior Colliculus (SC) of barn owl is important for its accurate localization for prey behavior. Prism learning or Blindness may interfere this alignment and cause loss of the capability of accurate prey. However, juvenile barn owl could recover its sensory map alignment by shifting its auditory map. The adaptation of this map alignment is believed based on activity dependent axon developing in Inferior Colliculus (IC). A model is built to explore this mechanism. In this model, axon growing process is instructed by an inhibitory network in SC while the strength of the inhibition adjusted by Spike Timing Dependent Plasticity (STDP). We test and analyze this mechanism by application of the neural structures involved in spatial localization in a robotic system.

This paper investigates whether spike-timing-dependent plasticity (STDP) can minimize the effect of mismatch within the context of a depth-from-motion algorithm. To improve noise rejection, this algorithm contains a spike prediction element, whose performance is degraded by analog very large scale integration (VLSI) mismatch. The error between the actual spike arrival time and the prediction is used as the input to an STDP circuit, to improve future predictions. Before STDP adaptation, the error reflects the degree of mismatch within the prediction circuitry. After STDP adaptation, the error indicates to what extent the adaptive circuitry can minimize the effect of transistor mismatch. The circuitry is tested with static and varying prediction times and chip results are presented. The effect of noisy spikes is also investigated. Under all conditions the STDP adaptation is shown to improve performance.

This paper presents some circuitry for use within a visual-processing depth-recovery algorithm based upon spike timing. The accuracy of the depth calculation relies on a prediction which, when implemented in analogue VLSI, will be degraded by transistor mismatch. An adaptive circuit based on spike timing dependent plasticity (STDP) was designed to reduce the effects of VLSI process variations on the algorithm's performance. Simulation results for the circuitry, designed using a 0.35mum process, are reported

Recent years have seen the rapid development of microsensor technology, system on chip design, wireless technology and ubiquitous computing. When assembled into a complex microsystem the technologies become powerful tools in medical diagnostics, environmental monitoring and personal connectivity. In this paper we describe the demonstration of a silicon chip that has all the attributes required of a microsystem for use in these applications. The design methodology we have employed is a variant of the system on chip approach whereby many intellectual property blocks are integrated at a high level in the design flow. Our intellectual property blocks include the analogue sensor instrumentation for temperature and pH, a data multiplexing and conversion module, a digital platform based around an 8-bit microcontroller, data encoding for spread-spectrum wireless transmission and a RF section requiring very few off-chip components. The chip has been fully evaluated and tested by connection to external sensors. Each block has well defined interfaces so that they can be easily reused in future designs targeted to different applications

A novel depth-from-motion vision model based on leaky integrate-and-fire (I&F) neurons incorporates the implications of recent neurophysiological findings into an algorithm for object discovery and depth analysis. Pulse-coupled I&F neurons capture the edges in an optical flow field and the associated time of travel of those edges is encoded as the neuron parameters, mainly the time constant of the membrane potential and synaptic weight. Correlations between spikes and their timing thus code depth in the visual field. Neurons have multiple output synapses connecting to neighbouring neurons with an initial Gaussian weight distribution. A temporally asymmetric learning rule is used to adapt the synaptic weights online, during which competitive behaviour emerges between the different input synapses of a neuron. It is shown that the competition mechanism can further improve the model performance. After training, the weights of synapses sourced from a neuron do not display a Gaussian distribution, having adapted to encode features of the scenes to which they have been exposed.

This paper presents the VLSI implementation of the continuous restricted Boltzmann machine (CRBM), a probabilistic generative model that is able to model continuous-valued data with a simple and hardware-amenable training algorithm. The full CRBM system consists of stochastic neurons whose continuous-valued probabilistic behavior is mediated by injected noise. Integrating on-chip training circuits, the full CRBM system provides a platform for exploring computation with continuous-valued probabilistic behavior in VLSI. The VLSI CRBM's ability both to model and to regenerate continuous-valued data distributions is examined and limitations on its performance are highlighted and discussed.

We propose a simplified depth-from-motion vision model based on leaky integrate-and-fire (LIF) neurons for edge detection and two-dimensional depth recovery. In the model, every LIF neuron is able to detect the irradiance edges passing through its receptive field in an optical flow field, and respond to the detection by firing a spike when the neuron's firing criterion is satisfied. If a neuron fires a spike, the time-of-travel of the spike-associated edge is transferred as the prediction information to the next synapse-linked neuron to determine its state. Correlations between input spikes and their timing thus encode depth in the visual field. The adaptation of synapses mediated by spike-timing-dependent plasticity is used to improve the algorithm's robustness against inaccuracy caused by spurious edge propagation. The algorithm is characterized on both artificial and real image sequences. The implementation of the algorithm in analog very large scale integrated (aVLSI) circuitry is also discussed.

This paper shows that the lamprey's neural swimming controller (central pattern generator (CPG)) is not a unique solution and that improved performance can be obtained by evolving the CPG's neural parameters and connection weights. Propulsion in the lamprey, an eel-like fish, is governed by activity in its spinal neural network. This CPG is simulated, in accordance with Ekeberg's model, and then evolved using genetic algorithm techniques to explore the domain of alternative configurations. Results indicate several suitable controllers exist. The best controller from forty experiments produces a controllable frequency range of 1.42 - 12.16 Hz, a substantial increase on the biological network frequency range (1.74 - 5.56 Hz). Internal connectivity of the evolved network is 16 connections, lower than the 26 connections of Ekeberg's model. Thus, evolved networks can operate over a wider frequency range (than their biological prototype and controllers evolving weights alone) whilst maintaining low system complexity. Evolving advanced yet simple controllers provides solutions which are more attainable in silicon (VLSI), improves performance for task specific control and determines the extent to which nature's solutions are unique.

A transient-detecting very large scale integration (VLSI) pixel is described, suitable for use in a visual-processing, depth-recovery algorithm based upon spike timing. A small array of pixels is coupled to an adaptive system, based upon spike timing dependent plasticity (STDP), that aims to reduce the effect of VLSI process variations on the algorithm's performance. Results from 0.35 microm CMOS temporal differentiating pixels and STDP circuits show that the system is capable of adapting to substantially reduce the effects of process variations without interrupting the algorithm's natural processes. The concept is generic to all spike timing driven processing algorithms in a VLSI.

We have mapped the contrastive divergence learning scheme of the product of experts (PoE) onto electrical circuits. The issues raised during that hardware translation are discussed in This work and some circuits presenting our solutions are described. The entire learning rule is implemented in mixed-signal VLSI on a 0.6 /spl mu/m CMOS process. Chips results validating our approach and methodology are also presented.

Spike-timing dependent synaptic plasticity (STDP) is a form of plasticity driven by precise spike-timing differences between presynaptic and postsynaptic spikes. Thus, the learning rules underlying STDP are suitable for learning neuronal temporal phenomena such as spike-timing synchrony. It is well known that weight-independent STDP creates unstable learning processes resulting in balanced bimodal weight distributions. In this paper, we present a neuromorphic analog very large scale integration (VLSI) circuit that contains a feedforward network of silicon neurons with STDP synapses. The learning rule implemented can be tuned to have a moderate level of weight dependence. This helps stabilise the learning process and still generates binary weight distributions. From on-chip learning experiments we show that the chip can detect and amplify hierarchical spike-timing synchrony structures embedded in noisy spike trains. The weight distributions of the network emerging from learning are bimodal.

We present test results from spike-timing correlation learning ex- periments carried out with silicon neurons with STDP (Spike Tim- ing Dependent Plasticity) synapses. The weight change scheme of the STDP synapses can be set to either weight-independent or weight-dependent mode. We present results that characterise the learning window implemented for both modes of operation. When presented with spike trains with di(cid:11)erent types of synchronisation the neurons develop bimodal weight distributions. We also show that a 2-layered network of silicon spiking neurons with STDP synapses can perform hierarchical synchrony detection.

This paper presents VLSI circuits with continuous-valued proba- bilistic behaviour realized by injecting noise into each computing unit(neuron). Interconnecting the noisy neurons forms a Contin- uous Restricted Boltzmann Machine (CRBM), which has shown promising performance in modelling and classifying noisy biomed- ical data. The Minimising-Contrastive-Divergence learning algo- rithm for CRBM is also implemented in mixed-mode VLSI, to adapt the noisy neurons’ parameters on-chip.

The paper presents Bayesian information fusion theory in the context of neural-network model combination. It shows how confidence measures can be combined with individual model estimates to minimize risk through the fusion process. The theory is illustrated through application to the real task of quality prediction in the papermaking industry. Prediction uncertainty estimates are calculated using approximate Bayesian learning. These are incorporated into model combination as confidence measures. Cost functions in the fusion center are used to control the influence of the confidence measures and improve the performance of the resultant committee.

Experimental data has shown that synaptic strength modification in some types of biological neurons depends upon precise spike tim(cid:173) ing differences between presynaptic and postsynaptic spikes. Sev(cid:173) eral temporally-asymmetric Hebbian learning rules motivated by this data have been proposed. We argue that such learning rules are suitable to analog VLSI implementation. We describe an eas(cid:173) ily tunable circuit to modify the weight of a silicon spiking neuron according to those learning rules. Test results from the fabrication of the circuit using a O.6J.lm CMOS process are given.

The history and some of the methods of analogue neural VLSI are described. The strengths of analogue techniques are described, along with residual problems to be solved. The nature of hardware-friendly and hardware-appropriate algorithms is reviewed and suggestions are offered as to where analogue neural VLSI's future lies.

Feedforward neural networks, particularly multilayer perceptrons, are widely used in regression and classification tasks. A reliable and practical measure of prediction confidence is essential. In this work three alternative approaches to prediction confidence estimation are presented and compared. The three methods are the maximum likelihood, approximate Bayesian, and the bootstrap technique. We consider prediction uncertainty owing to both data noise and model parameter misspecification. The methods are tested on a number of controlled artificial problems and a real, industrial regression application, the prediction of paper "curl". Confidence estimation performance is assessed by calculating the mean and standard deviation of the prediction interval coverage probability. We show that treating data noise variance as a function of the inputs is appropriate for the curl prediction task. Moreover, we show that the mean coverage probability can only gauge confidence estimation performance as an average over the input space, i.e., global performance and that the standard deviation of the coverage is unreliable as a measure of local performance. The approximate Bayesian approach is found to perform better in terms of global performance.

This paper addresses the issues of neural network model development and maintenance in the context of a complex task taken from the papermaking industry. In particular, it describes a comparison study of early stopping techniques and model selection, both to optimise neural network models for generalisation performance. The results presented here show that early stopping via use of a Bayesian model evidence measure is a viable way of optimising performance while also making maximum use of all the data. In addition, they show that ten-fold cross-validation performs well as a model selector and as an estimator of prediction accuracy. These results are important in that they show how neural network models may be optimally trained and selected for highly complex industrial tasks where the data are noisy and limited in number.

In this paper, different methods for training radial basis function (RBF) networks for regression problems are described and illustrated. Then, using data from the DELVE archive, they are empirically compared with each other and with some other well known methods for machine learning. Each of the RBF methods performs well on at least one DELVE task, but none are as consistent as the best of the other non-RBF methods.

We describe a method for non-parametric regression which combines regression trees with radial basis function networks. The method is similar to that of Kubat, who was first to suggest such a combination, but has some significant improvements. We demonstrate the features of the new method, compare its performance with other methods on DELVE data sets and apply it to a real world problem involving the classification of soybean plants from digital images.

This paper describes the application of neural network techniques to the papermaking industry, particularly for the prediction of paper "curl." Paper curl is an important quality measure that can only be measured reliably off-line after manufacture, making it difficult to control. Here we predict, before paper manufacture from characteristics of the current reel, whether the paper curl will be acceptable and the level of curl. For both the case of predicting the probability that paper will be "out-of-specification" and that of predicting the level of curl, we include confidence intervals indicating to the machine operator whether the predictions should be trusted. The results and the associated discussion describe a successful application of neural networks to a difficult, but important, real-world task taken from the papermaking industry. In addition the techniques described are widely applicable to industry where direct prediction of a quality measure and its acceptability are desirable, with a clear indication of prediction confidence.

This article introduces the concept of optimally distributed computation in feedforward neural networks via regularization of weight saliency. By constraining the relative importance of the parameters, computation can be distributed thinly and evenly throughout the network. We propose that this will have beneficial effects on fault-tolerance performance and generalization ability in large network architectures. These theoretical predictions are verified by simulation experiments on two problems: one artificial and the other a real-world task. In summary, this article presents regularization terms for distributing neural computation optimally.

In recent years, the efforts of analogue, neural-hardware designers have shifted from generic analogue neurocomputers to "niche" markets in sensor fusion and robotics, and we explain why this is so. We describe the main differences between digital and analogue computation, and consider the advantages of pure analogue and pulsed methods of design. We then investigate some important issues in analogue design of neural machines, namely weight storage (volatile and non-volatile), on-chip learning, and arithmetic accuracy and its relationship to noise. Finally, we outline those areas in which analogue techniques are likely to prove most useful, and speculate as to their likely long-term utility.

This paper investigates fault tolerance in feedforward neural networks, for a realistic fault model based on analog hardware. In our previous work with synaptic weight noise we showed significant fault tolerance enhancement over standard training algorithms. We proposed that when introduced into training, weight noise distributes the network computation more evenly across the weights and thus enhances fault tolerance. Here we compare those results with an approximation to the mechanisms induced by stochastic weight noise, incorporated into training deterministically via penalty terms. The penalty terms are an approximation to weight saliency and therefore, in addition, we assess a number of other weight saliency measures and perform comparison experiments. The results show that the first term approximation is an incomplete model of weight noise in terms of fault tolerance. Also the error Hessian is shown to be the most accurate measure of weight saliency.

Analogue hardware implementations of neural networks have hitherto been hampered by the lack of a straightforward analogue memory capability. The synaptic weights which are developed by the network learning process must be stored (preferably at each synapse site) in order that a network can adequately perform any recall or classification tasks. Ideally, the storage mechanism should be compact, non-volatile, easily reprogrammable, and simple to implement. Techniques which have been used to date include resistors (these are not generally reprogrammable, and suffer from being large and difficult to fabricate with any accuracy), dynamic capacitive storage (this is compact, reprogrammable and simple, but implies an increase in system complexity, arising from off-chip refresh circuitry), EEPROM ("floating gate") memory (which is compact, reprogrammable, and non-volatile, but cannot be reprogrammed in situ), and local digital storage (which is non-volatile, easily programmable and simple, but is very costly in "real estate" terms). In this paper, we demonstrate that novel amorphous silicon memory devices can be incorporated into standard CMOS synapse circuits, to provide an analogue weight storage mechanism which is compact, non-volatile, easily reprogrammable, and simple to implement.>

This paper presents for the first time a new chip, EPSILON II, which improves the electrical characteristics, interface flexibility and general architecture of the original EPSILON device. A system level framework is presented, built around the EPSILON II device, which allows the development of pulse stream neural systems for real-world applications. As a "first instance" example to demonstrate this an autonomous "instinct" driven robot is under construction.>

We explain the advantages of the Virtual-Targets algorithm for learning in a VLSI Multilayer Perceptron, outline the design issues, and describe fabricated circuits that implement the main algorithmic functions. The test-chip described is a precursor to a larger device incorporating full on-chip learning.>

This paper presents results from the first use of neural networks for the real-time feedback control of high temperature plasmas in a tokamak fusion experiment. The tokamak is currently the prin(cid:173) cipal experimental device for research into the magnetic confine(cid:173) ment approach to controlled fusion. In the tokamak, hydrogen plasmas, at temperatures of up to 100 Million K, are confined by strong magnetic fields. Accurate control of the position and shape of the plasma boundary requires real-time feedback control of the magnetic field structure on a time-scale of a few tens of mi(cid:173) croseconds. Software simulations have demonstrated that a neural network approach can give significantly better performance than the linear technique currently used on most tokamak experiments. The practical application of the neural network approach requires high-speed hardware, for which a fully parallel implementation of the multilayer perceptron, using a hybrid of digital and analogue technology, has been developed.

We analyze the effects of analog noise on the synaptic arithmetic during multilayer perceptron training, by expanding the cost function to include noise-mediated terms. Predictions are made in the light of these calculations that suggest that fault tolerance, training quality and training trajectory should be improved by such noise-injection. Extensive simulation experiments on two distinct classification problems substantiate the claims. The results appear to be perfectly general for all training schemes where weights are adjusted incrementally, and have wide-ranging implications for all applications, particularly those involving "inaccurate" analog neural VLSI.

We analyse the effects of analogue noise on the synaptic arithmetic during multilayer perceptron training by expanding the cost function to include noise-mediated penalty terms. Predictions are made in the light of these calculations which suggest that fault tolerance, generalisation ability and learning trajectory should be improved by such noise-injection. Extensive simulation experiments on two distinct classification problems substantiate the claims. The results appear to be perfectly general for all training schemes where weights are adjusted incrementally, and have wide-ranging implications for all applications, particularly those involving "inaccurate" analogue neural VLSI.

An analogue CMOS VLSI neural processing chip has been designed and fabricated. The device employs "pulse-stream" neural state signalling and is capable of computing some 360 million synaptic connections per second. In addition to basic characterisation results, the performance of the chip in solving "real-world" problems is also demonstrated. The experience gained from the development of this device has resulted in the design of a second "pulse-stream" chip with improved performance and features. It is anticipated that this second device will be integrated into a standard bus-based system and find early application in robotic control.

The authors develop a mathematical model of the effects of synaptic arithmetic noise in multilayer perceptron training. Predictions are made regarding enhanced fault-tolerance and generalization ability and improved learning trajectory. These predictions are subsequently verified by simulation. The results are perfectly general and have profound implications for the accuracy requirements in multilayer perceptron (MLP) training, particularly in the analog domain.

1 Requirement This paper describes an approach to multilayer perceptron (MLP) learning that is optimized for hardware implementation. Experimental results to date are promising, and it is the express aim of this paper to present these results concisely - a detailed mathematical analysis will follow in a subsequent, longer publication. Error backpropagation (Rumelhart et al. 1986) has achieved remarkable success as an algorithm for solving hard classification problems with MLP networks. It is, however, not readily amenable to VLSI integration, and the distinction it draws between hidden and output nodes renders it hostile to analog circuit forms. Use of the mathematical chain rule, to calculate the effect of a weight connecting to a hidden unit on the errors {Q} in the output layer, renders the error calculation scheme for hidden units different from, and more complicated than that for output units. The Virtual Targets learning scheme circumvents this problem by introducing an explicit desired state, or target for each of the hidden units, which is updated continuously, and stored along with the synapse weights. While this means that a target state must be stored for each input pattern and hidden node, it simplifies and renders homogeneous the process of weight evolution for all neurons. Furthermore, since a target state is already stored for each output neuron, the scheme essentially removes the distinction during learning between hidden and output nodes. Analog integrated circuits based on the virtual targets strategy will therefore be flexible in architectural terms, as all units will be configurable as either output or hidden layer neurons. The fundamental idea of adapting the internal representation, either as well as, or instead of the weights, is not itself new (Rohwer 1990; Grossman et al. 1990; Krogh et al. 1990). However, these pieces of work were not optimized for hardware implementation. The fundamental difference is that simplicity of implementation has been made the primary goal in the work described in this paper, to produce a system optimized for analog VLSI. There are also several important differences in detail between the work described in this paper and these earlier, similar Neural Computation 4, 366-381 (1992) @ 1992 Massachusetts Institute of Technology

Results from working analog VLSI implementations of two different pulse stream neural network forms are reported. The circuits are rendered relatively invariant to processing variations, and the problem of cascadability of synapses to form large systems is addressed. A strategy for interchip communication of large numbers of neural states has been implemented in silicon and results are presented. The circuits demonstrated confront many of the issues that blight massively parallel analog systems, and offer solutions.

The pulse-stream technique, which represents neural states as sequences of pulses, is reviewed. Several general issues are raised, and generic methods appraised, for pulsed encoding, arithmetic, and intercommunication schemes. Two contrasting synapse designs are presented and compared. The first is based on a fully analog computational form in which the only digital component is the signaling mechanism itself-asynchronous, pulse-rate encoded digital voltage pulses. In this circuit, multiplication occurs in the voltage/current domain. The second design uses more conventional digital memory for weight storage, with synapse circuits based on pulse stretching. Integrated circuits implementing up to 15000 analog, fully programmable synaptic connections are described. A demonstrator project is described in which a small robot localization network is implemented using asynchronous, analog, pulse-stream devices.