We present a novel 3D spike sorting network (3DSS) that detects a spike sequence efficiently and memorizes it upon a single presentation without configuration.We analyze the wiring and switches of alternatives and show that 3DSS reduces energy per spike quadratically compared to existing 2D networks.Applications include large-scale document retrieval and self-configuring hardware.

Neural circuits consist of many noisy, slow components, with individual neurons subject to ion channel noise, axonal propagation delays, and unreliable and slow synaptic transmission. This raises a fundamental question: how can reliable computation emerge from such unreliable components? A classic strategy is to simply average over a population of N weakly-coupled neurons to achieve errors that scale as [Formula: see text]. But more interestingly, recent work has introduced networks of leaky integrate-and-fire (LIF) neurons that achieve coding errors that scale superclassically as 1/N by combining the principles of predictive coding and fast and tight inhibitory-excitatory balance. However, spike transmission delays preclude such fast inhibition, and computational studies have observed that such delays can cause pathological synchronization that in turn destroys superclassical coding performance. Intriguingly, it has also been observed in simulations that noise can actually improve coding performance, and that there exists some optimal level of noise that minimizes coding error. However, we lack a quantitative theory that describes this fascinating interplay between delays, noise and neural coding performance in spiking networks. In this work, we elucidate the mechanisms underpinning this beneficial role of noise by deriving analytical expressions for coding error as a function of spike propagation delay and noise levels in predictive coding tight-balance networks of LIF neurons. Furthermore, we compute the minimal coding error and the associated optimal noise level, finding that they grow as power-laws with the delay. Our analysis reveals quantitatively how optimal levels of noise can rescue neural coding performance in spiking neural networks with delays by preventing the build up of pathological synchrony without overwhelming the overall spiking dynamics. This analysis can serve as a foundation for the further study of precise computation in the presence of noise and delays in efficient spiking neural circuits.

We present Polynomial in Temperature (PinT) decode weights, a novel approach to approximating functions with an ensemble of silicon neurons that increases thermal robustness. In mixed-signal neuromorphics, computing accurately across a wide range of temperatures is challenging because of individual silicon neurons' thermal sensitivity. To compensate for the resulting changes in the neuron's tuning-curves in the PinT framework, weights change continuously as a polynomial function of temperature. We validate PinT across a 38°C range by applying it to tuning curves measured for ensembles of 64 to 1936 neurons on Braindrop, a mixed-signal neuromorphic chip fabricated in 28-nm FDSOI CMOS. LinT, the Linear in Temperature version of PinT, reduces error by a small margin on test data, relative to an ensemble with temperature-independent weights. LinT and higher-order models show much greater promise on training data, suggesting that performance can be further improved. When implemented on-chip, LinT's performance is very similar to the performance with temperature-independent decode weights. SpLinT and SpLSAT, the Sparse variants of LinT and LSAT, are promising avenues for efficiently reducing error. In the SpLSAT model, up to 90% of neurons on chip can be deactivated while maintaining the same function-approximation error.

Braindrop is the first neuromorphic system designed to be programmed at a high level of abstraction. Previous neuromorphic systems were programmed at the neurosynaptic level and required expert knowledge of the hardware to use. In stark contrast, Braindrop's computations are specified as coupled nonlinear dynamical systems and synthesized to the hardware by an automated procedure. This procedure not only leverages Braindrop's fabric of subthreshold analog circuits as dynamic computational primitives but also compensates for their mismatched and temperature-sensitive responses at the network level. Thus, a clean abstraction is presented to the user. Fabricated in a 28-nm FDSOI process, Braindrop integrates 4096 neurons in 0.65 mm2. Two innovations-sparse encoding through analog spatial convolution and weighted spike-rate summation though digital accumulative thinning-cut digital traffic drastically, reducing the energy Braindrop consumes per equivalent synaptic operation to 381 fJ for typical network configurations.

Demonstration Setup: We will bring Braindrop, a mixed-signal neuromorphic chip that is configured to perform arbitrary computations using the Neural Engineering Framework (NEF). Fabricated in a 28-nm FDSOI process, Braindrop has 4,096 silicon neurons whose design we optimized for nonlinear function approximation (Fig. 1). Our optimization procedure consists of a pre-fabrication phase and a run-time phase. In the pre-fabrication phase, transistors are sized to introduce an intermediate amount of heterogeneity into the neurons' tuning curves: Not so little that spiking-thresholds bunch up in the middle of the function's domain and not so much that spiking-thresholds mostly fall outside the function's domain. In the run-time phase, the outliers-neurons that never spike or always spike-are rescued by adjusting programmable bias currents appropriately. We explored various choices of the number of programmable bias-current levels and the amount of transistor-mismatch during the design phase to determine the combination that yielded the highest number of good neurons. To facilitate design-space exploration, we developed a SPICE-derived compact model of the dependence of tuning-curve heterogeneity on transistor-mismatch.

Silicon neurons designed using subthreshold analog-circuit techniques offer low power and compact area but are exponentially sensitive to threshold-voltage mismatch in transistors. The resulting heterogeneity in the neurons' responses, however, provides a diverse set of basis functions for smooth nonlinear function approximation. For low-order polynomials, neuron spiking thresholds ought to be distributed uniformly across the function's domain. This uniform distribution is difficult to achieve solely by sizing transistors to titrate mismatch. With too much mismatch, many neuron's thresholds fall outside the domain (i.e. they either always spike or remain silent). With too little mismatch, all their thresholds bunch up in the middle of the domain. Here, we present a silicon-neuron design methodology that minimizes overall area by optimizing transistor sizes in concert with a few locally-stored programmable bits to adjust each neuron's offset (and gain). We validated this methodology in a 28-nm mixed analog-digital CMOS process. Compared to relying on mismatch alone, augmentation with digital correction effectively reduced silicon area by 38%.

The vast majority of optimization and online learning algorithms today require some prior information about the data (often in the form of bounds on gradients or on the optimal parameter value). When this information is not available, these algorithms require laborious manual tuning of various hyperparameters, motivating the search for algorithms that can adapt to the data with no prior information. We describe a frontier of new lower bounds on the performance of such algorithms, reflecting a tradeoff between a term that depends on the optimal parameter value and a term that depends on the gradients’ rate of growth. Further, we construct a family of algorithms whose performance matches any desired point on this frontier, which no previous algorithm reaches.

We present an approach to mapping synaptic models onto neuromorphic hardware using Time-Encoding and Decoding Machines. This framework allows us to transform a spiking soma-circuit into an analog-to-digital converter (ADC), one for each synapse circuit. We verify that the measurements obtained with these `virtual' ADCs closely match those obtained with an on-chip ADC. Using the massively parallel measurement capability that these virtual ADCs afford us, we demonstrate the first large-scale calibration of synapse circuits' dynamic parameters. This advance opens the door to programming neuromorphic chips on the more intuitive level of dimensionless models, rather than by setting raw voltage biases, as is currently done.

We present a novel approach to achieving temperature-robust behavior in neuromorphic systems that operates at the population level, trading an increase in silicon-neuron count for robustness across temperature. Our silicon neurons' tuning curves were highly sensitive to temperature, which could be decoded from a 400-neuron population with a precision of 0.07° C. We overcame this temperature-sensitivity by combining methods from robust optimization theory with the Neural Engineering Framework. We developed two algorithms and compared their temperature-robustness across a range of 2° C by decoding one period of a sinusoid-like function from populations with 25 to 800 neurons. We find that 560 neurons are required to achieve the same precision across this temperature range as 35 neurons achieved at a single temperature.

The Neural Engineering Framework (NEF) is a theory for mapping computations onto biologically plausible networks of spiking neurons. This theory has been applied to a number of neuromorphic chips. However, within both silicon and real biological systems, synapses exhibit higher-order dynamics and heterogeneity. To date, the NEF has not explicitly addressed how to account for either feature. Here, we analytically extend the NEF to directly harness the dynamics provided by heterogeneous mixed-analog-digital synapses. This theory is successfully validated by simulating two fundamental dynamical systems in Nengo using circuit models validated in SPICE. Thus, our work reveals the potential to engineer robust neuromorphic systems with well-defined high-level behaviour that harness the low-level heterogeneous properties of their physical primitives with millisecond resolution.

Most online optimization algorithms focus on one of two things: performing well in adversarial settings by adapting to unknown data parameters (such as Lipschitz constants), typically achieving $O(\sqrt{T})$ regret, or performing well in stochastic settings where they can leverage some structure in the losses (such as strong convexity), typically achieving $O(\log(T))$ regret. Algorithms that focus on the former problem hitherto achieved $O(\sqrt{T})$ in the stochastic setting rather than $O(\log(T))$. Here we introduce an online optimization algorithm that achieves $O(\log^4(T))$ regret in a wide class of stochastic settings while gracefully degrading to the optimal $O(\sqrt{T})$ regret in adversarial settings (up to logarithmic factors). Our algorithm does not require any prior knowledge about the data or tuning of parameters to achieve superior performance.

We propose an online convex optimization algorithm (RescaledExp) that achieves optimal regret in the unconstrained setting without prior knowledge of any bounds on the loss functions. We prove a lower bound showing an exponential separation between the regret of existing algorithms that require a known bound on the loss functions and any algorithm that does not require such knowledge. RescaledExp matches this lower bound asymptotically in the number of iterations. RescaledExp is naturally hyperparameter-free and we demonstrate empirically that it matches prior optimization algorithms that require hyperparameter optimization.

In this paper, we describe the design of Neurogrid, a neuromorphic system for simulating large-scale neural models in real time. Neuromorphic systems realize the function of biological neural systems by emulating their structure. Designers of such systems face three major design choices: 1) whether to emulate the four neural elements-axonal arbor, synapse, dendritic tree, and soma-with dedicated or shared electronic circuits; 2) whether to implement these electronic circuits in an analog or digital manner; and 3) whether to interconnect arrays of these silicon neurons with a mesh or a tree network. The choices we made were: 1) we emulated all neural elements except the soma with shared electronic circuits; this choice maximized the number of synaptic connections; 2) we realized all electronic circuits except those for axonal arbors in an analog manner; this choice maximized energy efficiency; and 3) we interconnected neural arrays in a tree network; this choice maximized throughput. These three choices made it possible to simulate a million neurons with billions of synaptic connections in real time-for the first time-using 16 Neurocores integrated on a board that consumes three watts.

This special issue focuses on elucidating computational neuroscience: an interdisciplinary field of scientific research in which one of the primary goals is to understand how electronic activity in brain cells and networks enables biological intelligence. The objective is to provide a selection of papers that expose and review research efforts in aspects of computational neuroscience that demonstrate its rapidly growing intersection with electrical, electronic and computer engineering, and the prospects for interaction in the near and long-term future.

Motor prostheses aim to restore function to disabled patients. Despite compelling proof of concept systems, barriers to clinical translation remain. One challenge is to develop a low-power, fully-implantable system that dissipates only minimal power so as not to damage tissue. To this end, we implemented a Kalman-filter based decoder via a spiking neural network (SNN) and tested it in brain-machine interface (BMI) experiments with a rhesus monkey. The Kalman filter was trained to predict the arm’s velocity and mapped on to the SNN using the Neural Engineer- ing Framework (NEF). A 2,000-neuron embedded Matlab SNN implementation runs in real-time and its closed-loop performance is quite comparable to that of the standard Kalman filter. The success of this closed-loop decoder holds promise for hardware SNN implementations of statistical signal processing algorithms on neuromorphic chips, which may offer power savings necessary to overcome a major obstacle to the successful clinical translation of neural motor prostheses.

We present a 1-change-in-4 (1c4) link for interchip communication that extends level-encoded dual-rail (LEDR). LEDR transmits a bit on every transition by using all four 2bit codewords, with the bit encoded by the current codeword (level-encoding) rather than the difference between it and the previous codeword (transition signaling). 1c4 transmits two bits on every transition by using all sixteen 4-bit codewords, preserving LEDR's level-encoding property. Delay-insensitive I/O pad implementations for 1c4 encoding and decoding are described. Measurements of this chip-to-chip link, fabricated in a 0.18μm CMOS process, yielded a peak data-rate of 315 Mb/s at 1.8V and an energy efficiency of 89.8pJ/bit.

We present a network of silicon neurons that achieve robust synchrony using mutual inhibition. Synchrony strongly influences neuronal spike timing within many brain regions, potentially playing a crucial role in computation. Yet it has been largely ignored in neuromorphic systems, which use mixed analog and digital circuits to model neurobiology in silicon. Our neurons synchronize by using shunting inhibition (conductance-based) with a synaptic rise-time. Synaptic rise-time promotes synchrony by delaying the effect of inhibition, providing an opportune period for neurons to spike together. Shunting inhibition, through its voltage dependence, inhibits neurons that are late more strongly (delaying the spike further), pushing them into phase (in the next cycle). We fabricated a chip with 256 inhibitory neurons and 1,024 excitatory neurons in 0.25μm CMOS. We show that synchronized inhibitory neurons (population of 256) spike with a period that is proportional to the synaptic rise-time. We use these neurons to entrain the excitatory neurons, implementing a form of object binding

The authors present a silicon neuron that responds with a high-frequency burst of spikes (> 250Hz) when appropriately primed; otherwise it responds with lower frequency single spikes (2chip in 0.25μm CMOS. We plan to use feedback from a cortex chip to prime the relay neurons and thereby realize attention.

We describe a neurobiologically plausible model to implement dynamic routing using the concept of neuronal communication through neuronal coherence. The model has a three-tier architecture: a raw input tier, a routing control tier, and an invariant output tier. The correct mapping between input and output tiers is re- alized by an appropriate alignment of the phases of their respective background oscillations by the routing control units. We present an example architecture, im- plemented on a neuromorphic chip, that is able to achieve circular-shift invariance. A simple extension to our model can accomplish circular-shift dynamic routing with only O(N) connections, compared to O(N 2) connections required by tradi- tional models. 1 Dynamic Routing Circuit Models for Circular-Shift Invariance Dynamic routing circuit models are among the most prominent neural models for invariant recogni- tion [1] (also see [2] for review). These models implement shift invariance by dynamically changing spatial connectivity to transform an object to a standard position or orientation. The connectivity between the raw input and invariant output layers is controlled by routing units, which turn certain subsets of connections on or off (Figure 1A). An important feature of this model is the explicit rep- resentation of what and where information in the main network and the routing units, respectively; the routing units use the where information to create invariant representations. Traditional solutions for shift invariance are neurobiologically implausible for at least two reasons. First, there are too many synaptic connections: for N input neurons, N output neurons and N possible input-output mappings, the network requires O(N 2) connections in the routing layer— between each of the N routing units and each set of N connections that that routing unit gates (Figure 1A). Second, these connections must be extremely precise: each routing unit must activate an input- output mapping (N individual connections) corresponding to the desired shift (as highlighted in Figure 1A). Other approaches that have been proposed, including invariant feature networks [3,4], also suffer from significant drawbacks, such as the inability to explicitly represent where information [2]. It remains an open question how biology could achieve shift invariance without profligate and precise connections. In this article, we propose a simple solution for shift invariance for quantities that are circular or periodic in nature—circular-shift invariance (CSI)—orientation invariance in vision and key invari- ance in music. The visual system may create orientation-invariant representations to aid recognition under conditions of object rotation or head-tilt [5,6]; a similar mechanism could be employed by the auditory system to create key-invariant representations under conditions where the same melody 1 Figure 1: Dynamic routing. A In traditional dynamic routing, connections from the (raw) input layer to the (invariant) output layer are gated by routing units. For instance, the mapping from A to 5, B to 6, . . . , F to 4 is achieved by turning on the highlighted routing unit. B In time-division multiplexing (TDM), the encoder samples input channels periodically (using a rotating switch) while the decoder sends each sample to the appropriate output channel (based on its time bin). TDM can be extended to achieve a circular-shift transformation by altering the angle between encoder and decoder switches (θ), thereby creating a rotated mapping between input and output channels (adapted from [7]). is played in different keys. Similar to orientation, which is a periodic quantity, musical notes one octave apart sound alike, a phenomenon known as octave equivalence [8]. Thus, the problems of key invariance and orientation invariance admit similar solutions. Deriving inspiration from time-division multiplexing (TDM), we propose a neural network for CSI that uses phase to encode and decode information. We modulate the temporal window of commu- nication between (raw) input and (invariant) output neurons to achieve the appropriate input–output mapping. Extending TDM, any particular circular-shift transformation can be accomplished by changing the relative angle, θ, between the rotating switches of the encoder (that encodes the raw input in time) and decoder (that decodes the invariant output in time) (Figure 1B). This obviates the need to hardwire routing control units that specifically modulate the strength of each possible input- output connection, thereby significantly reducing the complexity inherent in the traditional dynamic routing solution. Similarly, a remapping between the input and output neurons can be achieved by introducing a relative phase-shift in their background oscillations. 2 Dynamic Routing through Neuronal Coherence To modulate the temporal window of communication, the model uses a ring of neurons (the oscilla- tion ring) to select the pool of neurons (in the projection ring) that encode or decode information at a particular time (Figure 2A). Each projection pool encodes a specific value of the feature (for exam- ple, one of twelve musical notes). Upon activation by external input, each pool is active only when background inhibition generated by the oscillation ring (outer ring of neurons) is at a minimum. In addition to exciting 12 inhibitory interneurons in the projection ring, each oscillation ring neuron excites its nearest 18 neighbors in the clockwise direction around the oscillation ring. As a result, a wave of inhibition travels around the projection ring that allows only one pool to be excitable at any point in time. These neurons become excitable at roughly the same time (numbered sectors, inner ring) by virtue of recurrent excitatory intra-pool connections. Decoding is accomplished by a second tier of rings (Figure 2B). The projection ring of the first (in- put) tier connects all-to-all to the projection ring of the second (output) tier. The two oscillation rings create a window of excitability for the pools of neurons in their respective projection rings. Hence, the most effective communication occurs between input and output pools that become excitable at the same time (i.e. are oscillating in phase with one another [9]). The CSI problem is solved by introducing a phase-shift between the input and output tiers. If they are exactly in phase, then an input pool is simply mapped to the output pool directly above it. If their 2 Figure 2: Double-Ring Network for Encoding and Decoding. A The projection (inner) ring is divided into (numbered) pools. The oscillation (outer) ring modulates sub-threshold activity (wave- forms) of the projection ring by exciting (black distribution) inhibitory neurons that inhibit neigh- boring projection neurons. A wave of activity travels around the oscillation ring due to asymmetric excitatory connections, creating a corresponding wave of inhibitory activity in the projection ring, such that only one pool of projection neurons is excitable (spikes) at a given time. B Two instances of the double-ring structure from A. The input projection ring connects all-to-all to the output pro- jection ring (dashed lines). Because each input pool will spike only during a distinct time bin, and each output pool is excitable only in a certain time bin, communication occurs between input and output pools that are oscillating in phase with each other. Appropriate phase offset between input and output oscillation rings realizes the desired circular shift (input pool H to output pool 1, solid arrow). C Interactions among pools highlighted in B. phases are different, the input is dynamically routed to an appropriate circularly shifted position in the output tier. Such changes in phase are analogous to adjusting the angle of the rotating switch at either the encoder or the decoder in TDM (see Figure 1B). There is some evidence that neural systems could employ phase relationships of subthreshold oscillations to selectively target neural populations [9-11]. 3 Implementation in Silicon We implemented this solution to CSI on a neuromorphic silicon chip [12]. The neuromorphic chip has neurons whose properties resemble that of biological neurons; these neurons even have intrin- sic differences, thereby mimicking heterogeneity in real neurobiological systems. The chip uses a conductance-based spiking model for both inhibitory and excitatory neurons. Inhibitory neurons project to nearby excitatory and inhibitory neurons via a diffusor network that determines the spread of inhibition. A lookup table of excitatory synaptic connectivity is stored in a separate random- access memory (RAM) chip. Spikes occurring on-chip are converted to a neuron address, mapped to synapses (if any) via the lookup table, and routed to the targeted on-chip synapse. A universal serial bus (USB) interface chip communicates spikes to and from a computer, for external input and 3 Figure 3: Traveling-wave activity in the oscillation ring. A Population activity (5ms bins) of a pool of eighteen (adjacent) oscillation neurons. B Increasing the strength of feedforward excitation led to increasing frequencies of periodic firing in the θ and α range (1-10 Hz). Strength of excitation is the amplitude change in post-synaptic conductance due to a single pre-synaptic spike (measured relative to minimum amplitude used). data analysis, respectively. Simulations on the chip occur in real-time, making it an attractive option for implementing the model. We configured the following parameters: • Magnitude of a potassium M-current: increasing this current’s magnitude increased the post-spike repolarization time of the membrane potential, thereby constraining spiking to a single time bin per cycle. • The strength of excitatory and inhibitory synapses: a correct balance had to be established between excitation and inhibition to make only a small subset of neurons in the projection rings fire at a time—too much excitation led to widespread firing and too much inhibition led to neurons that were entirely silent or fired sporadically. • The space constant of inhibitory spread: increasing the spread was effective in preventing runaway excitation, which could occur due to the recurrent excitatory connections. We were able to create a stable traveling wave of background activity within the oscillation ring. We transiently stimulated a small subset of the neurons, which initiated a wave of activity that propagated in a stable manner around the ring after the transient external stimulation had ceased (Figure 3A). The network frequency determined from a Fourier transform of the network activity smoothed with a non-causal Gaussian kernel (FDHM = 80ms) was 7.4Hz. The frequency varied with the strength of the neurons’ excitatory connections (Figure 3B), measured as the amplitude of the step increase in membrane conductivity due to the arrival of a pre-synaptic spike. Over much of the range of the synaptic strengths tested, we observed stable oscillations in the θ and α bands (1-10Hz); the frequency appeared to increase logarithmically with synaptic strength. 4 Phase-based Encoding and Decoding In order to assess the best-case performance of the model, the background activity in the input and output projection rings was derived from the input oscillation ring. Their spikes were delivered to the appropriately circularly-shifted output oscillation neurons. The asymmetric feedforward con- nections were disabled in the output oscillation ring. For instance, in order to achieve a circular shift by k pools (i.e. mapping input projection pool 1 to output projection pool k + 1, input pool 2 to output pool k + 2, and so on), activity from the input oscillation neurons closest to input pool 1 was fed into the output oscillation neurons closest to output pool k. By providing the appropriate phase difference between input and output oscillation, we were able to assess the performance of the model under ideal conditions. In the Discussion section, we discuss a biologically plausible mechanism to control the relative phases. 4 Figure 4: Phase-based encoding. Rasters indicating activity of projection pools in 1ms bins, and mean phase of firing (×’s) for each pool (relative to arbitrary zero time). The abscissa shows firing time normalized by the period of oscillation (which may be converted to firing phase by multiplica- tion by 2π). Under constant input to the input projection ring, the input pools fire approximately in sequence. Two cycles of pool activity normalized by maximum firing rate for each pool are shown in left inset (for clarity, pools 1-6 are shown in the top panel and pools 7-12 are shown separately in the bottom panel); phase of background inhibition of pool 4 is shown (below) for reference. Phase-aligned average1 of activity (right inset) showed that the firing times were relatively tight and uniform across pools: a standard deviation of 0.0945 periods, or equivalently, a spread of 1.135 pools at any instant of time. We verified that the input projection pools fired in a phase-shifted fashion relative to one another, a property critical for accurate encoding (see Figure 2). We stimulated all pools in the input pro- jection ring simultaneously while the input oscillation ring provided a periodic wave of background inhibition. The mean phase of firing for each pool (relative to arbitrary zero time) increased nearly linearly with pool number, thereby providing evidence for accurate, phase-based encoding (Figure 4). The firing times of all pools are shown for two cycles of background oscillatory activity (Figure 4 left inset). A phase-aligned average1 showed that the timing was relatively tight (standard deviation 1.135 pools) and uniform across pools of neurons (Figure 4 right inset). We then characterized the system’s ability to correctly decode this encoding under a given circular shift. The shift was set to seven pools, mapping input pool 1 to output pool 8, and so on. Each input pool was stimulated in turn. We expected to see only the appropriately shifted output pool become highly active. In fact, not only was this pool active, but other pools around it were also active, though to a lesser extent (Figure 5A). Thus, the phase-encoded input was decoded successfully, and circularly shifted, except that the output units were broadly tuned. To quantify the overall precision of encoding and decoding, we constructed an input-locked aver- age of the tuning curves (Figure 5B): the curves were circularly shifted to the left by an amount corresponding to the stimulated input pool number, and the raw pool firing rates were averaged. If the phase-based encoding and decoding were perfect, the peak should occur at a shift of 7 pools. 1The phase-aligned average was constructed by shifting the pool-activity curves by the (# of the pool) × 12 of the period) to align activity across pools, which was then averaged. ( 1 5 Figure 5: Decoding phase-encoded input. A In order to assess decoding performance under a given circular shift (here 7 pools) each input pool was stimulated in turn and activity in each output pool was recorded and averaged over 500ms. The pool’s response, normalized by its maximum firing rate, is plotted for each stimulated input pool (arrows pointing to curves, color code as in Figure 4). Each input pool stimulation trial consistently resulted in peak activity in the appropriate output pool; however, adjacent pools were also active, but to a lesser extent, resulting in a broad tuning curve. B The best-fit Gaussian (dot-dashed grey curve, σ = 2.30 pools) to the input-locked average of the raw pool firing rates (see text for details) revealed a maximum between a shift of 7 and 8 pools (inverted grey triangle; expected peak at a shift of 7 pools). Indeed, the highest (average) firing rate corresponded to a shift of 7 pools. However, the activity corresponding to a shift of 8 pools was nearly equal to that of 7 pools, and the best fitting Gaus- sian curve to the activity histogram (grey dot-dashed line) peaked at a point between pools 7 and 8 (inverted grey triangle). The standard deviation (σ) was 2.30 pools, versus the expected ideal σ of 1.60, which corresponds to the encoding distribution (σ = 1.135 pools) convolved with itself.

We model ion channels in silicon by exploiting similarities between the thermodynamic principles that govern ion channels and those that govern transistors. Using just eight transistors, we replicate--for the first time in silicon--the sigmoidal voltage dependence of activation (or inactivation) and the bell-shaped voltage-dependence of its time constant. We derive equations describing the dynamics of our silicon analog and explore its flexibility by varying various parameters. In addition, we validate the design by implementing a channel with a single activation variable. The design's compactness allows tens of thousands of copies to be built on a single chip, facilitating the study of biologically realistic models of neural computation at the network level in silicon.

In this paper, we present a network of silicon interneurons that synchronize in the gamma frequency range (20-80 Hz). The gamma rhythm strongly influences neuronal spike timing within many brain regions, potentially playing a crucial role in computation. Yet it has largely been ignored in neuromorphic systems, which use mixed analog and digital circuits to model neurobiology in silicon. Our neurons synchronize by using shunting inhibition (conductance based) with a synaptic rise time. Synaptic rise time promotes synchrony by delaying the effect of inhibition, providing an opportune period for interneurons to spike together. Shunting inhibition, through its voltage dependence, inhibits interneurons that spike out of phase more strongly (delaying the spike further), pushing them into phase (in the next cycle). We characterize the interneuron, which consists of soma (cell body) and synapse circuits, fabricated in a 0.25-microm complementary metal-oxide-semiconductor (CMOS). Further, we show that synchronized interneurons (population of 256) spike with a period that is proportional to the synaptic rise time. We use these interneurons to entrain model excitatory principal neurons and to implement a form of object binding.

We present a silicon neuron that uses shunting inhibition (conductance-based) with a synaptic rise-time to achieve synchrony. Synaptic rise-time promotes synchrony by delaying the effect of inhibition, providing an opportune period for neurons to spike together. And shunting inhibition, through its voltage dependence, inhibits neurons that are late more strongly (delaying the spike further), thereby pushing them into phase (in the next cycle). We characterize the soma (cell body) and synapse circuits, fabricated in 0.25 mum CMOS. Further, we show that synchronized neurons (population of 256) spike with a period that is proportional to the synaptic rise-time

We present a simple silicon circuit for modelling voltage-dependent ion channels found within neural cells, capturing both the gating particle's sigmoidal activation (or inactivation) and the bell-shaped time constant. In its simplest form, our ion-channel analog consists of two MOS transistors and a unity-gain inverter. We present equations describing its nonlinear dynamics and measurements from a chip fabricated in a 0.25 mum CMOS process. The channel analog's simplicity allows tens of thousands to be built on a single chip, facilitating the implementation of biologically realistic models of neural computation

We describe a neuromorphic chip with a two-layer excitatory-inhibitory recurrent network of that exhibits localized clusters of neural activity. Unlike other recurrent networks, the clusters in our network are pinned to certain locations due to transistor mismatch introduced in fabrication. As described in previous work, our pinned clusters respond selectively to oriented stimuli and the neurons' preferred orientations are distributed similar to the visual cortex. Here we show that orientation computation is rapid when activity alternates between layers (staccato-like), dislodging pinned clusters, which promotes fast cluster diffusion

We present a silicon neuron with a dynamic, active leak that enables precise spike-timing with respect to a time-varying input signal. Our neuron models the mammalian bushy cell, which enhances the phase-locking of its acoustically driven inputs. Our model enhances phase-locking by up to 38% (quantified by vector strength) across a 60 dB range of acoustic intensities, and up to 22% over a passive leak. Its conductance-based log-domain design yields a compact and efficient circuit, fabricated in 0.25 /spl mu/m CMOS, that is an ideal timing-enhancing component for neuromorphic speech recognition systems.

We describe a neuromorphic chip that uses binary synapses with spike timing-dependent plasticity (STDP) to learn stimulated patterns of activ- ity and to compensate for variability in excitability. Specifically, STDP preferentially potentiates (turns on) synapses that project from excitable neurons, which spike early, to lethargic neurons, which spike late. The additional excitatory synaptic current makes lethargic neurons spike ear- lier, thereby causing neurons that belong to the same pattern to spike in synchrony. Once learned, an entire pattern can be recalled by stimulating a subset. 1 Variability in Neural Systems Evidence suggests precise spike timing is important in neural coding, specifically, in the hippocampus. The hippocampus uses timing in the spike activity of place cells (in addition to rate) to encode location in space [1]. Place cells employ a phase code: the timing at which a neuron spikes relative to the phase of the inhibitory theta rhythm (5-12Hz) conveys information. As an animal approaches a place cell’s preferred location, the place cell not only increases its spike rate, but also spikes at earlier phases in the theta cycle. To implement a phase code, the theta rhythm is thought to prevent spiking until the input synaptic current exceeds the sum of the neuron threshold and the decreasing inhibition on the downward phase of the cycle [2]. However, even with identical inputs and common theta inhibition, neurons do not spike in synchrony. Variability in excitability spreads the activity in phase. Lethargic neurons (such as those with high thresholds) spike late in the theta cycle, since their input exceeds the sum of the neuron threshold and theta inhibition only after the theta inhibition has had time to decrease. Conversely, excitable neurons (such as those with low thresholds) spike early in the theta cycle. Consequently, variability in excitability translates into variability in timing. We hypothesize that the hippocampus achieves its precise spike timing (about 10ms) through plasticity enhanced phase-coding (PEP). The source of hippocampal timing preci- sion in the presence of variability (and noise) remains unexplained. Synaptic plasticity can compensate for variability in excitability if it increases excitatory synaptic input to neurons in inverse proportion to their excitabilities. Recasting this in a phase-coding framework, we desire a learning rule that increases excitatory synaptic input to neurons directly related to their phases. Neurons that lag require additional synaptic input, whereas neurons that lead

We demonstrate the first fully hardware implementation of retinotopic self-organization, from photon transduction to neural map formation. A silicon retina transduces patterned illumination into correlated spike trains that drive a population of silicon growth cones to automatically wire a topographic mapping by migrating toward sources of a diffusible guidance cue that is released by postsynaptic spikes. We varied the pat- tern of illumination to steer growth cones projected by different retinal ganglion cell types to self-organize segregated or coordinated retinotopic maps.

We present a novel cochlear model implemented in analog very large scale integration (VLSI) technology that emulates nonlinear active cochlear behavior. This silicon cochlea includes outer hair cell (OHC) electromotility through active bidirectional coupling (ABC), a mech- anism we proposed in which OHC motile forces, through the mi- croanatomical organization of the organ of Corti, realize the cochlear amplifier. Our chip measurements demonstrate that frequency responses become larger and more sharply tuned when ABC is turned on; the de- gree of the enhancement decreases with input intensity as ABC includes saturation of OHC forces. 1 Silicon Cochleae Cochlear models, mathematical and physical, with the shared goal of emulating nonlinear active cochlear behavior, shed light on how the cochlea works if based on cochlear mi- cromechanics. Among the modeling efforts, silicon cochleae have promise in meeting the need for real-time performance and low power consumption. Lyon and Mead developed the first analog electronic cochlea [1], which employed a cascade of second-order filters with exponentially decreasing resonant frequencies. However, the cascade structure suf- fers from delay and noise accumulation and lacks fault-tolerance. Modeling the cochlea more faithfully, Watts built a two-dimensional (2D) passive cochlea that addressed these shortcomings by incorporating the cochlear fluid using a resistive network [2]. This par- allel structure, however, has its own problem: response gain is diminished by interference among the second-order sections’ outputs due to the large phase change at resonance [3]. Listening more to biology, our silicon cochlea aims to overcome the shortcomings of exist- ing architectures by mimicking the cochlear micromechanics while including outer hair cell (OHC) electromotility. Although how exactly OHC motile forces boost the basilar mem- brane’s (BM) vibration remains a mystery, cochlear microanatomy provides clues. Based on these clues, we previously proposed a novel mechanism, active bidirectional coupling (ABC), for the cochlear amplifier [4]. Here, we report an analog VLSI chip that implements this mechanism. In essence, our implementation is the first silicon cochlea that employs stimulus enhancement (i.e., active behavior) instead of undamping (i.e., high filter Q [5]). The paper is organized as follows. In Section 2, we present the hypothesized mechanism (ABC), first described in [4]. In Section 3, we provide a mathematical formulation of the

We describe a neuromorphic chip designed to model active dendrites, recurrent connectivity, and plastic synapses to support one-shot learning. Specifically, it is designed to capture neural firing patterns (short-term memory), memorize individual patterns (long-term memory), and retrieve them when primed (associative recall). It consists of a recurrently connected population of excitatory pyramidal cells and a recurrently connected population of inhibitory basket cells. In addition to their recurrent connections, the excitatory and inhibitory populations are reciprocally connected. The model is novel in that it utilizes recurrent connections and active dendrites to maintain short-term memories as well as to store long-term memories.

that utilizes We describe a neuromorphic chip transistor heterogeneity, introduced by the fabrication process, to generate orientation maps similar to those imaged in vivo. Our model consists of a recurrent network of excitatory and inhibitory cells in parallel with a push-pull stage. Similar to a previous model the recurrent network displays hotspots of activity that give rise to visual feature maps. Unlike previous work, however, the map for orientation does not depend on the sign of contrast. Instead, sign- independent cells driven by both ON and OFF channels anchor the map, while push-pull interactions give rise to sign-preserving cells. These two groups of orientation-selective cells are similar to complex and simple cells observed in V1. 1 Orientation Maps Neurons in visual areas 1 and 2 (V1 and V2) are selectively tuned for a number of visual features, the most pronounced feature being orientation. Orientation preference of individual cells varies across the two-dimensional surface of the cortex in a stereotyped manner, as revealed by electrophysiology [1] and optical imaging studies [2]. The origin of these preferred orientation (PO) maps is debated, but experiments demonstrate that they exist in the absence of visual experience [3]. To the dismay of advocates of Hebbian learning, these results suggest that the initial appearance of PO maps rely on neural mechanisms oblivious to input correlations. Here, we propose a model that accounts for observed PO maps based on innate noise in neuron thresholds and synaptic currents. The network is implemented in silicon where heterogeneity is as ubiquitous as it is in biology. 2 Patterned Activity Model Ernst et al. have previously described a 2D rate model that can account for the origin of visual maps [4]. Individual units in their network receive isotropic feedforward input from the geniculate and recurrent connections from neighboring units in a Mexican hat profile, described by short-range excitation and long-range inhibition. If the recurrent connections are sufficiently strong, hotspots of activity (or ‘bumps’) form periodically across space. In a homogeneous network, these bumps of activity are equally stable at any position in the network and are free to wander. Introducing random jitter to the Mexican hat connectivity profiles breaks the symmetry and reduces the number of stable states for the bumps. Subsequently, the bumps are pinned down at the locations that maximize their net local recurrent feedback. In this regime, moving gratings are able to shift the bumps away from their stability points such that the responses of the network resemble PO maps. Therefore, the recurrent network, given an ample amount of noise, can innately generate its own orientation specificity without the need for specific hardwired connections or visually driven learning rules. 2.1 Criticisms of the Bump model We might posit that the brain uses a similar opportunistic model to derive and organize its feature maps – but the parallels between the primary visual cortex and the Ernst et al. bump model are unconvincing. For instance, the units in their model represent the collective activity of a column, reducing the network dynamics to a firing-rate approximation. But this simplification ignores the rich temporal dynamics of spiking networks, which are known to affect bump stability. More fundamentally, there is no role for functionally distinct neuron types. The primary criticism of the Ernst et al.’s bump model is that its input only consists of a luminance channel, and it is not obvious how to replace this channel with ON and OFF rectified channels to account for simple and complex cells. One possibility would be to segregate ON-driven and OFF-driven cells (referred to as simple cells in this paper) into two distinct recurrent networks. Because each network would have its own innate noise profile, bumps would form independently. Consequently, there is no guarantee that ON-driven maps would line up with OFF-driven maps, which would result in conflicting orientation signals when these simple cells converge onto sign-independent (complex) cells. 2.2 Simple Cells Solve a Complex Problem To ensure that both ON-driven and OFF-driven simple cells have the same orientation maps, both ON and OFF bumps must be computed in the same recurrent network so that they are subjected to the same noise profile. We achieve this by building our recurrent network out of cells that are sign-independent; that is both ON and OFF channels drive the network. These cells exhibit complex cell-like behavior (and are referred to as complex cells in this paper) because they are modulated at double the spatial frequency of a sinusoidal grating input. The simple cells subsequently derive their responses from two separate signals: an orientation selective feedback signal from the complex cells indicating the presence of either an ON or an OFF bump, and an ON–OFF selection signal that chooses the appropriate response flavor. Figure 1 left illustrates the formation of bumps (highlighted cells) by a recurrent network with a Mexican hat connectivity profile. Extending the Ernst et al. model, these complex bumps seed simple bumps when driven by a grating. Simple bumps that match the sign of the input survive, whereas out-of-phase bumps are extinguished (faded cells) by push-pull inhibition. Figure 1 right shows the local connections within a microcircuit. An EXC (excitatory) cell receives excitatory input from both ON and OFF channels, and projects to other EXC (not shown) and INH (inhibitory) cells. The INH cell projects back in a reciprocal configuration to EXC cells. The divergence is indicated in left. ON-driven and OFF-driven simple cells receive input in a push-pull configuration (i.e., ON cells are excited by ON inputs and inhibited by OFF inputs, and vise-versa), while additionally receiving input from the EXC–INH recurrent network. In this model, we inhibitory connections, despite the fact that geniculate input is strictly excitatory. This simplification, while anatomically incorrect, yields a more efficient implementation that is functionally equivalent. implement our push-pull circuit using monosynaptic

We describe a self-configuring neuromorphic chip that uses a model of activity-dependent axon remodeling to automatically wire topographic maps based solely on input correlations. Axons are guided by growth cones, which are modeled in analog VLSI for the first time. Growth cones migrate up neurotropin gradients, which are represented by charge diffusing in transistor channels. Virtual axons move by rerouting address-events. We refined an initially gross topographic projection by simulating retinal wave input. 1 Neuromorphic Systems Neuromorphic engineers are attempting to match the computational efficiency of biological systems by morphing neurocircuitry into silicon circuits [1]. One of the most detailed implementations to date is the silicon retina described in [2] . This chip comprises thirteen different cell types, each of which must be individually and painstakingly wired. While this circuit-level approach has been very successful in sensory systems, it is less helpful when modeling largely unelucidated and exceedingly plastic higher processing centers in cortex. Instead of an explicit blueprint for every cortical area, what is needed is a developmental rule that can wire complex circuits from minimal specifications. One candidate is the famous "cells that fire together wire together" rule, which strengthens excitatory connections between coactive presynaptic and postsynaptic cells. We implemented a self-rewiring scheme of this type in silicon, taking our cue from axon remodeling during development.

We consider the problem of automatic object recognition by small, light-weight, low power, hardware systems. We abstract from biological function and organization and propose hardware architectures and a design methodology to engineer such hardware. Robust, miniature, and energetically efficient VLSI systems for AOR can ultimately be achieved by following a path which optimizes the design at and between all levels of system integration, i.e., from devices and circuit techniques all the way to algorithms and architectural level considerations. By way of example, we discuss two experimental systems for image acquisition and preprocessing fabricated in standard CMOS processes. The first one is a large scale analog system, a contrast sensitive silicon retina, with over 590, 000 transistors operating in subthreshold CMOS. The second system is a mixed analog-digital system for image acquisition and tracking compensation that incorporates a contrast sensitive silicon retina in the image sensing area.

An overview of the current-mode approach for designing analog VLSI neural systems in subthreshold CMOS technology is presented. Emphasis is given to design techniques at the device level using the current-controlled current conveyor and the translinear principle. Circuits for associative memory and silicon retina systems are used as examples. The design methodology and how it relates to actual biological microcircuits are discussed.

We present a new approach to the engineering of collective analog computing systems that emphasizes the role of currents as an appropriate signal representation and the need for low-power dissipation and simplicity in the basic functional circuits. The design methodology and implementation style that we describe are inspired by the functional and organizational principles of neuronal circuits in living systems. We have implemented synthetic neurons and synapses in analog CMOS VLSI that are suitable for building associative memories and self-organizing feature maps.