Event-Based cameras asynchronously detect changes in light intensity with high temporal resolution, making them a promising alternative to RGB camera for low-latency and low-power optical flow estimation. However, state-of-the-art convolutional neural network methods create frames from the event stream, therefore losing the opportunity to exploit events for both sparse computations and low-latency prediction. On the other hand, asynchronous event graph methods could leverage both, but at the cost of avoiding any form of time accumulation, which limits the prediction accuracy. In this paper, we propose to break this accuracy-latency trade-off with a novel architecture combining an asynchronous accumulation-free event branch and a periodic aggregation branch. The periodic branch performs feature aggregations on the event graphs of past data to extract global context information, which improves accuracy without introducing any latency. The solution could predict optical flow per event with a latency of tens of microseconds on asynchronous hardware, which represents a gain of three orders of magnitude with respect to state-of-the-art frame-based methods, with 48x less operations per second. We show that the solution can detect rapid motion changes faster than a periodic output. This work proposes, for the first time, an effective solution for ultra low-latency and low-power optical flow prediction from event cameras.

Graph neural networks have recently emerged as a promising approach for low-power and low-latency event-vision applications. Such event-graphs naturally exploit the sparsity of event-data and incorporate the temporal detail captured by event-based sensors directly into the edge features used in graph convolution. In this paper we study the promise of event-graphs for processing data from other event-based modalities beyond vision. Specifically, we describe how the approach can be adapted to the spectro-temporal domain to perform event-audio classification. We evaluate the approach using the spiking Heidelberg digits dataset and achieve a test accuracy of 94.3%. This is notably better than many state of the art spiking neural networks despite, in many cases, requiring an order of magnitude fewer parameters. Event-graph neural networks promise to be a powerful, general approach for processing a variety of event-based modalities, not only vision.

Since neuromorphic event-based pixels and cameras were first proposed, the technology has greatly advanced such that there now exists several industrial sensors, processors and toolchains. This has also paved the way for a blossoming new branch of AI dedicated to processing the event-based data these sensors generate. However, there is still much debate about which of these approaches can best harness the inherent sparsity, low-latency and fine spatiotemporal structure of event-data to obtain better performance and do so using the least time and energy. The latter is of particular importance since these algorithms will typically be employed near or inside of the sensor at the edge where the power supply may be heavily constrained. The two predominant methods to process visual events - convolutional and spiking neural networks - are fundamentally opposed in principle. The former converts events into static 2D frames such that they are compatible with 2D convolutions, while the latter computes in an event-driven fashion naturally compatible with the raw data. We review this dichotomy by studying recent algorithmic and hardware advances of both approaches. We conclude with a perspective on an emerging alternative approach whereby events are transformed into a graph data structure and thereafter processed using techniques from the domain of graph neural networks. Despite promising early results, algorithmic and hardware innovations are required before this approach can be applied close or within the Event-based sensor.

Spiking Neural Networks (SNNs) can unleash the full power of analog Resistive Random Access Memories (RRAMs) based circuits for low power signal processing. Their inherent computational sparsity naturally results in energy efficiency benefits. The main challenge implementing robust SNNs is the intrinsic variability (heterogeneity) of both analog CMOS circuits and RRAM technology. In this work, we assessed the performance and variability of RRAM-based neuromorphic circuits that were designed and fabricated using a 130 nm technology node. Based on these results, we propose a Neuromorphic Hardware Calibrated (NHC) SNN, where the learning circuits are calibrated on the measured data. We show that by taking into account the measured heterogeneity characteristics in the off-chip learning phase, the NHC SNN self-corrects its hardware non-idealities and learns to solve benchmark tasks with high accuracy. This work demonstrates how to cope with the heterogeneity of neurons and synapses for increasing classification accuracy in temporal tasks.

Dedicated hardware implementations of spiking neural networks that combine the advantages of mixed-signal neuromorphic circuits with those of emerging memory technologies have the potential of enabling ultra-low power pervasive sensory processing. To endow these systems with additional flexibility and the ability to learn to solve specific tasks, it is important to develop appropriate on-chip learning mechanisms. Recently, a new class of three-factor spike-based learning rules have been proposed that can solve the temporal credit assignment problem and approximate the error back-propagation algorithm on complex tasks. However, the efficient implementation of these rules on hybrid CMOS/memristive architectures is still an open challenge. Here we present a new neuromorphic building block, called PCM-trace, which exploits the drift behavior of phase- change materials to implement long lasting eligibility traces, a critical ingredient of three-factor learning rules. We demonstrate how the proposed approach improves the area efficiency by > 10× compared to existing solutions and demonstrates a technologically plausible learning algorithm supported by experimental data from device measurements.

Many edge computing and IoT applications require adaptive and on-line learning architectures for fast and low-power processing of locally sensed signals. A promising class of architectures to solve this problem is that of in-memory computing ones, based on event-based hybrid memristive-CMOS devices. In this work, we present an example of such systems that supports always-on on-line learning. To overcome the problems of variability and limited resolution of ReRAM memristive devices used to store synaptic weights, we propose to use only their High Conductive State (HCS) and control their desired conductance by modulating their programming Compliance Current (ICC). We describe the spike-based learning CMOS circuits that are used to modulate the synaptic weights and demonstrate the relationship between the synaptic weight, the device conductance, and the ICCused to set its weight, with experimental measurements from a 4kb array of HfO2-based devices. To validate the approach and the circuits presented, we present circuit simulation results for a standard CMOS 180nm process and system-level behavioral simulations for classifying hand-written digits from the MNIST data-set with classification accuracy of 92.68% on the test set.

Brain-inspired architectures in neuromorphic hardware are currently subject to intensive research as an alternative to the limits of traditional computer organisation. The remarkable computing performance and efficiency of biological nervous systems are widely attributed to the co-localisation of memory and computation spatially throughout the structure. Moreover, it appears that a number of local self-organising neural mechanisms play their part in efficient biological computation. An example is neuronal intrinsic plasticity, where a neuron adapts its parameters to maximise its information capacity based on the statistical properties of its input while minimising the power it consumes. CMOS circuits implementing neuron models have been proposed but require their parameters to be set by biases originating from a centralised memory. In this work, we propose a hybrid CMOS-RRAM circuit that addresses this problem through storing neuron parameters within programmable nonvolatile resistive memories incorporated into the CMOS neuron. Additional circuits exploit the stochastic switching properties of resisitive memories to map a local intrinsic plasticity algorithm onto the proposed neuron. We demonstrate the computational advantages of this algorithm through simulation, calibrated on experimental data, whereby the neuron maximises its information capacity while minimising its power consumption, as is the case for biological neurons.

Resistive Random Access Memory (RRAM)-based artificial Neural Networks (NNs) have been shown to be intrinsically robust to RRAM variability but no study has been done to clearly explain and quantify this robustness. In this paper, we fully characterize a 4kbit RRAM array under different programming conditions. The impact of the electrical characteristics of RRAM (resistance variability, memory window, endurance performance) on the detection rate of a NN designed for object tracking and trained with a stochastic Spike-Timing Dependent Plasticity (STDP) rule is studied. We introduce a new parameter called the Synaptic Window (SW), defined as the ratio between the arithmetic mean conductance values of the low and high resistance distributions. The network performance was found only to be sensitive to the value of the SW (a SW>100 is required to achieve the maximum NN performance). Moreover, we demonstrate that a high resistance variability increases the SW for a given window margin.