Event-Based cameras asynchronously detect changes in light intensity with high temporal resolution, making them a promising alternative to RGB camera for low-latency and low-power optical flow estimation. However, state-of-the-art convolutional neural network methods create frames from the event stream, therefore losing the opportunity to exploit events for both sparse computations and low-latency prediction. On the other hand, asynchronous event graph methods could leverage both, but at the cost of avoiding any form of time accumulation, which limits the prediction accuracy. In this paper, we propose to break this accuracy-latency trade-off with a novel architecture combining an asynchronous accumulation-free event branch and a periodic aggregation branch. The periodic branch performs feature aggregations on the event graphs of past data to extract global context information, which improves accuracy without introducing any latency. The solution could predict optical flow per event with a latency of tens of microseconds on asynchronous hardware, which represents a gain of three orders of magnitude with respect to state-of-the-art frame-based methods, with 48x less operations per second. We show that the solution can detect rapid motion changes faster than a periodic output. This work proposes, for the first time, an effective solution for ultra low-latency and low-power optical flow prediction from event cameras.

Compute-in-memory (CIM) accelerators for spiking neural networks (SNNs) are promising solutions to enable µs-level inference latency and ultra-low energy in edge vision applications. Yet, their current lack of flexibility at both the circuit and system levels prevents their deployment in a wide range of real-life scenarios. In this work, we propose FlexSpIM, a novel digital CIM macro that supports arbitrary operand resolution and shape within a unified CIM storage for weights and membrane potentials. These circuit-level techniques enable a hybrid weight- and output-stationary dataflow at the system level to maximize operand reuse, thereby minimizing costly on- and off-chip data movements during the SNN execution. Measurement results of a fabricated FlexSpIM prototype in 40-nm CMOS demonstrate a 2× increase in 1-bit-normalized energy efficiency compared to prior fixed-precision digital CIM-based SNNs, while providing resolution reconfiguration with bitwise granularity. Our approach can save up to 90% energy in large-scale systems, while reaching a state-of-the-art classification accuracy of 95.8% on the IBM DVS gesture dataset.

Since neuromorphic event-based pixels and cameras were first proposed, the technology has greatly advanced such that there now exists several industrial sensors, processors and toolchains. This has also paved the way for a blossoming new branch of AI dedicated to processing the event-based data these sensors generate. However, there is still much debate about which of these approaches can best harness the inherent sparsity, low-latency and fine spatiotemporal structure of event-data to obtain better performance and do so using the least time and energy. The latter is of particular importance since these algorithms will typically be employed near or inside of the sensor at the edge where the power supply may be heavily constrained. The two predominant methods to process visual events - convolutional and spiking neural networks - are fundamentally opposed in principle. The former converts events into static 2D frames such that they are compatible with 2D convolutions, while the latter computes in an event-driven fashion naturally compatible with the raw data. We review this dichotomy by studying recent algorithmic and hardware advances of both approaches. We conclude with a perspective on an emerging alternative approach whereby events are transformed into a graph data structure and thereafter processed using techniques from the domain of graph neural networks. Despite promising early results, algorithmic and hardware innovations are required before this approach can be applied close or within the Event-based sensor.

The NimbleAI Horizon Europe project leverages key principles of energy-efficient visual sensing and processing in biological eyes and brains, and harnesses the latest advances in$\mathbf{33D}$stacked silicon integration, to create an integral sensing-processing neuromorphic architecture that efficiently and accurately runs computer vision algorithms in area-constrained endpoint chips. The rationale behind the NimbleAI architecture is: sense data only with high information value and discard data as soon as they are found not to be useful for the application (in a given context). The NimbleAI sensing-processing architecture is to be specialized after-deployment by tunning system-level trade-offs for each particular computer vision algorithm and deployment environment. The objectives of NimbleAI are: (1)$\mathbf{100x}$performance per mW gains compared to state-of-the-practice solutions (i.e., CPU/GPUs processing frame-based video); (2)$\mathbf{50x}$processing latency reduction compared to CPU/GPUs; (3) energy consumption in the order of tens of mWs; and (4) silicon area of approx. 50 mm2.

A high input voltage headroom current mirror is presented. The proposed architecture takes advantage of all modes of operation of the input and output transistors for current recopy (from deep subthreshold to saturation and triode mode) by controlling the gate and drain voltages of the mirror transistors separately. This allows the output and input impedances to always match the input current, thus greatly increasing current recopy accuracy across more than 150db of dynamic range. This technique also allows for a low impedance at high current inputs, resulting in a 531mV input voltage headroom gain compared to a same size classic architecture.

Optogenetics therapy aims at recovering visual function in patients suffering from retinal degeneration and blindness. An external device is generally required to correctly encode light stimulation into neural electric activity and properly accomplish visual restoration. In this demo, we present photostimulation glasses for adequately activate optogenetically-treated retinal ganglion cells (RGC). This device performs light transduction at appropriate wavelength and intensity for an optogenetic therapy, and performs the visual computation in place of the degenerated retinal layers. We show the possible configurations and stimulation modes to be used for clinical trials or rehabilitation purposes.

Optogenetics can be used to restore light responses in patients affected by retinal degenerative diseases. The light-sensitivity of the molecule introduced by genetic therapy is however very limited in terms of wavelength and irradiance needed to activate a useful neural response, and thus needs an external device to be correctly stimulated. Moreover, the visual signal needs to be encoded so as to respect the code used by the target cells (e.g. retinal ganglion cells). In this demonstration we present a platform that can be used to stimulate optogenetically-treated retinal cells, and the algorithms associated with different types of stimulations. We show different stimulation strategies, varying accordingly to the type of cells transfected.

Optogenetic therapy holds the promise to restore visual function in patients affected by retinal degenerative diseases. However, the light-sensitivity of the molecule mediating light responses is much less than the one of healthy retinal cells so that no photo-stimulation is expected under natural environmental conditions. In this work, we present a platform set up to stimulate optogenetically-engineered retinal cells, and the algorithms associated with different types of stimulation. The system consists of a neuromorphic silicon retina as a visual frontend, a projecting device capable of delivering fast and precise light stimulation and a computing platform implementing the stimulation algorithms. We describe different strategies, varying depending on the type of cells transfected. The silicon retina provides a natural front-end for an artificial visual system, complying with the information encoding principles, timing properties and dynamic range of either photoreceptors or RGCs. The encoding of the visual information is performed with sub-millisecond accuracy, respecting the temporal characteristics of the neural system. The platform and algorithms hereby presented provide a basis for medical devices matching the requirements of optogenetic therapeutic use. An embedded version of this platform will be used in the forthcoming clinical trials of the GS030 vision restoration therapy.

This live demonstration shows ultra-low bandwidth video streaming based on a scene-driven event-encoding imaging sensor. The approach exploits the inherent focal-plane redundancy suppression / video compression achieved by an array of autonomous, auto-sampling pixels. The data readout from the camera is optimized for transmission bandwidth using variable bit-length pixel address encoding and spatio-temporal pre-filtering of the raw image data, resulting in instantaneous bit rates that vary between 0bps and a set rate, e.g. 256kbps, only depending on scene activity. The demonstrated device is a small form-factor stand-alone camera that, besides the sensor and optics, includes hardware-based data encoding and filtering, and a wireless data transmission module. The device is designed for e.g. surveillance applications in resource-limited environments such as sensor-networks or IoT.

This live demonstration shows real-time visual object recognition based on a spiking neural network adaptation of the HMAX model running on a purely event-based computational hardware platform. Visual input to the system is provided by an ATIS spiking silicon retina sensor. A SpiNNaker board processes the event-encoded visual information from the scene. Using a Leaky Integrate-and-Fire (LIF) neuron model implemented on SpiNNaker, an event-driven, multi-layer network is created that performs real-time orientation extraction and recombination. In this demonstration, the network will be tuned to recognize complex objects such as printed characters.

This paper presents a real-time spiking neural network adaptation of the HMAX object recognition model on an event-driven platform. Visual input is provided by a spiking silicon retina, while the SpiNNaker system is used as a computational hardware platform for implementation. We show the implementation of a simple Leaky Integrate-and-Fire (LIF) neuron model on SpiNNaker to create an event driven network, where a neuron only updates when it receives an interrupt indicating that a new input spike has been received. The model output consists of view tuned neurons which respond selectively to a particular view of an object. The network can be used to discriminate between objects, or between the same object at different views. On a 26 class character recognition task, the correct class is always assigned the highest probability (69.42% on average).

This letter presents a novel computationally efficient and robust pattern tracking method based on a time-encoded, frame-free visual data. Recent interdisciplinary developments, combining inputs from engineering and biology, have yielded a novel type of camera that encodes visual information into a continuous stream of asynchronous, temporal events. These events encode temporal contrast and intensity locally in space and time. We show that the sparse yet accurately timed information is well suited as a computational input for object tracking. In this letter, visual data processing is performed for each incoming event at the time it arrives. The method provides a continuous and iterative estimation of the geometric transformation between the model and the events representing the tracked object. It can handle isometry, similarities, and affine distortions and allows for unprecedented real-time performance at equivalent frame rates in the kilohertz range on a standard PC. Furthermore, by using the dimension of time that is currently underexploited by most artificial vision systems, the method we present is able to solve ambiguous cases of object occlusions that classical frame-based techniques handle poorly.

This paper introduces a spiking hierarchical model for object recognition which utilizes the precise timing information inherently present in the output of biologically inspired asynchronous address event representation (AER) vision sensors. The asynchronous nature of these systems frees computation and communication from the rigid predetermined timing enforced by system clocks in conventional systems. Freedom from rigid timing constraints opens the possibility of using true timing to our advantage in computation. We show not only how timing can be used in object recognition, but also how it can in fact simplify computation. Specifically, we rely on a simple temporal-winner-take-all rather than more computationally intensive synchronous operations typically used in biologically inspired neural networks for object recognition. This approach to visual computation represents a major paradigm shift from conventional clocked systems and can find application in other sensory modalities and computational tasks. We showcase effectiveness of the approach by achieving the highest reported accuracy to date (97.5% ± 3.5%) for a previously published four class card pip recognition task and an accuracy of 84.9% ± 1.9% for a new more difficult 36 class character recognition task.

This paper presents a frame-free time-domain imaging approach designed to alleviate the non-ideality of finite exposure measurement time (intrinsic to all integrating imagers), limiting the temporal resolution of the ATIS asynchronous time-based image sensor concept. The method uses the time-domain correlated double sampling (TCDS) and change detection circuitry already present in the data-driven autonomous ATIS pixels and does not involve any additional data to be transmitted by the sensor, but is entirely based on the data available in normal operation. Three consecutive exposure estimation / measurement steps apply different trade-offs between measurement speed, accuracy and noise. The early estimates yield between 10 and 100 times faster pixel updates than the standard full-swing integrating exposure measurement operation. The results from the three individual measurement steps can be used separately or in combination, enabling event-driven asynchronous high-speed imaging at moderate light levels.

This paper introduces a new methodology to process asynchronously sampled image data captured by a new generation of biomimetic vision sensors. Unlike conventional cameras, these neuromorphic sensors acquire data not at fixed points in time for the entire array (frame-based) but sparse in space and time, i.e., pixel-individually and precisely timed only if new information is available (event-based). In this paper, we introduce a filtering methodology for asynchronously acquired gray-level data from an event-driven time-encoding imager. The paper first studies the properties of level-crossing sampling parameters in order to define threshold level properties and associated bandwidth needs. In a second stage, we introduce asynchronous linear and nonlinear filtering techniques. Examples are shown and examined on real data. Finally, the paper introduces a methodology to compare frame-based versus event-based computational costs. Implementations and experiments show that event-based gray-level filtering produces equivalent filtering accuracy as compared to frame-based ones. The main result of this work shows that, based on the number of operations to be carried out, beyond 3 frames per second (fps), event-based processing outperforms frame-based processing in terms of computational cost.

State-of-the-art image sensors suffer from significant limitations imposed by their very principle of operation. These sensors acquire the visual information as a series of “snapshot” images, recorded at discrete points in time. Visual information gets time quantized at a predetermined frame rate which has no relation to the dynamics present in the scene. Furthermore, each recorded frame conveys the information from all pixels, regardless of whether this information, or a part of it, has changed since the last frame had been acquired. This acquisition method limits the temporal resolution, potentially missing important information, and leads to redundancy in the recorded image data, unnecessarily inflating data rate and volume. Biology is leading the way to a more efficient style of image acquisition. Biological vision systems are driven by events happening within the scene in view, and not, like image sensors, by artificially created timing and control signals. Translating the frameless paradigm of biological vision to artificial imaging systems implies that control over the acquisition of visual information is no longer being imposed externally to an array of pixels but the decision making is transferred to the single pixel that handles its own information individually. In this paper, recent developments in bioinspired, neuromorphic optical sensing and artificial vision are presented and discussed. It is suggested that bioinspired vision systems have the potential to outperform conventional, frame-based vision systems in many application fields and to establish new benchmarks in terms of redundancy suppression and data compression, dynamic range, temporal resolution, and power efficiency. Demanding vision tasks such as real-time 3-D mapping, complex multiobject tracking, or fast visual feedback loops for sensory-motor action, tasks that often pose severe, sometimes insurmountable, challenges to conventional artificial vision systems, are in reach using bioinspired vision sensing and processing techniques.

A QVGA array of autonomous, event-based temporal contrast sensitive pixels is at the basis of an asynchronous, time-based CMOS dynamic vision and image sensor (ATIS). In this sensor, exposure measurements are initiated and carried out locally by individual pixels upon detection of temporal contrast in their field-of-view. The change detection pixel circuits respond with low latency to temporal contrast (i.e. relative changes in illumination) by generating `spike' pulse events. The vision sensor has been implemented in a 0.18μm CMOS process, fabricated and characterized. To assess the quality of the change detector array, contrast sensitivity, noise behavior and response uniformity are crucial performance figures. In this paper, we propose a test method that allows acquiring and evaluating these three performance parameters simultaneously. The results are shown to agree well with predictions from theoretical considerations, validating the proposed test method. The ATIS change detector array exhibits RMS noise equivalent contrast between 2% and 4% and achieves array response mismatch of 1.9% RMS.

Epipolar geometry, the cornerstone of perspective stereo vision, has been studied extensively since the advent of computer vision. Establishing such a geometric constraint is of primary importance, as it allows the recovery of the 3-D structure of scenes. Estimating the epipolar constraints of nonperspective stereo is difficult, they can no longer be defined because of the complexity of the sensor geometry. This paper will show that these limitations are, to some extent, a consequence of the static image frames commonly used in vision. The conventional frame-based approach suffers from a lack of the dynamics present in natural scenes. We introduce the use of neuromorphic event-based--rather than frame-based--vision sensors for perspective stereo vision. This type of sensor uses the dimension of time as the main conveyor of information. In this paper, we present a model for asynchronous event-based vision, which is then used to derive a general new concept of epipolar geometry linked to the temporal activation of pixels. Practical experiments demonstrate the validity of the approach, solving the problem of estimating the fundamental matrix applied, in a first stage, to classic perspective vision and then to more general cameras. Furthermore, this paper shows that the properties of event-based vision sensors allow the exploration of not-yet-defined geometric relationships, finally, we provide a definition of general epipolar geometry deployable to almost any visual sensor.

The four chips presented in the special session on "Activity-driven, event-based vision sensors" quickly output compressed digital data in the form of events. These sensors reduce redundancy and latency and increase dynamic range compared with conventional imagers. The digital sensor output is easily interfaced to conventional digital post processing, where it reduces the latency and cost of post processing compared to imagers. The asynchronous data could spawn a new area of DSP that breaks from conventional Nyquist rate signal processing. This paper reviews the rationale and history of this event-based approach, introduces sensor functionalities, and gives an overview of the papers in this session. The paper concludes with a brief discussion on open questions.

This paper presents a general purpose address-event (AER) processor based on a SPARC-compatible LEON3 core with a custom data interface for asynchronous sensor data. The main focus in the design of the sensor interface was on precisely maintaining the inherent timing information of AER sensor data while providing robust peak-rate handling, DMA functionality and a novel event-rate dependent system control mechanism. Hardware-accelerated event pre-processing includes pre-FIFO high-resolution time-stamping, address masking for ROI and event-rate dependent IRQ generation without loading the processor core. The System-on-Chip has been implemented in a 0.18μm CMOS process and achieves peak AER input event rates of 33M AE/s and sustained event rates of 5.125M AE/s at 10ns time-stamp resolution. The core processes AEs at >1M AE/s sustained rate. We discuss design considerations and implementation details and show measurement results from the fabricated chip.

This paper describes concept and implementation of an asynchronous, column-parallel readout method for event-based pulse-width-modulation (PWM) image sensors. These time-based imaging devices transmit exposure information in the form of asynchronous spike-events (AER) via an arbitrated asynchronous data bus that is common to all pixels. Event-collisions on the bus distort the time information and lead to errors in the instantaneous illumination measurement of the concerned pixel. The effect becomes manifest when imaging uniform, homogenous (parts of) scenes. One method to balance the load on the communication channel is to avoid a global reset signal. The proposed concept spreads the reset times according to the varying local illumination in the array after a global starting point, in fact realizing an asynchronous, column-parallel, light-dependent “rolling-shutter” mode. This novel concept has been realized in the design of a spiking asynchronous, time-based QVGA image sensor. We present theoretical considerations and preliminary measurement results from the chip fabricated in a standard 0.18μm CMOS process.

The presented asynchronous, time-based CMOS dynamic vision and image sensor is based on a QVGA (304×240) array of fully autonomous pixels containing event-based change detection and PWM imaging circuitry. Exposure measurements are initiated and carried out locally by the individual pixel that has detected a brightness change in its field-of-view. Thus pixels do not rely on external timing signals and independently and asynchronously request access to an (asynchronous arbitrated) output channel when they have new illumination values to communicate. Communication is address-event based (AER)-gray-levels are encoded in inter-event intervals. Pixels that are not stimulated visually do not produce output. This pixel-autonomous and massively parallel operation ideally results in optimal lossless video compression through complete temporal redundancy suppression at the focal-plane. Compression factors depend on scene activity. Due to the time-based encoding of the illumination information, very high dynamic range - intra-scene DR of 143dB static and 125dB at 30fps equivalent temporal resolution - is achieved. A novel time-domain correlated double sampling (TCDS) method yields array FPN of56dB (9.3bit) for >10Lx.

This live demonstration shows high DR, high temporal resolution, frame-free image/video acquisition based on asynchronous events. The presented camera features 9-bit gray-level imaging at up to 143dB DR and <;0.25% FPN with hardware-based lossless video compression and time-domain correlated double sampling. The main components of the camera - an asynchronous, time-based image sensor (ATIS) and a general purpose Address-Event processor with 20-Bit 10ns-resolution sensor data interface - have been specifically designed for the application. The presented system optimally combines the advantages of time-based (PWM) imaging, bio-inspired temporal contrast dynamic vision and event-based (AER) information encoding and data communication.

This paper presents a time-domain correlated double sampling (CDS) method for time-based/PWM image sensors. The concept has been realized in the pixel circuit design for a spiking asynchronous, time-based image sensor (ATIS). The pixel circuitry includes a two-stage voltage comparator with tunable hysteresis and dynamic current control, and pixel-level state logic. The sensor, based on a 240times304 pixel array, was implemented in a standard 0.18 mum CMOS process. We present measurements from the fabricated chip and compare them to results from theoretical considerations. Implications of the proposed CDS method on the comparator design in terms of chip area and power consumption are discussed and quantified.

This paper presents a 64times64 pixel micro-bolometer based temporal contrast IR sensor. The pixels of the sensor independently and asynchronously respond to changes in thermal IR radiation and communicate respective events via arbitrated AER. A 46 dB-gain, small-area, ultra low-power, low-mismatch differencing switched-capacitor amplifier has been designed for reading out the bolometer front-end. The chip has been fabricated in a 0.35 mum standard CMOS process and covers about 4times4 mm2of silicon area. An amorphous silicon (a-Si) microbolometer array has been processed on top and contacted to the pixel circuits. Measurement results of the ROIC and the bolometer element and first results of the final temporal contrast IR sensor are presented.

In this paper we propose a fully asynchronous, time- based image sensor, which is characterized by high temporal resolution, low data rate (near complete temporal redundancy suppression), high dynamic range, and low power consumption. Autonomous pixels asynchronously communicate the detection of relative changes in light intensity, and the time from change detection to the threshold crossing of a photocurrent integrator, so encoding the instantaneous pixel illumination shortly after the time of a detected change. The chip is being implemented in a standard 0.18 mum CMOS process and measures less than 10times8 mm2at 304times240 pixel resolution.

This paper presents a 2×256 pixel dual-line temporal contrast vision sensor and the use of this sensor in exemplary high-speed machine vision applications over a wide range of target illumination. The sensor combines an asynchronous, data-driven pixel circuit with an on-chip precision time-stamp generator and a 3-stage pipelined synchronous bus-arbiter. With a temporal resolution of down to 100ns, corresponding to a line rate of 10MHz, the sensor is ideal for high-speed machine vision tasks that do not rely on conventional image data. The output data rate depends on the dynamic contents of the target scene and is typically orders of magnitude lower than equivalent data output produced by conventional clocked line sensors in this type of applications. 120dB dynamic range makes high-speed operation possible at low lighting levels or uncontrolled lighting conditions. The sensor features two parallel pixel lines with a line separation of 250μm and a pixel pitch of 15μm. A prototype was fabricated in a standard 0.35μm CMOS technology. Results on high-speed edge angle resolution and edge gradient extraction as well as wide dynamic range operation are presented.

We present a 100dB dynamic range 2times64 pixel dual-line optical sensor with asynchronous event-based readout. Each individual pixel of the sensor operates autonomously and responds with <100mus latency to relative intensity changes. It operates largely independent of overall scene illumination, directly encodes object reflectance, and greatly reduces redundancy while preserving precise timing information. The line sensor was fabricated using a 0.35mum standard CMOS technology. Results of the measurement performance of the sensor are presented. The intended application area is precision timing measurement under variable lighting conditions