International audience

Event-based cameras, with their high temporal resolution and low energy consumption, offer significant advantages over conventional cameras for computer vision tasks. However, current semantic segmentation algorithms for event-based data face challenges in achieving optimal performance for two main reasons: 1) the mismatch between sparse event streams (even when encoded into pseudo-frames) and the dense data structure of traditional frame-based images for which most existing implementations were designed, and 2) the lack of texture information in events, which solely detect temporal variations in brightness. To address this challenge, we propose a novel Cross-Modal (CM) Knowledge Distillation (KD) approach. Our method transfers knowledge from a high-performing Artificial Neural Network (ANN) processing fused grey-scale images and events to a Spiking Neural Network (SNN) - a bio-inspired, energy-efficient computing paradigm - operating on event data alone. Experiments on the DDD17 and DSEC-semantic datasets demonstrate that our approach significantly improves semantic segmentation results while reducing SNN energy consumption. This work represents the first application of cross-modal knowledge distillation to neuromorphic semantic segmentation, paving the way for more efficient event-based vision systems.

The complexity of event-based object detection (OD) poses considerable challenges. Spiking Neural Networks (SNNs) show promising results and pave the way for efficient event-based OD. Despite this success, the path to efficient SNNs on embedded devices remains a challenge. This is due to the size of the networks required to accomplish the task and the ability of devices to take advantage of SNNs benefits. Even when "edge" devices are considered, they typically use embedded GPUs that consume tens of watts. In response to these challenges, our research introduces an embedded neuromorphic testbench that utilizes the SPiking Low-power Event-based ArchiTecture (SPLEAT) accelerator. Using an extended version of the Qualia framework, we can train, evaluate, quantize, and deploy spiking neural networks on an FPGA implementation of SPLEAT. We used this testbench to load a state-of-the-art SNN solution, estimate the performance loss associated with deploying the network on dedicated hardware, and run real-world event-based OD on neuromorphic hardware specifically designed for low-power spiking neural networks. Remarkably, our embedded spiking solution, which includes a model with 1.08 million parameters, operates efficiently with 490 mJ per prediction.

In the recent years, Spiking Neural Networks have gain much attention from the research community. They can now be trained using the powerful gradient descent and have drifted from the neuroscience to the Machine Learning community. An abundant literature shows that they can perform well on classical Artificial Intelligence tasks such as image or signal classification while consuming less energy than state-of-the-art models like Convolutional Neural Networks. Yet, there is very little work about their performance on unsupervised anomaly detection and time-series prediction. Indeed, the processing of such temporal data requires different encoding and decoding mechanisms and rises questions about their capacity to model a dynamical signal with long term temporal dependencies. In this paper, we propose for the first time a Sparse Recurrent Spiking Neural Network with specific encoding and decoding mechanisms to successfully predict time-series and do Unsupervised Anomaly Detection. We also provide a framework to describe in detail our model computational costs and fairly compare them with state-of-the-art models. Despite improvable performances, we show that our model perform well on these tasks and open a door for further studies of such applications for Spiking Neural Networks.

Recent progress in the fields of AI and cognitive sciences opens up new challenges and problems that were previously inaccessible to study. One of such modern tasks is recovering lost data of one modality by using the data from another one. A similar effect (called the McGurk Effect) has been found in the functioning of the human brain. Observing this effect, one modality of information interferes with another, changing its perception. In this paper, we propose a way to reproduce such an effect and use it to reconstruct lost data modalities by combining Variational Auto-Encoders, Self-Organizing Maps, and Hebb connections in a unified ReD-SOM (Reentering Deep Self-organizing Map) model. We are inspired by human's capability to use different zones of the brain in different modalities, in case of having a lack of information in one of the modalities. This new approach not only improves the analysis of ambiguous data but also restores the intended signal. The results obtained on the multimodal dataset show an increase of quality of the signal reconstruction. The effect is remarkable both visually and quantitatively, specifically in presence of a significant degree of signal's distortion.

In this paper, we present SPLEAT, a SPiking Low-power Event-based ArchiTecture for the hardware deployment of Spiking Neural Networks (SNN). SPLEAT is designed as a configurable architecture that allows integrating several hardware modules representing different neural layers and, therefore, gives the ability to deploy a wide range of deep convolutional SNN topologies. Thanks to its event-based structure, SPLEAT takes advantage of the asynchronous spiking behavior to realize prediction efficiently. SPLEAT has been synthesized on a Cyclone V Field-Programmable Gate Array (FPGA) embedded on OPSSAT, a 3U nano-satellite launched on December 18, 2019 by the European Space Agency (ESA). SPLEAT has been used to perform in-orbit binary cloud classification on images provided by a 50 meters resolution sensor, which is still operational to this date. A second goal of the in-orbit experiment was to confront bio-inspired AI with classical deep learning. Consequently, in addition to SNNs executed using SPLEAT, standard Convolutional Neural Networks (CNNs) have been deployed on the same hardware target and evaluated on the same satellite images. The comparison between these neural network accelerators reveal a notable gain in terms of resources occupation for SPLEAT, reducing them by a factor of 6.62 for equivalent classification performances and an FPGA power consumption reduced by a factor of 5.91. To our best knowledge, this in-orbit experiment constitutes a world premiere in the fields of bio-inspired neural networks and aerospace.

Automotive embedded algorithms have very high constraints in terms of latency, accuracy and power consumption. In this work, we propose to train spiking neural networks (SNNs) directly on data coming from event cameras to design fast and efficient automotive embedded applications. Indeed, SNNs are more biologically realistic neural networks where neurons communicate using discrete and asynchronous spikes, a naturally energy-efficient and hardware friendly operating mode. Event data, which are binary and sparse in space and time, are therefore the ideal input for spiking neural networks. But to date, their performance was insufficient for automotive real-world problems, such as detecting complex objects in an uncontrolled environment. To address this issue, we took advantage of the latest advancements in matter of spike backpropagation - surrogate gradient learning, parametric LIF, SpikingJelly framework - and of our new voxel cube event encoding to train 4 different SNNs based on popular deep learning networks: SqueezeNet, VGG, MobileNet, and DenseNet. As a result, we managed to increase the size and the complexity of SNNs usually considered in the literature. In this paper, we conducted experiments on two automotive event datasets, establishing new state-of-the-art classification results for spiking neural networks. Based on these results, we combined our SNNs with SSD to propose the first spiking neural networks capable of performing object detection on the complex GEN1 Automotive Detection event dataset.

Spiking neural networks are expected to bring high resources, power, and energy efficiency to machine learning hardware implementations. In this regard, they could facilitate the integration of Artificial Intelligence in highly constrained embedded systems, such as image classification in drones or satellites. If their logic resource efficiency is widely accepted in the literature, their energy efficiency still remains debated. In this article, a novel high-level metric is used to characterize the expected energy efficiency gain when using Spiking Neural Networks (SNN) instead of Formal Neural Networks (FNN) for hardware implementation: Synaptic Activity Ratio (SAR). This metric is applied to a selection of classification tasks including images and 1D signals. Moreover, a high-level estimator for logic resources, power usage, execution time, and energy is introduced for neural network hardware implementations on FPGA, based on four existing accelerator architectures covering both sequential and parallel implementation paradigms for both spiking and formal coding domains. This estimator is used to evaluate the reliability of the Synaptic Activity Ratio metric to characterize spiking neural network energy efficiency gain on the proposed dataset benchmark. This study led to the conclusion that spiking domain offers significant power and energy savings in sequential implementations. This study also shows that synaptic activity is a critical factor that must be taken into account when addressing low-energy systems.

Convolutional neural networks (CNNs) are now the de facto solution for computer vision problems thanks to their impressive results and ease of learning. These networks are composed of layers of connected units called artificial neurons, loosely modeling the neurons in a biological brain. However, their implementation on conventional hardware (CPU/GPU) results in high power consumption, making their integration on embedded systems difficult. In a car for example, embedded algorithms have very high constraints in term of energy, latency and accuracy. To design more efficient computer vision algorithms, we propose to follow an end-to-end biologically inspired approach using event cameras and spiking neural networks (SNNs). Event cameras output asynchronous and sparse events, providing an incredibly efficient data source, but processing these events with synchronous and dense algorithms such as CNNs does not yield any significant benefits. To address this limitation, we use spiking neural networks (SNNs), which are more biologically realistic neural networks where units communicate using discrete spikes. Due to the nature of their operations, they are hardware friendly and energy-efficient, but training them still remains a challenge. Our method enables the training of sparse spiking convolutional neural networks directly on event data, using the popular deep learning framework PyTorch. The performances in terms of accuracy, sparsity and training time on the popular DVS128 Gesture Dataset make it possible to use this bio-inspired approach for the future embedding of real-time applications on low-power neuromorphic hardware.

Bringing artificial intelligence to embedded devices has become a central research topic in many scientific domains (environment, agriculture, sociology, health...). For Human Activity Recognition, Artificial Neural Networks (ANNs) have shown their capability to provide better performance compared to other machine learning methods. However, ANNs suffer from two major limitations. First, ANNs are often trained using supervised learning requiring labelled databases, which are often difficult to build in real applications. Then, those algorithms are usually very expensive in terms of computing power. For that reason, their integration into low-power microcontrollers has been so far only evaluated to a limited extent. In this paper, we propose to evaluate quantitatively and qualitatively the embedded implementation of different neural networks for human activity recognition. First, supervised learning approaches are presented, followed by an exploratory study of unsupervised learning approaches using Self-Organizing Maps. Finally, some aspects of embedded unsupervised online learning are investigated to improve classification results using subject-specific data over a more general training. Each neural network is tested on a Human Activity Recognition dataset acquired from a smartphone using accelerometer and gyroscope sensing information (UCI HAR) and deployed on the SparkFun Edge board. This board hosts a low-power ARM Cortex-M4F-based microcontroller.

Recent literature considers that Spiking Neural Networks are now a serious alternative to Formal Neural Networks for embedded artificial intelligence. The changes in the information coding and the elementary neural computation make them more efficient than FNNs in terms of power consumption and chip surface occupation. However, these results are often based on simple neural network topologies with basic data-sets. In this paper, we study the behavior of Spiking Convolutional Neural Networks when applied to two different classification tasks. To do so, we analyze the spiking activity on both MNIST and GTSRB data-sets using different rate-based and temporal coding schemes. Notably, the Spike Select method is confronted to First Spike and Jittered Periodic methods in terms of prediction accuracy and spiking activity. Finally, we conclude about spike generation within spiking CNNs for embedded hardware classification.

Spiking Neural Networks are well known for their hardware-friendliness, as their implementation enables important hardware resources and energy savings when compared to classical Neural Network accelerators. Those advantages are mostly due to their simple Integrate & Fire neuron model, and the event-based aspect of their computation. On the other hand, convolution layers and maxpooling layers enable robust feature extraction, thus yielding very high recognition accuracy in image classification applications. Unfortunately, those layer models are not well-suited to the spiking model when dealing with static image classification on FPGA. Based on this statement, we developed an innovative hybrid Neural Network embedded accelerator. Our architecture interfaces classical non-spiking convolution Neural network for feature extraction; and spiking Dense Layers for classification. The implementation showed recognition performances (87%) equivalent to its classical counterpart (88%), while reducing hardware resource intensiveness of the classification stage (12% less occupied logic cells in total, including 60% reduction on the classification stage).

Inspired from the brain, neuromorphic computing would be the right alternative to traditional Von-Neumann architecture computing that knows its end of growth as predicted by Moore's law. In this paper, we explore bio-inspired neural networks as an AI-accelerator for embedded systems. To do so, we first map neural networks from formal to spiking domain, then choose the information coding method resulting in better performances. Afterwards, we present the design of two different hardware architectures: time-multiplexed and fully-parallel. Finally, we compare their performances and their hardware cost to select at the end the adequate architecture and conclude about spike-based neural networks as a potential solution for embedded artificial intelligence applications.

During the last years, Deep Neural Networks have reached the highest performances in image classification. Nevertheless, such a success is mostly based on supervised and off-line learning: they require thus huge labeled datasets for learning, and once it is done, they cannot adapt to any change in the data from the environment. In the context of brain-inspired computing, we apply Kohonen-based Self-Organizing Maps for unsupervised learning without labels, and we explore original extensions such as the Dynamic SOM that enables continuous learning and the Pruning Cellular SOM that includes synaptic pruning in neuromorphic circuits. After presenting the three models and the experimental setup for MNIST classification, we compare different methods for automatic labeling based on very few labeled data (1% of the training dataset), and then we compare the performances of the three Kohonen-based SelfOrganizing Maps with STDP-based Spiking Neural Networks in terms of accuracy, dynamicity and scalability.

Artificial neural networks are experiencing today an unprecedented interest thanks to two main changes: the explosion of open data that is necessary for their training, and the increasing computing power of today's computers that makes the training part possible in a reasonable time. The recent results of deep neural networks on image classification has given neural networks the leading role in machine learning algorithms and artificial intelligence research. However, most applications such as smart devices or autonomous vehicles require an embedded implementation of neural networks. Their implementation in CPU/GPU remains too expensive, mostly in energy consumption, due to the non-adaptation of the hardware to the computation model, which becomes a limit to their use. It is therefore necessary to design neuromorphic architectures, i.e. hardware accelerators that fit to the parallel and distributed computation paradigm of neural networks for reducing their hardware cost implementation. We mainly focus on the optimization of energy consumption to enable integration in embedded systems. For this purpose, we implement two models of artificial neural networks coming from two different scientific domains: the multi- layer perceptron derived from machine learning and the spiking neural network inspired from neuroscience. We compare the performances of both approaches in terms of accuracy and hardware cost to find out the most attractive architecture for the design of embedded artificial intelligence.

Self-organization is a powerful bio-inspired feature that has been poorly investigated in the context of hardware computing architectures. The neural foundations of brain self-organization enable structural plasticity, continuous learning, and tolerance to lesions but at the cost of massive lateral synaptic connections. A cellular formulation of the related neural models would be able to tackle this limitation by iterating the propagation of the information in the network. This paper introduces the SOMA hardware architecture, which relies on this principle to develop a Self-Organizing Machine Architecture. We show the benefits of our cellular implementation compared to the existing neural models. This contribution leads to a hardware compliant algorithm thanks to decentralized operations and communications. A generalization of the technique is proposed to implement different Kohonen-like self-organizing models. This particular cellular implementation reaches the same behavior as the centralized models but drastically reduces their computing complexity. We present the performance results and discuss the benefits of cellular computing approaches in the perspective of designing hardware neural processors for embedded applications.

This paper provides an overview of the challenges faced by hardware implemented Spiking Neural Networks, from device to circuit design, reliability and test. We present a comprehensive description of the state-of-the-art neuromorphic architectures inspired by brain computation, with special emphasis on Spiking Neural Networks (SNNs), together with emerging technologies that have enabled such systems, namely Phase Change and Metal Oxide Resistive Memories. Finally, we discuss the main challenges faced by hardware implementations of SNNs, their reliability and post-fabrication test issues.

Neurobiological systems have often been a source of inspiration for computational science and engineering, but in the past their impact has also been limited by the understanding of biological models. Today, new technologies lead to an equilibrium situation where powerful and complex computers bring new biological knowledge of the brain behavior. At this point, we possess sufficient understanding to both imagine new brain-inspired computing paradigms and to sustain a classical paradigm which reaches its end programming and intellectual limitations. In this context, we propose to reconsider the computation problem first in the specific domain of mobile robotics. Our main proposal consists in considering computation as part of a global adaptive system, composed of sensors, actuators, a source of energy and a controlling unit. During the adaptation process, the proposed brain-inspired computing structure does not only execute the tasks of the application but also reacts to the external stimulation and acts on the emergent behavior of the system. This approach is inspired by cortical plasticity in mammalian brains and suggests developing the computation architecture along the system's experience. This article proposes modeling this plasticity as a problem of estimating a probability density function. This function would correspond to the nature and the richness of the environment perceived through multiple modalities. We define and develop a novel neural model solving the problem in a distributed and sparse manner. And we integrate this neural map into a bio-inspired hardware substrate that brings the plasticity property into parallel many-core architectures. The approach is then called Hardware Plasticity. The results show that the self-organization properties of our model solve the problem of multimodal sensory data clusterization. The properties of the proposed model allow envisaging the deployment of this adaptation layer into hardware architectures embedded into the robot's body in order to build intelligent controllers.

Recent FPGAs allow the design of efficient and complex Heterogeneous Systems-on-Chip (HSoC). Namely, these systems are composed of several processors, hardware accelerators as well as communication media between all these components. Performances provided by HSoCs make them really interesting for data-flow applications, especially image processing applications. The use of this kind of architecture provides good performances but the drawback is an increase of the programming complexity. This complexity is due to the heterogeneous deployment of the application on the platform. Some functions are implemented in software to run on a processor, whereas other functions are implemented in hardware to run in a reconfigurable partition of the FPGA. This article aims to define a programming model based on the Synchronous Data-Flow model, in order to abstract the heterogeneity of the implementation and to leverage the communication issue between software and hardware actors.

By incorporating reconfigurable hardware in embedded system architectures it has become easier to satisfy the performance constraints of demanding applications while lowering system cost. In order to evaluate the performance of a candidate architecture, the nodes (tasks) of the data flow graphs that describe an application must be assigned to the computing resources of the architecture: programmable processors and reconfigurable FPGA, whose run-time reconfiguration capabilities must be exploited. In this paper we present a novel design exploration tool - based on a local search algorithm with global convergence properties - which simultaneously explores choices for computing resources, assignments of nodes to these resources, task schedules on the programmable processors and context definitions for the reconfigurable circuits. The tool finds a solution that minimizes system cost while meeting the performance constraints; more precisely it lets the designer select the quality of the optimization (hence its computing time) and finds accordingly a solution with close-to-minimal cost.

In current embedded system design practice, only few architectural solutions and mappings of the functionalities of a system on the architecture's components are examined. This paper presents an optimization-based method and the associated tool developed to help designers take architectural decisions. The principle of this approach is to efficiently explore the design space and to dynamically provide the user with the capabilities to visualize the evolution of selected criteria. The first objective is solved by developing an enhanced version of the adaptive simulated annealing algorithm. Since the method is iterative, multiple solutions may be examined and the tool lets the user stop exploration at any time, tune parameters and select solutions. Moreover we present an approach for systems whose functionalities are specified by means of multiple models of computation, in order to handle descriptions of digital signal applications at several levels of detail. The tool has been applied to a motion detection application in order to determine architectural parameters