In dynamic environments where new concepts continuously emerge, Deep Neural Networks (DNNs) must adapt by learning new classes while retaining previously acquired ones. This challenge is addressed by Class-Incremental Learning (CIL). This paper introduces Generative Binary Memory (GBM), a novel CIL pseudo-replay approach which generates synthetic binary pseudo-exemplars. Relying on Bernoulli Mixture Models (BMMs), GBM effectively models the multi-modal characteristics of class distributions, in a latent, binary space. With a specifically-designed feature binarizer, our approach applies to any conventional DNN. GBM also natively supports Binary Neural Networks (BNNs) for highly-constrained model sizes in embedded systems. The experimental results demonstrate that GBM achieves higher than state-of-the-art average accuracy on CIFAR100 ( + 2.9 % ) and TinyImageNet ( + 1.5 % ) for a ResNet-18 equipped with our binarizer. GBM also outperforms emerging CIL methods for BNNs, with + 3.1 % in final accuracy and × 4.7 memory reduction, on CORE50.

Binary Neural Networks (BNNs) are a promising approach to enable Artificial Neural Network (ANN) implementation on ultra-low power edge devices. Such devices may compute data in highly dynamic environments, in which the classes targeted for inference can evolve or even novel classes may appear, requiring continual learning. Class Incremental Learning (CIL) is an important type of continual learning for classification problems, yet it has been scarcely addressed in the context of BNNs. Furthermore, most of existing BNNs models are not fully binary, as they require several real-valued network layers, at the input, the output, and for batch normalization. This article goes a step further, enabling class incremental learning in Fully-Binarized NNs (FBNNs) through four main contributions. We firstly revisit the FBNN design and its training procedure that is suitable to CIL. Secondly, we explore loss balancing, a method to tradeoff the performance of past and current classes. Thirdly, we propose a semi-supervised method to pre-train the feature extractor of the FBNN for transferable representations. Fourthly, two conventional CIL methods, i . e ., Latent and Native replay, are thoroughly compared. These contributions are exemplified first on the CIFAR100 dataset, before being scaled up to the CORE50 continual learning benchmark. The final results based on our 3Mb FBNN on CORE50 , exhibit performance that is at par with, or better than conventional, larger, real-valued NN models.

Recent advances in Binary Neural Networks (BNNs) are opening up new possibilities for disruptive hardware accelerators. This paper extends prior work on incremental learning to BNNs, by proposing a specifically-designed fully-binarized net-work and evaluating it on two learning variants, i.e., native and latent replay. The proposed BNN achieves a 53.3% test accuracy on the CIFAR-100 benchmark while relying on a binary-only arithmetic, for a 4.1Mb model size. Given a class-incremental learning experimental setup, we evaluate the influence of replay buffer size on the strategy, highlighting a turning point where latent replay offers a better classification performance than Native replay. In addition, our approach exhibits robustness against a large number of successive retrainings with an accuracy always 10% higher than a full-precision counterpart.

This paper explores the combination of neural network quantization and entropy coding for memory footprint minimization. Edge deployment of quantized models is hampered by the harsh Pareto frontier of the accuracy-to-bitwidth tradeoff, causing dramatic accuracy loss below a certain bitwidth. This accuracy loss can be alleviated thanks to mixed precision quantization, allowing for more flexible bitwidth allocation. However, standard mixed precision benefits remain limited due to the 1-bit frontier, that forces each parameter to be encoded on at least 1 bit of data. This paper introduces an approach that combines mixed precision, zero-point quantization and entropy coding to push the compression boundary of Resnets beyond the 1-bit frontier with an accuracy drop below 1% on the ImageNet benchmark. From an implementation standpoint, a compact decoder architecture features reduced latency, thus allowing for inference-compatible decoding.

Post-training quantization of neural networks consists in quantizing a model without retraining nor hyperparameter search, while being fast and data frugal. In this paper, we propose LatticeQ, a novel post-training weight quantization method designed for deep convolutional neural networks (DC-NNs). Contrary to scalar rounding widely used in state-of-the-art quantization methods, LatticeQ uses a quantizer based on lattices - discrete algebraic structures. LatticeQ exploits the inner correlations between the model parameters to the benefit of minimizing quantization error. We achieve state-of-the-art results in post-training quantization. In particular, we achieve ImageNet classification results close to full precision on Resnet-18/50, with little to no accuracy drop for 4-bit models. Our code is available here, and a more thorough version of the paper here.

The increasing need for intelligent sensors in a wide range of everyday objects requires the existence of low power information processing systems which can operate autonomously in their environment. In particular, merging and processing the outputs of different sensors efficiently is a necessary requirement for mobile agents with cognitive abilities. In this work, we present a multi-layer spiking neural network for inference of relations between stimuli patterns in dedicated neuromorphic systems. The system is trained with a new version of the backpropagation algorithm adapted to on-chip learning in neuromorphic hardware: Error gradients are encoded as spike signals which are propagated through symmetric synapses, using the same integrate-and-fire hardware infrastructure as used during forward propagation. We demonstrate the strength of the approach on an arithmetic relation inference task and on visual XOR on the MNIST dataset. Compared to previous, biologically-inspired implementations of networks for learning and inference of relations, our approach is able to achieve better performance with less neurons. Our architecture is the first spiking neural network architecture with on-chip learning capabilities, which is able to perform relational inference on complex visual stimuli. These features make our system interesting for sensor fusion applications and embedded learning in autonomous neuromorphic agents.

We introduce a deep spiking convolutional neural network of integrate-and-fire (IF) neurons, which extracts hierarchical features from a stream of event-based vision data in an unsupervised fashion and online. Our network operates with a simple spike-timing dependent plasticity (STDP) rule, which does not require the definition of an input timescale. We demonstrate how our network is able to learn translational invariant features from the event-based N-MNIST dataset, while preserving dynamic information in the data. We demonstrate how these features can be used to perform online classification. Our network is the first neuromorphic system which can be used to learn complex hierarchical features unsupervised from a continuous stream of address-event-representation (AER) data, operating outside of a database framework and on multiple timescales. Additionally, all the mechanisms we use are simple and generic. This could open the possibility to implement our system on current neuromorphic hardware, to build a real-world fully adaptive event-based vision system.

In this paper is presented a High Dynamic Range (HDR) extension technique that applies to an imager without modifying any of its other specifications (as speed, noise floor or pixel scheme). The technique relies on the division of the focal plane into blocks that are able to choose the integration time from a set of eleven exposures, reaching +60db extension compared to a standard CMOS imager. The exposure time of each block can also be controlled by an external frame buffer, paving the way to advanced bracketing techniques. This system has been explored with a proof of concept sensor fabricated in a 0.18μm CMOS process.

In this paper we present an effective method for super resolution (SR). The proposed method is inspired by the Non-Local Means (NLM) algorithm but shows a lot of improvements over it. Firstly, we show the sensitivity to the standard deviation parameter of the NLM algorithm in detail regions that has low contrast. This analysis leads to an improvement of the NLM algorithm by selecting the best neighbors for each pixel in the SR image to be reconstructed. Secondly, this novel selection of neighbors is combined with a segmentation step in order to build a SR framework that can be spatially adapted to contrast. The proposed method allows reconstruction of SR images that conserves low-contrast detail while ensuring noise canceling.

A smart ultra-low power CMOS image sensor comprising an analog programmable processor array is reported. Compact and efficient motion detection algorithms are implemented to process sub-sampled images made of so-called macropixels. Only Regions of Interest (ROI) consisting of macropixels containing moving objects are read out. This drastically reduces power consumption: the 110×240 pixel image sensor fabricated in a 0.35μm technology features a power consumption of 120μW at 25fps.

In a classical video monitoring system, though for most of the time the captured images contain no relevant information, it cannot prevent the monitoring system from useless power consuming for image processing. To improve this, one technique proved to be effective for a video monitoring system to reduce its power consumption is based on macro-pixels or blocks of pixels for region of interest (ROI) finding. The key feature necessary for ROI finding corresponds to motion detection. To be able to implement ROI detection, two CMOS-based motion detection circuits are proposed in this paper. Both are designed and optimized to have fewer transistors' number, lower power consumption, higher sensitivity and better uniformity detection within the designed input common mode range.

This paper presents the evolutions of CMOS image sensors. From the early works, highly image processing oriented, the main research effort has then emphasized on image acquisition. To overcome the rising limitations of standard approaches and to promote new functionalities, several research directions are underway with promising results.

This article introduces a new hierarchical version of a set of motion detection algorithms called ¿¿. These new algorithms are designed to preserve as much as possible the computational efficiency of the basic ¿¿ estimation, in order to target real-time implementation for low power consumption processors and embedded systems.

To be implemented on an analog CMOS image sensor, a robust algorithm based on recursive operations is presented. It allows sensor's acuity adaptation to the scene activity. The main interest of the presented motion detection with adaptive thresholding is that, in a context of embedded steady camera, such a system allows focusing on targets with high resolution while keeping background in low resolution. Drastic power consumption reduction is achieved by tremendously reducing the amount of processed data.

Based on domino-like asynchronous propagation an original and efficient way of reading 2-D binary arrays is proposed. The functional principle is first described and a hardware implementation is proposed. Both speed-up and energy consumption for typical applications are analyzed