In dynamic environments where new concepts continuously emerge, Deep Neural Networks (DNNs) must adapt by learning new classes while retaining previously acquired ones. This challenge is addressed by Class-Incremental Learning (CIL). This paper introduces Generative Binary Memory (GBM), a novel CIL pseudo-replay approach which generates synthetic binary pseudo-exemplars. Relying on Bernoulli Mixture Models (BMMs), GBM effectively models the multi-modal characteristics of class distributions, in a latent, binary space. With a specifically-designed feature binarizer, our approach applies to any conventional DNN. GBM also natively supports Binary Neural Networks (BNNs) for highly-constrained model sizes in embedded systems. The experimental results demonstrate that GBM achieves higher than state-of-the-art average accuracy on CIFAR100 ( + 2.9 % ) and TinyImageNet ( + 1.5 % ) for a ResNet-18 equipped with our binarizer. GBM also outperforms emerging CIL methods for BNNs, with + 3.1 % in final accuracy and × 4.7 memory reduction, on CORE50.

The RAWtoRGB reconstruction involves rendering a high-quality color image from raw data acquired by an image sensor. Many recent approaches, including deep learning models to their training paradigms, exhibit limitations in terms of algorithm complexity or image quality due to unwanted artifacts. Existing solutions often inadvertently introduce visual distortions that are often overlooked and underrepresented in evaluation metrics. Furthermore, the challenge of balancing hardware portability and image quality is often underrated. This paper therefore introduces the Grid Regularization Enhancement Network (GRENet), a lightweight RAWtoRGB model with only 16.7k parameters. GRENet demonstrates high-end performance, achieving state-of-the-art PSNR scores of small models (20.63 dB on ZRR and 24.76 dB on MAI2021). A key innovation is a novel supervised training loss, termed Grid loss, specifically designed to mitigate checkerboard-alike artifacts. This loss function not only improves reference metrics, with a gain of +0.25 dB on MAI2021 PSNR but also significantly enhances the visual quality of rendered images. To ensure a comprehensive evaluation, we assess GRENet with a wide range of carefully selected no reference image quality metrics along with the full reference ones. In addition and unlike most of the prior work, we conduct an in-depth visual rendering inspection, comparing GRENet’s reconstructions with those of state-of-the-art models. This analysis reveals a taxonomy of image artifacts commonly observed in reconstructions, providing valuable insights into the strengths and limitations of current approaches. Our comparisons demonstrate that GRENet exhibits a high reliability in mitigating visual distortions among existing methods.

This paper proposes a deep learning model (RCNet) for Delta-Sigma (ΔΣ) ADCs. Recurrent Neural Networks (RNNs) allow to describe both modulators and filters. This analogy is applied to Incremental ADCs (IADC). High-end optimizers combined with full-custom losses are used to define additional hardware design constraints: quantized weights, signal saturation, temporal noise injection, devices area, capacitor mismatch, and finite-gain amplifier. Focusing on DC conversion, our early results demonstrate thatSNRdefined as an Effective Number Of Bits (ENOB) can be optimized under a certain hardware mapping complexity. The proposed RCNet succeeded to provide design tradeoffs in terms ofSNR(>13bit) versus area constraints (OSR(80 samples). Interestingly, it appears that the best RCNet architectures do not necessarily rely on high-order modulators, leveraging additional topology exploration degrees of freedom.

Binary Neural Networks (BNNs) are a promising approach to enable Artificial Neural Network (ANN) implementation on ultra-low power edge devices. Such devices may compute data in highly dynamic environments, in which the classes targeted for inference can evolve or even novel classes may appear, requiring continual learning. Class Incremental Learning (CIL) is an important type of continual learning for classification problems, yet it has been scarcely addressed in the context of BNNs. Furthermore, most of existing BNNs models are not fully binary, as they require several real-valued network layers, at the input, the output, and for batch normalization. This article goes a step further, enabling class incremental learning in Fully-Binarized NNs (FBNNs) through four main contributions. We firstly revisit the FBNN design and its training procedure that is suitable to CIL. Secondly, we explore loss balancing, a method to tradeoff the performance of past and current classes. Thirdly, we propose a semi-supervised method to pre-train the feature extractor of the FBNN for transferable representations. Fourthly, two conventional CIL methods, i . e ., Latent and Native replay, are thoroughly compared. These contributions are exemplified first on the CIFAR100 dataset, before being scaled up to the CORE50 continual learning benchmark. The final results based on our 3Mb FBNN on CORE50 , exhibit performance that is at par with, or better than conventional, larger, real-valued NN models.

This paper presents a near-imager inference hardware module enabling complex spatio-temporal pattern recognition (e.g., hand gesture or human fall). It relies on an algorithmic-architecture co-design approach leading to high accuracy at a low power consumption, optimized to handle raw data provided by an imager (row-by-row). Thanks to its 2-part deep learning model, leveraging both pipelined RTL design and near-SRAM computing, our 1Mb ASIC exhibits an estimated power consumption below 200µW at 20fps. Among our contributions is the definition (with its dedicated training) of fully binarized Gated Recurrent Units compatible with an optimized near-SRAM hardware.

Recent advances in Binary Neural Networks (BNNs) are opening up new possibilities for disruptive hardware accelerators. This paper extends prior work on incremental learning to BNNs, by proposing a specifically-designed fully-binarized net-work and evaluating it on two learning variants, i.e., native and latent replay. The proposed BNN achieves a 53.3% test accuracy on the CIFAR-100 benchmark while relying on a binary-only arithmetic, for a 4.1Mb model size. Given a class-incremental learning experimental setup, we evaluate the influence of replay buffer size on the strategy, highlighting a turning point where latent replay offers a better classification performance than Native replay. In addition, our approach exhibits robustness against a large number of successive retrainings with an accuracy always 10% higher than a full-precision counterpart.

Adjusting the quantization according to the data or to the model loss seems mandatory to enable a high accuracy in the context of quantized neural networks. This work presents Histogram-Equalized Quantization (HEQ), an adaptive framework for linear and symmetric quantization. HEQ automatically adapts the quantization thresholds using a unique step size optimization. We empirically show that HEQ achieves state-of-the-art performances on CFAR-10. Experiments on the STL-10 dataset even show that HEQ enables a proper training of our proposed logic-gated (OR, MUX) residual networks with a higher accuracy at a lower hardware complexity than previous work.

Even if Application-Specific Integrated Circuits (ASIC) have proven to be a relevant choice for integrating inference at the edge, they are often limited in terms of applicability. In this paper, we demonstrate that an ASIC neural network accelerator dedicated to image processing can be applied to multiple tasks of different levels: image classification and compression, while requiring a very limited hardware. The key component is a reconfigurable, mixed-precision (3b/2b/1b) encoder that takes advantage of proper weight and activation quantizations combined with convolutional layer structural pruning to lower hardware-related constraints (memory and computing). We introduce an automatic adaptation of linear symmetric quantizer scaling factors to perform quantized levels equalization, aiming at stabilizing quinary and ternary weights training. In addition, a proposed layer-shared Bit-Shift Normalization significantly simplifies the implementation of the hardware-expensive Batch Normalization. For a specific configuration in which the encoder design only requires 1Mb, the classification accuracy reaches 87.5% on CIFAR-10. Besides, we also show that this quantized encoder can be used to compress image patch-by-patch while the reconstruction can performed remotely, by a dedicated full-frame decoder. This solution typically enables an end-to-end compression almost without any block artifacts, outperforming patch-based state-of-the-art techniques employing a patch-constant bitrate.

This paper reports the main State-Of-The-Art algorithmic enablers for compact Neural Network topology design, while relying on basic numerical experiments. Embedding insensor intelligence to perform inference tasks generally requires a proper definition of a Neural Network architecture dedicated to specific purposes under Hardware limitations. Hardware design constraints known as power consumption, silicon surface, latency and maximum clock frequency cap available resources related to the topology, i.e., memory capacity and algorithmic complexity. We propose to categorize into 4 types the algorithmic enablers that force the hardware constraints as low as possible while keeping the accuracy as high as possible. First, Dimensionality Reduction (DR) is used to reduce memory needs thanks to predefined, hardware-coded patterns. Secondly, low-precision Quantization with Normalization (QN) can both simplify hardware components as well as limiting overall data storage. Thirdly, Connectivity Pruning (CP) involves an improvement against over-fitting while limiting needless computations. Finally, during the inference at the feed-forward pass, a Dynamical Selective Execution (DSE) of topology parts can be performed to limit the activation of the entire topology, therefore reducing the overall power consumption.

This paper presents a new compressive sensing acquisition scheme well adapted for highly constrained hardware implementations. The proposed sensing model being basically designed to meet both theoretical (i.e., Restricted Isometry Property) and hardware requirements (i.e., power consumption, silicon footprint), is highly suitable for image sensors applications addressing both image rendering and embedded decision making tasks. In fact, for a pixels array, the proposed framework consists in applying for each row a random modulation ±1 and a random permutation of the pixels, and then averaging the outputs by column to extract a compressed vector. This model is shown to be relevant as it has the same theoretical performance as a randomly generated sensing scheme as well as a low silicon footprint for physical implementation. Various numerical results and a discussion on possible implementations will be presented to show the robustness and the efficiency of the proposed model.

This work presents a compact VGA (480 × 640) CMOS Image Sensor (CIS) architecture with dedicated end-of-column Compressive Sensing (CS) scheme allowing embedded object recognition. The architecture takes advantage of a low-footprint pseudo-random data mixing circuit and a first order incremental Sigma-Delta (ΣΔ) Analog to Digital Converter (ADC) to extract compressed features. The proposed CIS achieves an object recognition accuracy of ≃ 93% on the Georgia Tech face recognition database (GIT, 10 classes out of 50) thanks to a linear Support Vector Machine (SVM) classifier implemented by an optimized Digital Signal Processing (DSP). We stress that the signal independent dimensionality reduction performed by our dedicated CS scheme (1/480) allows to dramatically reduce memory requirements (≈ 32 kbit) related -in our case- to the ex-situ learned affine function of the linear SVM.

This paper explores hierarchical clustering methods to learn a hierarchical multi-class classifier on compressed measurements in the context of highly constrained hardware (e.g., always-on ultra low power vision systems). In contrast to the popular multi-class classification approaches based on multiple binary classifiers (i.e., one-vs.-all and one-vs.one [1]), a hierarchical classifier requires only O(log2C) binary classifiers in a decision tree. In this work, we investigate three clustering methods used to construct balanced clusters at each node thus reducing the depth of the decision tree in order to lower hardware requirements to its minimum. A binary Support Vector Machine (SVM) [2] classifier is then learned on Compressive Sensing measurements [3] at each node of the hierarchical tree. Our results, based on two object recognition databases (AT&T and COIL-100 databases), show the competitiveness of hierarchical classification in terms of hardware requirements (lower memory and computational complexity) as well as its classification accuracy.

This paper discusses the impact of Fixed Pattern Noise (FPN) on image compression embedded in CMOS sensors. This FPN is mainly due to technology dispersions as well as a non uniform topology of the layout. FPN is generally modelled as a non uniform affine mapping of the pixels. The Dark Signal Non-Uniformity (DSNU) represents the level of offset variations while Photo Response Non-Uniformity (PRNU) is the level of pixel gain variations. Once those two parameters are properly calibrated for each pixel, the image can be corrected by linear interpolations (i.e. affine correction). This operation requires specific processing and large memory resources to be implemented directly in the focal plane. If this 2-points correction is not performed inside the sensor, some problems arise in the context of embedded compression because the hypothesis on image sparsity is no longer satisfied. By considering this context, this paper presents a comparative study between a 2D Haar wavelet compression, separable Compressive Sensing (CS) and a novel approach combining wavelet based compression and row based CS. In the case of a high non uniformity of the focal plane (e.g. considerable technological dispersion), it appears that a proper combination of CS and Haar takes advantage of both enabling FPN correction at the reconstruction stage and outperforming the alternative techniques.

In this paper is presented a High Dynamic Range (HDR) extension technique that applies to an imager without modifying any of its other specifications (as speed, noise floor or pixel scheme). The technique relies on the division of the focal plane into blocks that are able to choose the integration time from a set of eleven exposures, reaching +60db extension compared to a standard CMOS imager. The exposure time of each block can also be controlled by an external frame buffer, paving the way to advanced bracketing techniques. This system has been explored with a proof of concept sensor fabricated in a 0.18μm CMOS process.