Embedded systems based on Microcontroller Units (MCUs) often gather significant quantities of data and solve various issues. Convolutional Neural Networks (CNNs) have proven their effectiveness in solving computer vision and natural language processing tasks. However, implementing CNNs within MCUs is challenging due to their high inference costs, which varies widely depending on hardware targets and CNN topologies. Despite state-of-the-art advancements, no efficient design space exploration solutions handle the wide variety of implementation solutions. In this article, we introduce the ZIP-CNN design space exploration methodology, which facilitates CNN implementation within MCUs. We developed a model that quantitatively estimates the latency, energy consumption, and memory space required to run a CNN within an MCU. This model accounts for algorithmic reductions such as knowledge distillation, pruning, or quantization and applies to any CNN topology. To demonstrate the efficiency of our methodology, we investigated LeNet5, ResNet8, and ResNet26 within three different MCUs. We made materials and supplementary results available in a GitHub repository: https://github.com/ThGbay/ZIP-CNN . The proposed method was empirically verified on three hardware targets running at 14 different operating frequencies. The three CNN topologies investigated were implemented in their default configuration in FP32, and also reduced with INT8 quantization, pruning at five different rates and with knowledge distillation. The estimates of our model are very reliable with an error of 3.29% to 15.23% for latency, 3.12% to 10.34% for energy consumption, and 1.95% to 6.31% for memory space. These results are based on on-device measurements.

In the helicopter industry, estimating the flight loads acting on mechanical components is a prerequisite for fatigue computation, and maintenance work. Currently, the flight loads assessments are done during the development phase by the manufacturer, in line with the certification guidance material specifying that maintenance intervals be set – at fleet level - to an assumed usage that is “as severe as expected in service”. This paper aims at introducing a predictive maintenance approach, based on the real usage of each helicopter, where a reliable estimation of the flight loads could help to derive the damage of the helicopter components and therefore determine when to replace them. To this end, a study is conducted to link the quasi-static evolution of the flight parameters to the dynamic evolution of the flight loads. The proposed methodology is based on the locally periodic variations of the flight loads, by applying a harmonic decomposition over each period and by extracting their real and imaginary parts. With this decomposition, different types of Machine Learning models have been compared to predict each harmonic part. To evaluate the approach, the load is finally reconstructed using the predictions and compared to the original signal. The application on real data from proprietary databases shows the relevance of the approach. As locally periodic signals are represented in many domains, this approach could therefore be useful in other fields such as healthcare or the automotive industry, where signal estimation can be crucial.

Joint sparse representation (JSR) model has recently emerged as a powerful technique with wide variety of applications. In this paper, the JSR model is extended to error concealment (EC) application, being effective to recover the original image from its corrupted version. This model is based on jointly learning a dictionary pair and two mapping matrices that are trained offline from external training images. Given the trained dictionaries and mappings, the restoration is done by transferring the recovery problem into the sparse representation domain with respect to the trained dictionaries, which is further transformed into a common space using the respective mapping matrices. Then, the reconstructed image is obtained by back projection into the spatial domain. In order to improve the accuracy and stability of the proposed JSR-based EC algorithm and avoid unexpected artifacts, the local and non-local priors are seamlessly integrated into the JSR model. The non-local prior is based on the self-similarity within natural images and helps to find an accurate sparse representation by taking a weighted average of similar areas throughout the image. The local prior is based on learning the local structural regularity of the natural images and helps to regularize the sparse representation, exploiting the strong correlation in the small local areas within the image. Compared with the state-of-the-art EC algorithms, the results show that the proposed method has better reconstruction performance in terms of objective and subjective evaluations.

Image and video transmission over heterogeneous networks may encounter packet loss due to the channel impairments, leading to quality degradation of the received image. In this paper, a novel robust image transmission system is proposed by casting the error concealment challenge into a sparse recovery framework. To this purpose, a robust encoder is carefully designed in order to mitigate the negative effects of the packet loss. After wavelet decomposition, a quadtree structure of the wavelet coefficients is used to rearrange them into independent partitions. Random linear combinations of coefficients for each partition are then adopted to provide a high error recovery capability. This linear process coupled with a simple packetization process introduces more robustness and error resilience into the transmission system. At the receiver side, the tree-sparse structure of the wavelet coefficients is explicitly exploited in order to model the error recovery problem as a sparse recovery framework. This is achieved by adaptation of a well-known iterative sparse reconstruction algorithm to the defined tree structure built in the wavelet domain. Compared with the state-of-the-art error concealment algorithms, experimental results show that the proposed method has better reconstruction performance in terms of objective and subjective evaluations over a range of packet loss rates, and ensure that a high-quality image can be recovered for the high packet loss scenarios.

Sparse representation is a common approach for reducing the spatial redundancy by modelling an image as a linear combination of few atoms taken from an analytic or trained dictionary. This paper introduces a new image codec based on adaptive sparse representations wherein the visual salient information is considered into the rate allocation process. Firstly, the regions of the image that are more conspicuous to the human visual system are extracted using a classical graph-based method. Further, block-based sparse representation over a trained dictionary coupled with an adaptive sparse representation is proposed, such that the adaptivity is achieved by appropriately assigning more atoms of the dictionary to the blocks belonging to the salient regions. Experimental results show that the proposed method outperforms the existing image coding standards, such as JPEG and JPEG2000, which use an analytic dictionary, as well as the state-of-the-art codecs based on trained dictionaries.

This paper introduces a novel image error concealment technique, wherein the correlation among the correctly received pixels is implicitly exploited to recover the missing areas. This correlation is modeled as sparse representations of the correctly received surrounding areas of a lost regions, hence as a linear combination of very few atoms chosen from an over-complete dictionary. Under mild conditions, the sparse representation coefficients of a given zone, including both known and unknown pixels, can be correctly recovered from the sparse representation coefficients of its neighboring area. This linear process coupled with a simple smoothing process introduces a high quality concealed image. Compared with the state-of-the-art error concealment algorithms, experimental results show that the proposed method has better reconstruction performance in terms of objective and subjective evaluations.

With the growing number of cores on chips, conventional electrical interconnects reach scalability limits, leading the way for alternatives like Radio Frequency (RF), optical and 3D. Due to the variability of applications, communication needs change over time and across regions of the chip. To address these issues, a dynamically reconfigurable Network on Chip (NoC) is proposed. It uses RF and Orthogonal Frequency Division Multiple Access (OFDMA) to create communication channels whose allocation allows dynamic reconfiguration. We describe the NoC architecture and the distributed mechanism of dynamic allocation. We study the feasibility of the NoC based on state of the art components and analyze its performances. Static analysis shows that, for point to point communications, its latency is comparable with a 256-node electrical mesh and becomes lower for wider networks. A major feature of this architecture is its broadcast capacity. The RF~NoC becomes faster with 32 nodes, achieving a x3 speedup with 1024. Under realistic traffic models, its dynamic reconfigurability provides up to x6 lower latency while ensuring fairness.

Neurobiological systems have often been a source of inspiration for computational science and engineering, but in the past their impact has also been limited by the understanding of biological models. Today, new technologies lead to an equilibrium situation where powerful and complex computers bring new biological knowledge of the brain behavior. At this point, we possess sufficient understanding to both imagine new brain-inspired computing paradigms and to sustain a classical paradigm which reaches its end programming and intellectual limitations. In this context, we propose to reconsider the computation problem first in the specific domain of mobile robotics. Our main proposal consists in considering computation as part of a global adaptive system, composed of sensors, actuators, a source of energy and a controlling unit. During the adaptation process, the proposed brain-inspired computing structure does not only execute the tasks of the application but also reacts to the external stimulation and acts on the emergent behavior of the system. This approach is inspired by cortical plasticity in mammalian brains and suggests developing the computation architecture along the system's experience. This article proposes modeling this plasticity as a problem of estimating a probability density function. This function would correspond to the nature and the richness of the environment perceived through multiple modalities. We define and develop a novel neural model solving the problem in a distributed and sparse manner. And we integrate this neural map into a bio-inspired hardware substrate that brings the plasticity property into parallel many-core architectures. The approach is then called Hardware Plasticity. The results show that the self-organization properties of our model solve the problem of multimodal sensory data clusterization. The properties of the proposed model allow envisaging the deployment of this adaptation layer into hardware architectures embedded into the robot's body in order to build intelligent controllers.

The embedded image processing systems like smartphones and digital cameras have tight limits on storage, computation power, network connectivity, and battery usage. These limitations make it important to ensure efficient image coding. In the article, we present a novel heap-based priority queue structure employed by an Adaptive Scanning of Wavelet Data scheme (ASWD) targeting an embedded platform. ASWD is a context modeling block implemented via priority queues in a wavelet-based image coder to reorganize the wavelet coefficients into locally stationary sequences. The architecture we propose exploits efficient use of FPGA’s on-chip dual-port memories in an adaptive manner. Innovations of index-aware system linked to each element in the queue makes the location of queue element traceable in the heap as per the requirements of the ASWD algorithm. Moreover, use of 4-port memories along with intelligent data concatenation of queue elements yielded in a cost effective enhanced memory access. The memory ports are adaptively assigned to different units during different processing phases in a manner to optimally take advantage of memory access required by that phase. The architectural innovations can also be exploited in other applications that require efficient hardware implementations of generic priority queue or classical sorting applications which sort into the index. We designed and validated the hardware on an Altera’s Stratix IV FPGA as an IP accelerator in a Nios II processor based System on Chip. We show that our architecture at 150MHz can provide 45X speedup compared to an embedded ARM Cortex-A9 processor at 666MHz targeting the throughput of 10MB/s.

This paper introduces flexible radio techniques inside integrated circuits in order to tackle the interconnect issue for many-core chips. We propose to take benefits from OFDMA for a RF-interconnect associated to a carrier allocation policy and adaptive modulation. A 20 GHz bandwidth is shared between 32 tile sets made of 32 tiles of 4 cores each, for a 4096 cores chip. We adopt a cognitive radio approach in order to dynamically share 1024 carriers, which avoids inter-cluster communication contention and decreases latency compared to conventional static approaches.

We present a hardware architecture of a heapsort algorithm, the sorting is employed in a subband coding block of a wavelet-based image coder termed Öktem image coder [1]. Although this coder provides good image quality, the sorting is time consuming, and is application specific, as the sorting is repetitively used for different volume of data in the subband coding, thus a simple hardware implementation with fixed sorting capacity will be difficult to scale during runtime. To tackle this problem, the time/power efficiency and the sorting size flexibility have to be taken in to account. We proposed an improved FPGA heapsort architecture based on Zabołotny's work [2] as an IP accelerator of the image coder. We present a configurable architecture by using adaptive layer enable elements so the sorting capacity could be adjusted during runtime to efficiently sort different amount of data. With the adaptive memory shutdown, our improved architecture provides up to 20.9% power reduction on the memories compared to the baseline implementation. Moreover, our architecture provides 13x speedup compared to ARM CortexA9.

This abstract describes MPI-HCL, a hardware middleware for VHDL designers of scalable Multi processing reconfigurable System on Chip (MP-RSoC) using distributed memory paradigm. Some tools, like Planahead or Goahead [1] help to integrate dynamic reconfiguration inside MP-RSoC design but the conception phase of the MP-RSoC is separated from the deployment phase and it is very painful for designers to handle hundreds of hardware tasks and interconnect them properly. Some engineers and researchers proposed hardware operating systems denoted as HW OS [2] to manage deployment of hardware tasks in reconfigurable FPGA but HW-OS requirements adds constraints on hardware tasks. With our platform, heterogenous hardware tasks are modeled as MPI processes, and are compliant with MPI2 standard described in [3]. MPI relies on message passing to exchange data between hardware tasks and this paradigm is known to scale well in multiprocessing architecture.

This paper presents an adaptive heap sort architecture for an image coding implementation on FPGA, which specifically addresses the issue of sorting different amount of data located in each subband during the coding. The proposed sorting architecture is easily scalable. Performance of the sorter only depends on the amount of data sorted. The efficient usage of dual port memories yields high throughput up to 50 Msamples/s and their adaptive trigger/shutdown provide the average dynamic power reduction up to 20.9%. We designed this architecture and incorporated it in our Adaptive Scanning of Wavelet Data (ASWD) module which reorganizes the wavelet coefficients into locally stationary sequences for a wavelet-based image encoder. We validated the hardware on an Altera's Stratix IV FPGA as an IP accelerator in a Nios II processor based System on Chip. The architectural innovations can also be exploited in other applications that require high throughput and scalable sorting. Our experiments show that compared to an embedded ARM CortexA9 processor running at 666 MHz, our architecture at 100 MHz can provide around 13X speedup while consuming 242 mW average core dynamic power.

A paradigm shift is apparent in Chip Multiprocessor (CMP) design, as the new performance bottleneck is becoming communication rather than computation. It is widely provisioned that number of cores on a single chip will reach thousands in a decade. Thus, new high rate interconnects such as optical or RF have been proposed by various researchers. However, these interconnect structures fail to provide essential requirements of heterogeneous on-chip traffic; bandwidth reconfigurability and broadcast support with a low complex design. In this paper we investigate the feasibility of a new Orthogonal Frequency Division Multiple Access (OFDMA) RF interconnect for the first time to the best of our knowledge. In addition we provide a novel dynamic bandwidth arbitration and modulation order selection policy, that is designed regarding the bimodal on-chip packets. The proposed approach decreases the average latency up to 3.5 times compared to conventional static approach.

Wireless capsule endoscopy (WCE) is commonly used for noninvasive gastrointestinal tract evaluation, including the identification of polyps. In this paper, a new multimodal embeddable method for polyp detection and classification in wireless capsule endoscopic images was developed and tested. The multimodal wireless capsule used both 2D and 3D data to identify possible polyps and to deliver cancerous information of the polyps based on 3D geometric features. Possible polyps within the image (2D) were extracted using simple geometric shape features and, in a second step, the candidate regions of interest (ROI) were evaluated with a boosting-based method using textural features. Once the 2D identification of polyps has been performed, the two-class (“malignant” or “begnin”) classification of the polyps is achieved using the 3D parameters computed from the preselected ROI using an active stereo vision system. At this stage, a Support Vector Machine (SVM) classifier is used to proceed to the final classification and to make possible a pre diagnosis. The new proposed multimodal approach based on 2D-3D feature extraction improves WCE capabilities to identify and classify polyps: The boosting-based polyp classification demonstrated a sensitivity of 91%, a specificity of 95% and a false detection rate of 4.8% on a database composed of 300 hundred positive examples and 1200 negative ones; Considering the 3D performance, a large scale demonstrator was evaluated and tested to perform in vitro experiments on an ad hoc polyp database. The performance of the 3D approach achieved a correct classification rate (malignant or benin) of approximately 95%.

In this paper we present a novel high performance, low resource utilization and power efficient hardware architecture of an entropy coding scheme. The proposed architecture implements the Hierarchical Enumerative Coding algorithm (HENUC) on an embedded soft-processor based System-on-Chip, in which HENUC is an integral part of a wavelet based encoder oriented for locally stationary image source. Though HENUC has been implemented on an embedded DSP architecture before, the throughput was low. This paper proposes an optimized parallel architecture for HENUC, which is validated on a Terasic DE4-230 board containing Altera Stratix IV FPGA. Our implementation at 100MHz provides 5.7x speedup over Intel Xeon 8-core CPU and 12.3x speedup over TI DSP for 512 × 512 image while consuming less than 500 mw FPGA core power.

A comparative study of Hierarchical Enumerative Coding (HENUC) for FPGA and DSP implementation is presented. HENUC is a lossless fixed-point entropy coding algorithm employed by a wavelet-based image encoder, which provides good compression performance for the locally stationary image data. It has been implemented in our previous work on an Altera's 40nm Stratix IV EP4SGX230 FPGA as a hardware IP accelerator in a Nios II based system. In this paper, we implemented it on a Texas Instruments's (TI) 40nm Integra C6A816x/AM389x DSP. We present experimental results regarding the execution time, resource utilization and core power consumption of the two implementations and we evaluate their throughput and power efficiency. Our results show that a highly parallelized FPGA implementation at 100MHz is 12.3x faster than a highly tuned DSP implementation running at 1.5 GHz and consumes 2.4x less power, they also show that the proposed algorithm is more suitable for hardware implementation.

High-energy physics experiments require high-speed triggering systems capable of performing complex pattern recognition at rates of Megahertz to Gigahertz. Neural networks implemented in hardware have been the solution of choice for certain experiments. The neural triggering problem is presented here via a detailed look at the H1 level 2 trigger at the HERA accelerator, Hamburg, Germany, followed by a section on the importance of hardware preprocessing for such systems, and finally some new architectural ideas for using field programmable gate arrays in very high-speed neural-network triggers at upcoming experiments.