Large Language Models (LLMs) have become foundational tools in natural language processing, achieving state-of-the-art performance across a variety of tasks. However, their immense size and computational requirements make them impractical for deployment in resource-constrained environments, such as edge devices and embedded systems. In this work, we introduce Magnitude and Gradient-Informed Pruning (MaGrIP) , a novel framework for task-agnostic pruning and compression of LLMs. MaGrIP employs a dual-threshold strategy combining magnitude- and gradient-based saliency measures to efficiently prune redundant neurons while retaining task performance. Our results demonstrate the effectiveness of MaGrIP in compressing state-of-the-art models. The compression reduced the total computational complexity of the FFN layers from \(\mathcal {O}(d \cdot h)\) to \(\mathcal {O}((d - q) \cdot h)\) . In terms of model size, our pruning approach significantly reduces both model parameters and storage requirements while maintaining competitive perplexity scores evaluated on WikiText-2. For the Gemma 7B model, our method reduces the total size from 28 GB to 5 GB, while for Gemma 2B, MaGrIP achieves a size reduction from 8 GB to 1.5 GB. MaGrIP furthermore exhibits robust performance across multiple benchmarks, such as BOOLQ, ARC-E, and CSQA. Specifically, the pruned Gemma 7B model at 50% pruning achieved 59.26% accuracy on ARC-E compared to 81.06% for the baseline, and 64.74% accuracy on BoolQ compared to 59.98% for the baseline. Similarly, the pruned Llama 3 8B at 50% pruning achieved 46.76% accuracy on ARC-E compared to 77.57% for the baseline, reflecting the tradeoff between compression and accuracy. LLMs compressed using MaGrIP, when deployed on the Nvidia Jetson Orin Nano, achieved a 2.16× improvement in throughput and a 2.3× improvement in performance compared to baseline LLMs.

Cutting-edge Large Language Models (LLMs) play a crucial role in improving autonomous navigation by offering efficient solutions. While LLMs require powerful computers to operate, security concerns and maintaining a stable connection with the cloud can be challenging due to various factors. To address this issue, we propose a metareasoning approach for edge-cloud collaborative LLM planning which leads to an efficient autonomous navigation. The proposed approach allows the system to seamlessly switch between cloud and edge devices to fulfill the mission even in the event of a lost connection or entering a GPS-denied environment. Moreover, we deploy state-of-the-art LLM models on resource-constrained systems like the NVIDIA Jetson Orin Nano 8GB, integrated with ROSMASTER X3. These LLMs have demonstrated exceptional utility in dynamic planning for multi-room or maze environments. A comprehensive LLM profiling of TinyLLM models was performed for five different LLMs. The LLM profiling result shows that while certain models with smaller sizes and lower power consumption were available, their accuracy was insufficient for our application requirements. As a result, LLaMa2-7B is considered the edge LLM model due to its optimal balance of performance and accuracy. The experimental results show that under weak signal conditions (<-50 dB), the metareasoning approach improves energy consumption by up to 4x while the cloud-based implementation exceeds the energy consumption of the onboard LLM implementation. Moreover, with delays of 10-20 seconds, cloud implementation becomes impractical for real-time applications in weak signal environments. This underscores the need for metareasoning, which optimizes energy consumption and response time, providing a balanced solution by adapting to signal strength. A real-world implementation of the proposed approach on ROSMASTER X3 with NVIDIA Jetson Orin Nano board can be found in this video which shows that the mission was completed despite losing the connection with cloud-based LLM.

Vision Transformers (ViTs) have demonstrated strong capabilities in interpreting complex medical imaging data. However, their significant computational and memory demands pose challenges for deployment in real-time, resource-constrained mobile and wearable devices used in clinical environments. We introduce, BitMedVit, a new class of Edge ViTs serving as medical AI assistants that perform structured analysis of medical images directly on the edge. BitMedVit utilizes ternary-quantized linear layers tailored for medical imaging and combines a training procedure with multi-query attention, preserving stability under ternary weights with low-precision activations. Furthermore, BitMedVit employs task-aware distillation from a high-capacity teacher to recover accuracy lost due to extreme quantization. Lastly, we also present a pipeline that maps the ternarized ViTs to a custom CUDA kernel for efficient memory bandwidth utilization and latency reduction on the Jetson Orin Nano. Finally, BitMedVit achieves 86% diagnostic accuracy (89% SOTA) on MedMNIST across 12 datasets, while reducing model size by 43×, memory traffic by 39×, and enabling 16.8 ms inference at an energy efficiency up to 41× that of SOTA models at 183.62 GOPs/J on the Orin Nano. Our results demonstrate a practical and scientifically grounded route for extreme-precision medical imaging ViTs deployable on the edge, narrowing the gap between algorithmic advances and deployable clinical tools.

The safety, energy efficiency, and small size of smart drones have led to the broad use of autonomous Unmanned Aerial Vehicles (UAVs) across various applications, creating opportunities for human-machine collaboration. Machine Learning (ML) algorithms like Neural Networks (NNs) offer promising solutions for vision-based navigation and autonomous systems. However, these algorithms are computationally intensive, making it challenging to deploy them on robots expanded by resource-constrained edge devices with limited computational power and low energy consumption requirements. In this paper, we propose an Energy-Efficient Framework for Video Streaming and Augmented Reality called E2AR to enable ML-based multi-edge device video streaming to a AR device while applying augmented reality to enhance human-machine teaming. For this aim, a YOLO is deployed on the edge device for energy-efficient computation and higher performance. Moreover, video streaming to HoloLens is optimized to improve communication latency and power consumption. To evaluate the proposed method, we implemented it on edge devices such as Crazyflie drone while streaming video to the HoloLens. Crazyflie drones with LiDAR sensors and the GAP8 processor has consisted of an octa-core RISC-V. We measured the power consumption, latency, and core usage of the GAP8 processor while implementing the proposed approach. View a video demonstration of the E2AR concept at: Video.

The increasing demand for efficient deep learning model deployment on Tiny Machine Learning (tinyML) and Edge platforms necessitates the development of methods that enable automated and effective network pruning, tailored to tinyML hardware constraints. In this paper, we present a novel differentiable pruning method that accepts total available memory on a tinyML hardware and employs saliency based measurements to identify and prune less significant connections within a deep neural network (DNN). Our approach integrates network compression within the training process, adapting resource utilization to the specific constraints of FPGAs, particularly focusing on on-chip memory. By leveraging a custom tinyML accelerator, we enable an efficient hardware-software co-design. Our framework further quantizes the model to int-8 to optimize the balance between model size and accuracy, crucial for tinyML applications. The efficacy of our approach is examined for the compression of LeNet and VGG16 DNNs. When compared to similar state of the art pruning techniques, our approach for no drop in accuracy further compresses LeNet by 1.15 ×. In the case of VGG16, compared to the baseline implementation, for a 4% drop in accuracy we compress the model up to 55 ×. A comparative analysis of our FPGA hardware accelerator against leading image classification accelerators emphasizes the merits of our approach with a marked improvement in throughput by 1.46 × for LeNet and energy efficiency by 1.7 × for VGG16.

Fine-tuning deep neural networks is pivotal for creating inference modules that can be suitably imported to edge or field-programmable gate array (FPGA) platforms. Traditionally, exploration of different parameters throughout the layers of deep neural networks has been done using grid search and other brute force techniques. Although these methods lead to the optimal choice of network parameters, the search process can be very time consuming and may not consider deployment constraints across different target platforms. This work addresses this problem by proposing Reg-Tune, a regression-based profiling approach to quickly determine the trend of different metrics in relation to hardware deployment of neural networks on tinyML platforms like FPGAs and edge devices. We start by training a handful of configurations belonging to different combinations of \(\mathcal {NN}\scriptstyle \langle q (quantization),\,s (scaling)\rangle \displaystyle\) or \(\mathcal {NN}\scriptstyle \langle r (resolution),\,s\rangle \displaystyle\) workloads to generate the accuracy values respectively for their corresponding application. Next, we deploy these configurations on the target device to generate energy/latency values. According to our hypothesis, the most energy-efficient configuration suitable for deployment on the target device is a function of the variables q , r , and s . Finally, these trained and deployed configurations and their related results are used as data points for polynomial regression with the variables q , r , and s to realize the trend for accuracy/energy/latency on the target device. Our setup allows us to choose the near-optimal energy-consuming or latency-driven configuration for the desired accuracy from the contour profiles of energy/latency across different tinyML device platforms. To this extent, we demonstrate the profiling process for three different case studies and across two platforms for energy and latency fine-tuning. Our approach results in at least 5.7 \(\times\) better energy efficiency when compared to recent implementations for human activity recognition on FPGA and 74.6% reduction in latency for semantic segmentation of aerial imagery on edge devices compared to baseline deployments.

With the evaluation of Artificial Intelligence (AI), there has been a resurgence of interest in how to use AI algorithms on low-power embedded systems to broaden potential use cases of the Internet of Things (IoT). To mimic multimodal human perception, multimodal deep neural networks (M-DNN) have recently become very popular with the classification task due to their impressive performance for computer vision and audio processing tasks. This article presents TinyM 2 Net-V2 —a compact low-power software hardware architecture for m ulti m odal deep neural networks for resource-constrained tiny devices. To compress the models to implement on tiny devices, cyclicly sparsification and hybrid quantization (4-bits weights and 8-bits activations) methods are used. Although model compression techniques are an active research area, we are the first to demonstrate their efficacy for multimodal deep neural networks, using cyclicly sparsification and hybrid quantization of weights/activations. TinyM 2 Net-V2 shows that even a tiny multimodal deep neural network model can improve the classification accuracy more than that of any unimodal counterparts. Parameterized M-DNN model architecture was designed to be evaluated in two different case-studies: vehicle detection from multimodal images and audios and COVID-19 detection from multimodal audio recordings. The most compressed TinyM 2 Net-V2 achieves 92.5% COVID-19 detection accuracy (6.8% improvement from the unimodal full precision model) and 90.6% vehicle classification accuracy (7.7% improvement from the unimodal full precision model). A parameterized and flexible FPGA hardware accelerator was designed as well for TinyM 2 Net-V2 models. To the best of our knowledge, this is the first work accelerating multimodal deep neural network models on low-power Artix-7 FPGA hardware. We achieved energy efficiency of 9.04 GOP/s/W and 15.38 GOP/s/W for case-study 1 and case-study 2, respectively, which is comparable to the state-of-the-art results. Finally, we compared our tiny FPGA hardware implementation results with off-the-shelf resource-constrained devices and showed our implementation is faster and consumed less power compared to the off-the-shelf resource-constrained devices.

Safety, low-cost, small size, and Artificial Intelli-gence (AI) capabilities of drones have led to the proliferation of autonomous tiny Unmanned Aerial Vehicles (UAVs) in many applications which are dangerous, unknown, or time-consuming for humans. Deep Neural Networks (DNNs) have enabled au-tonomous navigation while using captured data by drone sensors as input to the model. Due to the extreme complexity of DNNs, cloud-based approaches have been highly addressed in which a drone is connected to the cloud and sends the data to the cloud, and takes the result. On the other hand, emerging tiny machine learning models and edge computing brings significant improvement in energy efficiency and latency with respect to cloud-based approaches. However, there is a trade-off in these two implementations for model accuracy, latency, and energy efficiency. For instance, applying tiny machine learning models leads to lower latency but it sacrifices model accuracy in comparison to cloud-based computing. To address these challenges, we consider multiple models and introduce a new approach named MLAE2 which applies Metareasoning approach for Latency-Aware Energy-Efficient autonomous drones. Metareasoning mon-itors parameters such as latency and energy consumption for different algorithms and chooses the appropriate algorithm due to the environmental situation changes. To Evaluate our approach we extract the power consumption and latency for both cloud-based computing and edge computing while deploying multiple models on a tiny drone named Crazyflie. The experimental results show that MLAE2 successfully meets the latency constraint while maximizing model accuracy and improving energy efficiency.

Artificial Intelligence (AI) and Deep Neural Networks (DNNs) have attracted attention as a solution within autonomous systems fields as they enable applications such as visual perception and navigation. Although cloud-based approaches have already been highly addressed, there is a growing interest in using both AI and DNNs on the edge as this allows for lower latency and avoids the potential security concerns of transmitting data to a remote server. However, deploying DNNs on edge devices is challenging due to the limited computational power available, as well as energy efficiency being of the utmost importance. In this work, we introduce an approach named E2EdgeAI for Energy-Efficient Edge computing that takes advantage of AI for autonomous tiny drones. This approach optimizes the energy efficiency of DNNs by considering the effects of memory access and core utilization on the energy consumption of tiny UAVs. To perform the experiment, we used a tiny drone named Crazyflie with the AI -deck expansion, which includes an octa-core RISC-V processor. The experimental results show the proposed approach reduces the model size by up to 14.4x, improves energy per inference by 78%, and increases energy efficiency by 5.6x. A recorded video for the proposed approach can be found here: Video.

Recently, Reinforcement Learning (RL) has shown great performance in solving sequential decision-making and control in dynamic environment problems. Despite its achievements, deploying Deep Neural Network (DNN)-based RL is expensive in terms of time and power due to the large number of episodes required to train agents with high dimensional image representations. Additionally, at the interference the large energy footprint of deep neural networks can be a major drawback. Embedded edge devices as the main platform for deploying RL applications are intrinsically resource-constrained and deploying deep neural network-based RL on them is a challenging task. As a result, reducing the number of actions taken by the RL agent to learn desired policy, along with the energy-efficient deployment of RL, is crucial. In this article, we propose Energy Efficient Hierarchical Reinforcement Learning (E2HRL), which is a scalable hardware architecture for RL applications. E2HRL utilizes a cross-layer design methodology for achieving better energy efficiency, smaller model size, higher accuracy, and system integration at the software and hardware layers. Our proposed model for RL agent is designed based on the learning hierarchical policies, which makes the network architecture more efficient for implementation on mobile devices. We evaluated our model in three different RL environments with different level of complexity. Simulation results with our analysis illustrate that hierarchical policy learning with several levels of control improves RL agents training efficiency and the agent learns the desired policy faster compared to a non-hierarchical model. This improvement is specifically more observable as the environment or the task becomes more complex with multiple objective subgoals. We tested our model with different hyperparameters to achieve the maximum reward by the RL agent while minimizing the model size, parameters, and required number of operations. E2HRL model enables efficient deployment of RL agent on resource-constraint-embedded devices with the proposed custom hardware architecture that is scalable and fully parameterized with respect to the number of input channels, filter size, and depth. The number of processing engines (PE) in the proposed hardware can vary between 1 to 8, which provides the flexibility of tradeoff of different factors such as latency, throughput, power, and energy efficiency. By performing a systematic hardware parameter analysis and design space exploration, we implemented the most energy-efficient hardware architectures of E2HRL on Xilinx Artix-7 FPGA and NVIDIA Jetson TX2. Comparing the implementation results shows Jetson TX2 boards achieve 0.1 ∼ 1.3 GOP/S/W energy efficiency while Artix-7 FPGA achieves 1.1 ∼ 11.4 GOP/S/W, which denotes 8.8× ∼ 11× better energy efficiency of E2HRL when model is implemented on FPGA. Additionally, compared to similar works our design shows better performance and energy efficiency.

This article presents a low-power, programmable, domain-specific manycore accelerator, Binarized neural Network Manycore Accelerator (BiNMAC), which adopts and efficiently executes binary precision weight/activation neural network models. Such networks have compact models in which weights are constrained to only 1 bit and can be packed several in one memory entry that minimizes memory footprint to its finest. Packing weights also facilitates executing single instruction, multiple data with simple circuitry that allows maximizing performance and efficiency. The proposed BiNMAC has light-weight cores that support domain-specific instructions, and a router-based memory access architecture that helps with efficient implementation of layers in binary precision weight/activation neural networks of proper size. With only 3.73% and 1.98% area and average power overhead, respectively, novel instructions such as Combined Population-Count-XNOR , Patch-Select , and Bit-based Accumulation are added to the instruction set architecture of the BiNMAC, each of which replaces execution cycles of frequently used functions with 1 clock cycle that otherwise would have taken 54, 4, and 3 clock cycles, respectively. Additionally, customized logic is added to every core to transpose 16×16-bit blocks of memory on a bit-level basis, that expedites reshaping intermediate data to be well-aligned for bitwise operations. A 64-cluster architecture of the BiNMAC is fully placed and routed in 65-nm TSMC CMOS technology, where a single cluster occupies an area of 0.53 mm 2 with an average power of 232 mW at 1-GHz clock frequency and 1.1 V. The 64-cluster architecture takes 36.5 mm 2 area and, if fully exploited, consumes a total power of 16.4 W and can perform 1,360 Giga Operations Per Second (GOPS) while providing full programmability. To demonstrate its scalability, four binarized case studies including ResNet-20 and LeNet-5 for high-performance image classification, as well as a ConvNet and a multilayer perceptron for low-power physiological applications were implemented on BiNMAC. The implementation results indicate that the population-count instruction alone can expedite the performance by approximately 5×. When other new instructions are added to a RISC machine with existing population-count instruction, the performance is increased by 58% on average. To compare the performance of the BiNMAC with other commercial-off-the-shelf platforms, the case studies with their double-precision floating-point models are also implemented on the NVIDIA Jetson TX2 SoC (CPU+GPU). The results indicate that, within a margin of ∼2.1%--9.5% accuracy loss, BiNMAC on average outperforms the TX2 GPU by approximately 1.9× (or 7.5× with fabrication technology scaled) in energy consumption for image classification applications. On low power settings and within a margin of ∼3.7%--5.5% accuracy loss compared to ARM Cortex-A57 CPU implementation, BiNMAC is roughly ∼9.7×--17.2× (or 38.8×--68.8× with fabrication technology scaled) more energy efficient for physiological applications while meeting the application deadline.

This article presents an energy-efficient and flexible multichannel Electroencephalogram (EEG) artifact identification network and its hardware using depthwise and separable convolutional neural networks. EEG signals are recordings of the brain activities. EEG recordings that are not originated from cerebral activities are termed artifacts . Our proposed model does not need expert knowledge for feature extraction or pre-processing of EEG data and has a very efficient architecture implementable on mobile devices. The proposed network can be reconfigured for any number of EEG channel and artifact classes. Experiments were done with the proposed model with the goal of maximizing the identification accuracy while minimizing the weight parameters and required number of operations. Our proposed network achieves 93.14% classification accuracy using an EEG dataset collected by 64-channel BioSemi ActiveTwo headsets, averaged across 17 patients and 10 artifact classes. Our hardware architecture is fully parameterized with number of input channels, filters, depth, and data bit-width. The number of processing engines (PE) in the proposed hardware can vary between 1 to 16, providing different latency, throughput, power, and energy efficiency measurements. We implement our custom hardware architecture on Xilinx FPGA (Artix-7), which on average consumes 1.4 to 4.7 mJ dynamic energy with different PE configurations. Energy consumption is further reduced by 16.7× implementing on application-specified integrated circuit at the post layout level in 65-nm CMOS technology. Our FPGA implementation is 1.7 × to 5.15 × higher in energy efficiency than some previous works. Moreover, our Application-Specified Integrated Circuit implementation is also 8.47 × to 25.79 × higher in energy efficiency compared to previous works. We also demonstrated that the proposed network is reconfigurable to detect artifacts from another EEG dataset collected in our lab by a 14-channel Emotiv EPOC+ headset and achieved 93.5% accuracy for eye blink artifact detection.

In deep convolutional neural networks (DCNNs), model size and computation complexity are two important factors governing throughput and energy efficiency when deployed to hardware for inference. Recent works on compact DCNNs as well as pruning methods are effective, yet with drawbacks. For instance, more than half the size of all MobileNet models lies in their last two layers, mainly because compact separable convolution (CONV) layers are not applicable to their last fully connected (FC) layers. Also, in pruning methods, the compression is gained at the expense of irregularity in the DCNN architecture, which necessitates additional indexing memory to address nonzero weights, thereby increasing memory footprint, decompression delays, and energy consumption. In this article, we propose cyclic sparsely connected (CSC) architectures, with memory/computation complexity of$\mathcal {O}(N\log {}N)$, where$N$is the number of nodes/channels given a DCNN layer that, contrary to compact depthwise separable layers, can be used as an overlay for both FC and CONV layers of$\mathcal {O}(N^{2})$. Also, contrary to pruning methods, CSC architectures are structurally sparse and require no indexing due to their cyclic nature. We show that both standard convolution and depthwise convolution layers are special cases of the CSC layers, whose mathematical function, along with FC layers, can be unified into one single formulation and whose hardware implementation can be carried out under one arithmetic logic component. We examine the efficacy of the CSC architectures for compression of LeNet, AlexNet, and MobileNet DCNNs with precision ranging from 2 to 32 bits. More specifically, we surge upon the compact 8-bit quantized 0.5 MobileNet V1 and show that by compressing its last two layers with CSC architectures, the model is compressed by$\sim 1.5\times $with a size of only 873 kB and little accuracy loss. Finally, we design a configurable hardware that implements all types of DCNN layers including FC, CONV, depthwise, CSC-FC, and CSC-CONV indistinguishably within a unified pipeline. We implement the hardware on a tiny Xilinx field-programmable gate array (FPGA) for total on-chip processing of the compressed MobileNet that, compared to the related work, has the highest Inference/J while utilizing the smallest FPGA.

Cache hierarchy was designed to allow CPU cores to process instructions faster by bridging the significant latency gap between the main memory and processor. In addition, various cache replacement algorithms are proposed to predict future data and instructions to boost the performance of the computer systems. However, recently proposed cache-based Side-Channel Attacks (SCAs) have shown to effectively exploiting such a hierarchical cache design. The cache-based SCAs are exploiting the hardware vulnerabilities to steal secret information from users by observing cache access patterns of cryptographic applications and thus are emerging as a serious threat to the security of the computer systems. Prior works on mitigating the cache-based SCAs have mainly focused on cache partitioning techniques and/or randomization of mapping between main memory. However, such solutions though effective, require modification in the processor hardware which increases the complexity of architecture design and are not applicable to current as well as legacy architectures. In response, this paper proposes a lightweight system and architecture level randomization technique to effectively mitigate the impact of side-channel attacks on last-level caches with no hardware redesign overhead for current as well as legacy architectures. To this aim, by carefully adapting the processor frequency and prefetchers operation and adding proper level of noise to the attackers' cache observations we attempt to protect the critical information from being leaked. The experimental results indicate that the concurrent randomization of frequency and prefetchers can significantly prevent cache-based side-channel attacks with no need for a new cache design. In addition, the proposed randomization and adaptation methodology outperforms the stat-of-the-art solutions in terms of the performance and execution time by reducing the performance overhead from 32.66% to nearly 20%.

Hardware-Assisted Malware Detection (HMD) techniques deploy Machine Learning (ML) classifiers to detect patterns of malicious applications based on microarchitectural features captured by modern microprocessors' Hardware Performance Counters (HPCs). Existing HMD methods have limited their analysis on detecting malicious applications that are spawned as a separate thread during application execution, hence detecting embedded malware patterns at run-time still remains an important challenge. Embedded malware refers to harmful stealthy cyber attacks in which the malicious code is hidden within benign applications and remains undetected by traditional malware detection approaches. In HMD methods, when the HPC data is directly fed into a machine learning classifier, embedding malicious code inside the benign applications leads to contamination of HPC information, as the collected HPC features combine benign and malware microarchitectural events together. To address this challenge, in this paper we propose StealthMiner, a specialized time series machine learning approach to accurately detect embedded malware at run-time using branch instructions feature, the most prominent microarchitectural feature. The results indicate that StealthMiner can detect embedded malware at run-time with 94% detection performance on average with only one HPC feature, outperforming the detection performance of state-of-the-art HMD methods by 42%.

Reinforcement learning (RL) has shown great performance in solving sequential decision-making problems. While a lot of works have done on processing state information such as images, there has been some effort towards integrating natural language instructions into RL. In this paper, we propose an energy-efficient architecture which is designed to receive both images and text inputs as a step towards designing RL agents that can understand human language and act in real-world environments. Different configurations are proposed to illustrate the trade off between the number of parameters and the model accuracy, and a custom low power hardware is designed and implemented on FPGA based on the best configuration. The hardware designed to be configurable with different parameters such as number of processing elements, so that it can easily balance power and performance. The high throughput configuration achieves 217 frames per second throughput with 1.2 mJ energy consumption per classification on Xilinx Artix-7 FPGA, while the low power configuration consumes less than 139 mW for 30 frames per second classification. Compared to the similar works using FPGA for hardware implementation, our design is more energy efficient and need less energy for generating each output.

Microarchitectural Side-Channel Attacks (SCAs) have posed serious threats to the security of modern computing systems. Such attacks exploit side-channel vulnerabilities stemming from fundamental performance-enhancing components such as cache memories. The existing works on detection of SCAs based on low-level microarchitectural features have considered collecting both victim and attack applications' hardware events that are captured from processors' hardware performance counter (HPC) registers. However, in such techniques the attack HPCs data can be easily manipulated and/or corrupted resulting in misleading the SCAs detection mechanism. In addition, the prior studies have explored the suitability of a limited number of Machine Learning (ML) algorithms in detecting microarchitectural SCAs. In response, in this paper, we conduct a comprehensive evaluation of various machine learning-based countermeasures for real-time side-channel attack detection based on low-level microarchitectural features. For this purpose, the victim applications' behavior is collected using the HPC features and analyzed under no attack and attack conditions to avoid potential manipulation of attackers' HPCs. We further explore the HPCs monitoring overhead when microarchitectural features are sampled at different intervals to find out the appropriate sampling interval for SCAs detection. For the purpose of thorough analysis, various types of ML classifiers are implemented and precisely compared across different evaluation metrics including detection accuracy, F-measure, robustness (Area Under the ROC Curve), and computational latency to identify the most efficient ML classifiers for real-time microarchitectural SCAs detection

The increasing heterogeneity in the applications to be processed ceased ASICs to exist as the most efficient processing platform. Hybrid processing platforms such as CPU+FPGA are emerging as powerful processing platforms to support an efficient processing for a diverse range of applications. Hardware/Software co-design enabled designers to take advantage of these new hybrid platforms such as Zynq. However, dividing an application into two parts that one part runs on CPU and the other part is converted to a hardware accelerator implemented on FPGA, is making the platform selection difficult for the developers as there is a significant variation in the application's performance achieved on different platforms. Developers are required to fully implement the design on each platform to have an estimation of the performance. This process is tedious when the number of available platforms is large. To address such challenge, in this work we propose XPPE, a neural network based cross-platform performance estimation. XPPE utilizes the resource utilization of an application on a specific FPGA to estimate the performance on other FPGAs. The proposed estimation is performed for a wide range of applications and evaluated against a vast set of platforms. Moreover, XPPE enables developers to explore the design space without requiring to fully implement and map the application. Our evaluation results show that the correlation between the estimated speed up using XPPE and actual speedup of applications on a Hybrid platform over an ARM processor is more than 0.98.

In deep neural networks (DNNs), model size is an important factor affecting performance, energy efficiency and scalability. Recent works on weight pruning have shown significant reduction in model size at the expense of irregularity in the DNN architecture, which necessitates additional indexing memory to address non-zero weights, thereby increasing chip size, energy consumption and delays. In this paper, we propose cyclic sparsely connected (CSC) layers, with a memory/computation complexity of O(NlogN), that can be used as an overlay for fully connected (FC) layers whose number of parameters, O(N2), can dominate the parameters of the entire DNN model. The CSC layers are composed of a few sequential layers, referred to as support layers, which result in full connectivity between the Inputs and Outputs of each CSC layer. We introduce an algorithm to train models with FC layers replaced with CSC layers in a bottom-up approach by incrementally increasing the CSC layers characteristics such as connectivity and number of synapses, to achieve the desired accuracy given a compression rate. One advantage of the CSC layers is that there will be no requirement for indexing the non-zero weights. Our experimental results using AlexNet on ImageNet and LeNet300100 on MNIST indicate that by substituting FC layers with CSC layers, we can achieve 10× to 46× compression within a margin of 2% accuracy loss, which is comparable to non-structural pruning methods. A scalable parallel hardware architecture to implement CSC layers, and an equivalent scalable parallel architecture to efficiently implement non-structurally pruned FC layers are designed and fully placed and routed on Artix-7 FPGA and ASIC 65nm CMOS technology for LeNet300100 model. The results indicate that the proposed CSC hardware outperforms the conventional non-structurally pruned architecture with an equal compression rate by ~2× in power, energy, area and resource utilization when running at the same frequency.

Hardware-assisted Malware Detection (HMD) has emerged as a promising solution to improve the security of computer systems using Hardware Performance Counters (HPCs) information collected at run-time. While several recent studies proposed machine learning-based solutions to identify malware using HPCs, they rely on a large number of microarchitectural events to achieve high accuracy and detection rate. More importantly, they have largely overlooked complexity-effective prediction of malware classes at run-time. As we show in this work, the detection performance of malware classifiers is highly dependent on the number of available HPCs and varies significantly across classes of malware. The limited number of available HPCs in modern microprocessors that can be simultaneously captured makes run-time malware detection with high detection performance using existing solutions a challenging problem, as they require multiple runs of applications to collect a sufficient number of microarchitectural events. In response, in this paper, we first identify the most important HPCs for HMD using an effective feature reduction method. We then develop a specialized two-stage run-time HMD referred as 2SMaRT. 2SMaRT first classifies applications using a multiclass classification technique into either benign or one of the malware classes (Virus, Rootkit, Backdoor, and Trojan). In the second stage, to have a high detection performance, 2SMaRT deploys a machine learning model that works best for each class of malware. To realize an effective run-time solution that relies on only available HPCs, 2SMaRT is further customized using an ensemble learning technique to boost the performance of general malware detectors. The experimental results show that 2SMaRT using ensemble technique with just 4HPCs outperforms state-of-the-art classifiers with 8HPCs by up to 31.25% in terms of detection performance, on average across different classes of malware.

In autonomous embedded systems, it is often vital to reduce the amount of actions taken in the real world and energy required to learn a policy. Training reinforcement learning agents from high dimensional image representations can be very expensive and time consuming. Autoencoders are deep neural network used to compress high dimensional data such as pixelated images into small latent representations. This compression model is vital to efficiently learn policies, especially when learning on embedded systems. We have implemented this model on the NVIDIA Jetson TX2 embedded GPU, and evaluated the power consumption, throughput, and energy consumption of the autoencoders for various CPU/GPU core combinations, frequencies, and model parameters. Additionally, we have shown the reconstructions generated by the autoencoder to analyze the quality of the generated compressed representation and also the performance of the reinforcement learning agent. Finally, we have presented an assessment of the viability of training these models on embedded systems and their usefulness in developing autonomous policies. Using autoencoders, we were able to achieve 4-5X improved performance compared to a baseline RL agent with a convolutional feature extractor, while using less than 2W of power.

Datacenters provide high performance and flexibility for users and cost efficiency for operators. Hyperscale datacenters are harnessing massively scalable computer resources for large-scale data analysis. However, cloud/datacenter infrastructure does not scale as fast as the input data volume and computational requirements of big data and analytics technologies. Thus, more applications need to share CPU at the node level that could have large impact on performance and operational cost. To address this challenge, in this paper we show that, concurrently fine-tune parameters at the application, microarchitecture, and system levels are creating opportunities to co-locate applications at the node level and improve energy-efficiency of the server while maintaining performance. Co-locating and self-tuning of unknown applications are challenging problems, especially when co-locating multiple big data applications concurrently with many tuning knobs, potentially requiring exhaustive brute-force search to find the right settings. This research challenge upsurges an imminent need to develop a technique that co-locates applications at a node level and predict the optimal system, architecture and application level configure parameters to achieve the maximum energy efficiency. It promotes the scale-down of computational nodes by presenting the Energy-Efficient Co-Locating and Self-Tuning (ECoST) technique for data intensive applications. ECoST proof of concept was successfully tested on MapReduce platform. ECoST can also be deployed on other data-intensive frameworks where there are several parameters for power and performance tuning optimizations. ECoST collects run-time hardware performance counter data and implements various machine learning models from as simple as a lookup table or decision tree based to as complex as neural network based to predict the energy-efficiency of co-located applications. Experimental data show energy efficiency is achieved within 4% of the upper bound results when co-locating multiple applications at a node level. ECoST is also scalable, being within 8% of upper bound on an 8-node server.

Deep neural networks have become the readiest answer to a range of application challenges including image recognition, stock analysis, natural language processing, and biomedical applications such as seizure detection. All while outperforming prior leading solutions that relied heavily on hand-engineered techniques. However, deployment of these neural networks often requires high-computational and memory-intensive solutions. These requirements make it challenging to deploy Deep Neural Networks (DNNs) in embedded, real-time low-power applications where classic architectures, GPUs and CPUs, still impose significant power burden. Systems-on-Chip (SoC) with Field-programmable Gate Arrays (FPGAs) can be used to improve performance and allow more fine-grain control of resources than CPUs or GPUs, but it is difficult to find the optimal balance between hardware and software to improve DNN efficiency. In the current research literature there have been few proposed solutions to address optimizing hardware and software deployments of DNNs in embedded low-power systems. To address the computation resource restriction and low-power needs for deploying these networks, we describe and implement a domain-specific metric model for optimizing task deployment on differing platforms, hardware and software. Next, we propose a DNN hardware accelerator called Scalable Low-power Accelerator for real-time deep neural Networks (SCALENet) that includes multithreaded software workers. Finally, we propose a heterogeneous aware scheduler that uses the DNN-specific metric models and the SCALENet accelerator to allocate a task to a resource based on solving a numerical cost for a series of domain objectives. To demonstrate the applicability of our contribution, we deploy nine modern deep network architectures, each containing a different number of parameters within the context of two different neural network applications: image processing and biomedical seizure detection. Utilizing the metric modeling techniques integrated into the heterogeneous aware scheduler and the SCALENet accelerator, we demonstrate the ability to meet computational requirements, adapt to multiple architectures, and lower power by providing an optimized task to resource allocation. Our heterogeneous aware scheduler improves power saving by decreasing power consumption by 10% of the total system power, does not affect the accuracy of the networks, and still meets the real-time deadlines. We demonstrate the ability to achieve parity with or exceed the energy efficiency of NVIDIA GPUs when evaluated against Jetson TK1 with embedded GPU SoC and with a 4× power savings in a power envelope of 2.0W. When compared to existing FPGA-based accelerators, SCALENet’s accelerator and heterogeneous aware scheduler achieves a 4× improvement in energy efficiency.

This paper presents a low power, domain-specific manycore accelerator referred to as "BiNMAC"- Binarized neural Network Manycore ACcelerator, which effectively maps and executes Binary Deep Neural Networks (BNNs). With only 2.40% and 1.88% area and power overhead, novel instructions such as Population-Count and Patch-Select are added to the ISA of the BiNMAC, each of which replaces frequently used functions that would have taken 52 and 4 clock cycles respectively with 1 clock cycle. A 64-cluster architecture of the BiNMAC is fully placed and routed in 65~nm TSMC CMOS technology, where a single cluster occupies an area of 0.53 mm^2 with a power of 223 mW at 1 GHz clock frequency. The 64-cluster architecture takes 36.5 mm^2 area and, if fully utilized, consumes a power of 16.4 W. We also propose a multilayer perceptron (MLP) neural network for multimodal time-series data classification. Binarized versions of the 3-layers MLP and ResNet-20 are implemented on BiNMAC. The implementation results show that BiNMAC consumes 0.02 mJ and 3.8 mJ energy which is 13 times and 30 times lower than the implementation of standard non-binarized MLP and ResNet-20 on an equivalent predecessor platform. To compare the performance of the BiNMAC with other off-the-shelf platforms, the two networks are also implemented on the NVIDIA Jetson TX2 SoC (CPU+GPU). BiNMAC achieves 22 times and 78 times higher throughput and 23 times and 41 times lower energy consumption compared to TX2 SoC for the binarized MLP and ResNet-20, respectively.

The paper presents "MC3A"- Markov Chain Monte Carlo Many Core Accelerator, a high-throughput, domain-specific, programmable manycore accelerator, which effectively generates samples from a provided target distribution. MCMC samplers are used in machine learning, image and signal processing applications that are computationally intensive. In such scenarios, high-throughput samplers are of paramount importance. To achieve a high-throughput platform, we add two domain-specific instructions with dedicated hardware whose functions are extensively used in MCMC algorithms. These instructions bring down the number of clock cycles needed to implement the respective functions by 10x and 21x. A 64-cluster architecture of the MC3A is fully placed and routed in 65 nm, TSMC CMOS technology, where the VLSI layout of each cluster occupies an area of 0.577 mm^2 while consuming a power of 247 mW running at 1 GHz clock frequency. Our proposed MC3A achieves 6x higher throughput than its equivalent predecessor (PENC) and consumes 4x lower energy per sample. Also, when compared to other off-the-shelf platforms, such as Jetson TX1 and TX2 SoC, MC3A results in 195x and 191x higher throughput and consumes 808x and 726x lower energy per sample generation, respectively.

In this special discussion session on machine learning, the panel members discuss various issues related to building secure and low power neuromorphic systems. The security of neuromorphic systems may be discussed in term of the reliability of the model, trust in the model, and security of the underlying hardware. The low power aspect of neuromorphic computing systems may be discussed in terms of adaptation of new devices and technologies, the adaptation of new computational models, development of heterogeneous computing frameworks, or dedicated engines for processing neuromorphic models. This session may include discussion on the design space of such supporting hardware, exploring tradeoffs between power/energy, security, scalability, hardware area, performance, and accuracy.

As deep learning networks mature and improve classification performance, a significant challenge is their deployment in embedded settings. Modern network typologies, such as convolutional neural networks, can be very deep and impose considerable complexity that is often not feasible in resource bound, real-time systems. Processing of these networks requires high levels of parallelization, maximizing data throughput, and support for different network types, while minimizing power and resource consumption. In response to these requirements, in this paper, we present a low power FPGA based neural network accelerator named SCALENet: a SCalable Low power AccELerator for real-time deep neural Networks. Key features include optimization for power with coarse and fine grain scheduler, implementation flexibility with hardware only or hardware/software co-design, and acceleration for both fully connected and convolutional layers. The experimental results evaluate SCALENet against two different neural network applications: image processing, and biomedical seizure detection. The image processing networks, implemented on SCALENet, trained on the CIFAR-10 and ImageNet datasets with eight different networks, are implemented on an Arty A7 and Zedboard#8482; FPGA platforms. The highest improvement came with the Inception network on an ImageNet dataset with a throughput of 22x and decrease in energy consumption of 13x compared to the ARM processor implementation. We then implement SCALENet for time series EEG seizure detection using both a Direct Convolution and FFT Convolution method to show its design versatility with a 99.7% reduction in execution time and a 97.9% improvement in energy consumption compared to the ARM. Finally, we demonstrate the ability to achieve parity with or exceed the energy efficiency of NVIDIA GPUs when evaluated against Jetson TK1 with embedded GPU System on Chip (SoC) and with a 4x power savings in a power envelope of 2.07 Watts.

In this work, we present a completely new cognitive computing approach to real-time and online seizure detection with minimal power consumption and latency, suitable for wearable devices. We use Metropolis-Hastings (MH) sampler, one of the Markov-Chain Monte-Carlo (MCMC) algorithms, to sample from a probability distribution function (PDF) that adapts its parameters to a patient's real-time signals, to predict the occurrence of seizures in their brain with a minimal power consumption hardware implementable on tiny Field Programmable Gate Arrays (FPGA). We use Gaussian Mixture Model (GMM) to model the likelihood PDF, whose parameters are tuned to the patient's real-time signals, and over which the MH sampler generates samples with respect to time. The generated samples are compared with the actual online signals to detect the occurrence of seizures in the brain. With this model, we achieved an average seizure prediction accuracy of 81.47%, an average Sensitivity of 90% and onset Sensitivity of 100%, that outperform those of traditional machine learning algorithms such as Support Vector Machine (SVM). The complete system has been implemented using Artix7 FPGA minimizing energy/power requirements with a logic utilization of 1198 slices and a dynamic power of 3.37 uW, that outperforms by 3 times and 39 times respectively to those of the SVM implementation on a similar platform.

Three-dimensional DRAMs (3D-DRAMs) are emerging as a promising solution to address the memory wall problem in computer systems. However, high fabrication cost per bit and thermal issues are the main reasons that prevent architects from using 3D-DRAM alone as the main memory building block. In this article, we address this issue by proposing a heterogeneous memory system that combines a double data rate (DDRx) DRAM with an emerging 3D hybrid memory cube (HMC) technology. Bandwidth and temperature management are the challenging issues for this heterogeneous memory architecture. To address these challenges, first we introduce a memory page allocation policy for the heterogeneous memory system to maximize performance. Then, using the proposed policy, we introduce a temperature-aware algorithm that dynamically distributes the requested bandwidth between HMC and DDRx DRAM to reduce the thermal hotspot while maintaining high performance. We take into account the impact of both core count and HMC channel count on performance while using the proposed policies. The results show that the proposed memory page allocation policy can utilize the memory bandwidth close to 99% of the ideal bandwidth utilization. Moreover, our temperate-aware bandwidth adaptation reduces the average steady-state temperature of the HMC hotspot across various workloads by 4.5 K while incurring 2.5% performance overhead.

Big data processing on hardware gained immense interest among the hardware research community to take advantage of fast processing and reconfigurability. Though the computation latency can be reduced using hardware, big data processing cost is dominated by data transfers. In this article, we propose a low overhead framework based on compressive sensing (CS) to reduce data transfers up to 67% without affecting signal quality. CS has two important kernels: “sensing” and “reconstruction.” In this article, we focus on CS reconstruction is using orthogonal matching pursuit (OMP) algorithm. We implement the OMP CS reconstruction algorithm on a domain-specific PENC many-core platform and a low-power Jetson TK1 platform consisting of an ARM CPU and a K1 GPU. Detailed performance analysis of OMP algorithm on each platform suggests that the PENC many-core platform has 15× and 18× less energy consumption and 16× and 8× faster reconstruction time as compared to the low-power ARM CPU and K1 GPU, respectively. Furthermore, we implement the proposed CS-based framework on heterogeneous architecture, in which the PENC many-core architecture is used as an “accelerator” and processing is performed on the ARM CPU platform. For demonstration, we integrate the proposed CS-based framework with a hadoop MapReduce platform for a face detection application. The results show that the proposed CS-based framework with the PENC many-core as an accelerator achieves a 26.15% data storage/transfer reduction, with an execution time and energy consumption overhead of 3.7% and 0.002%, respectively, for 5,000 image transfers. Compared to the CS-based framework implementation on the low-power Jetson TK1 ARM CPU+GPU platform, the PENC many-core implementation is 2.3× faster for the image reconstruction part, while achieving 29% higher performance and 34% better energy efficiency for the complete face detection application on the Hadoop MapReduce platform.

Wearable personalized health monitoring systems can offer a cost-effective solution for human health care. These systems must constantly monitor patients' physiological signals and provide highly accurate, and quick processing and delivery of the vast amount of data within a limited power and area footprint. These personalized biomedical applications require sampling and processing multiple streams of physiological signals with a varying number of channels and sampling rates. The processing typically consists of feature extraction, data fusion, and classification stages that require a large number of digital signal processing (DSP) and machine learning (ML) kernels. In response to these requirements, in this paper, a tiny, energy-efficient, and domain-specific manycore accelerator referred to as power-efficient nanoclusters (PENC) is proposed to map and execute the kernels of these applications. Simulation results show that the PENC is able to reduce energy consumption by up to 80% and 25% for DSP and ML kernels, respectively, when optimally parallelized. In addition, we fully implemented three compute-intensive personalized biomedical applications, namely, multichannel seizure detection, multiphysiological stress detection, and standalone tongue drive system (sTDS), to evaluate the proposed manycore performance relative to commodity embedded CPU, graphical processing unit (GPU), and field-programmable gate array (FPGA)-based implementations. For these three case studies, the energy consumption and the performance of the proposed PENC manycore, when acting as an accelerator along with an Intel Atom processor as a host, are compared with the existing commercial off-the-shelf general-purpose, customizable, and programmable embedded platforms, including Intel Atom, Xilinx Artix-7 FPGA, and NVIDIA TK1 advanced RISC machine -A15 and K1 GPU system on a chip. For these applications, the PENC manycore is able to significantly improve throughput and energy efficiency by up to 1872× and 276×, respectively. For the most computational intensive application of seizure detection, the PENC manycore is able to achieve a throughput of 15.22 giga-operations-per-second (GOPs), which is a 14× improvement in throughput over custom FPGA solution. For stress detection, the PENC achieves a throughput of 21.36 GOPs and an energy efficiency of 4.23 GOP/J, which is 14.87× and 2.28× better over FPGA implementation, respectively. For the sTDS application, the PENC improves a throughput by 5.45× and an energy efficiency by 2.37× over FPGA implementation.

Ever-growing IoT demands big data processing and cognitive computing on mobile and battery operated devices. However, big data processing on low power embedded cores is challenging due to their limited communication bandwidth and on-chip storage. Additionally, IoT and cloud-based computing demand low overhead security kernel to avoid data breaches. In this paper, we propose a Light-weight Encryption using Scalable Sketching (LESS) framework for big data sketching and encryption using One-Time Random Linear Projections (OTRLP). OTRLP encoded matrix makes the Known Plaintext Attacks (KPA) ineffective, and attackers cannot gain significant information from plaintext-ciphertext pair. LESS framework can reduce data up to 67% with 3.81 dB signal-to-reconstruction error rate (SRER). This framework has two important kernels “sketching” and “sketch-reconstruction”, the latter is computationally intensive and costly. We propose to accelerate the sketch reconstruction using Orthogonal Matching Pursuit (OMP) on a domain specific many-core hardware named Power Efficient Nano Cluster (PENC) designed by authors of this paper. To demonstrate efficiency of LESS framework, we integrate it with Hadoop MapReduce platform for objects and scenes identification application. The full hardware integration consists of tiny ARM cores which perform task scheduling and objects identification application, while PENC acts as an accelerator for sketch reconstruction. The full hardware integration results show that the LESS framework achieves 46% reduction in data transfers with very low execution overhead of 0.11% and negligible energy overhead of 0.001% when tested for 2.6 GB streaming input data. The heterogeneous LESS framework requires 2× less transfer time and achieves 2.25× higher throughput per watt compared to MapReduce platform.

The rapid growth in the data yields challenges to process data efficiently using current high-performance server architectures such as big Xeon cores. Furthermore, physical design constraints, such as power and density, have become the dominant limiting factor for scaling out servers. Heterogeneous architectures that combine big Xeon cores with little Atom cores have emerged as a promising solution to enhance energy-efficiency by allowing each application to run on an architecture that matches resource needs more closely than a one-size-fits-all architecture. Therefore, the question of whether to map the application to big Xeon or little Atom in heterogeneous server architecture becomes important. In this paper, we characterize Hadoop-based applications and their corresponding MapReduce tasks on big Xeon and little Atom-based server architectures to understand how the choice of big vs little cores is affected by various parameters at application, system and architecture levels and the interplay among these parameters. Furthermore, we have evaluated the operational and the capital cost to understand how performance, power and area constraints for big data analytics affects the choice of big vs little core server as a more cost and energy efficient architecture.

Markov Chain Monte Carlo (MCMC) algorithms are used to obtain samples from any target probability distribution and are widely used in stochastic processing techniques. Stochastic processing techniques such as machine learning and image processing need to compute large amounts of data in real-time, thus high throughput MCMC samplers are of utmost importance. Parallel Tempering (PT) MCMC has proven better mixing and convergence for high-dimensional and multi-modal distributions compared to other popular MCMC algorithms. In this paper, we employ a special case of Dth order Markov chains to modify the PT-MCMC algorithm, named "Multiple Parallel Tempering" (MPT). The modification converts one MCMC sampler into multiple independent samplers that generate and interleave their samples on one output line each clock cycle. A fully scalable and pipelined hardware accelerator for the PT and proposed MPT sampler is designed and implemented on Artix-7 Xilinx FPGA for chain numbers of 1, 2, and 8. The post-place and route FPGA implementation results indicate that the throughput of the proposed MPT sampler for chain numbers 1, 2, and 8 achieves 31x, 31x, and 28x respectively higher as compared to PT sampler with the same chain number configuration.

This paper presents a stand-alone Dual-mode Tongue DriveSystem (sdTDS) which is designed for people with severedisabilities to control their environment using their tonguemotion and speech. The sdTDS detects user's tongue motion using a magnetic tracer placed on tongue and an array of magnetic sensors embedded in a wireless headset and at the same time it can capture the user's voice using a small microphone embedded in the same headset. A real-time FPGA-based local processor is proposed which can perform all required signal processing at sensor side, rather than sending all raw data out to a PC or smartphone. The proposed sdTDS significantly reduces the transmitter power consumption and subsequently increases the battery life.

FueXeA by ILSVRC and COCO competitions, Convolutional Neural Network (CNN) has become important in computer vision, and natural language processing. However state-of-the-art CNNs are computationally and memory intensive, thus energy efficient implementation on embedded platform is challenging. Recently VGGNet and ResNet showed that deep neural networks with more convolution layers (CV) and few fully connected layer (FC) can achieve lower error rates, thus reducing the complexity of convolution layers is of utmost importance. To reduce computations and shared memory usage in convolution layers, in this paper we evaluate the performance of direct convolution (Direct-Conv), Fast Fourier Transform (FFT) based convolution (FFT-Conv), and Overlap and Add FFT convolution (FFT-OVA-Conv) in embedded architecture including a low power domain specific many-core architecture called Power Efficient Nano Clusters (PENC) and ARM Cortex A53 CPU. To demonstrate the efficiency of FFT-Conv and FFT-OVA-Conv, we map ResNet-20 for the CIFAR-10 dataset on PENC as well as in ARM Cortex A53 CPU. Results are evaluated and compared with respect to throughput per watt, energy delay product, and execution time for three methods. Using built-in FFT instruction in PENC, the FFT-OVA-Conv performs 2.9× and 1.65× faster and achieves 6.7× and 2.3× better throughput per watt than Direct-Conv and FFT-Conv respectively. In ARM A53 CPU, the FFT-OVA-Conv achieves 3.36× and 1.38× improvement in execution time and 2.72× and 1.32× better throughput than Direct-Conv and FFT-Conv.

Electroencephalogram (EEG) data is used for a variety of purposes, including brain-computer interfaces, disease diagnosis, and determining cognitive states. Yet EEG signals are susceptible to noise from many sources, such as muscle and eye movements, and motion of electrodes and cables. Traditional approaches to this problem involve supervised training to identify signal components corresponding to noise so that they can be removed. However these approaches are artifact specific. In this paper, we present a novel software-hardware system that uses a weak supervisory signal to indicate that some noise is occurring, but not what the source of the noise is or how it is manifested in the EEG signal. The EEG data is decomposed into independent components using ICA, and these components form bags that are labeled and classified by a multi-instance learning algorithm that can identify the noise components for removal to reconstruct a clean EEG signal. We also performed extensive hyperparameter optimization for the model with the goal of improving accuracy without increasing execution time. This resulted the execution time to be reduced from 282 s to 8.8 s when running the model on an embedded ARM CPU processor at 1.6 GHz clock frequency. In this paper, we present the overall system which includes ICA, SAX and MIL, along with preliminary results for software and hardware implementation when using real EEG data from 64 electrodes. The proposed system consumes 909 mW power during processing above a baseline of 2.32 W idle, while achieving 91.2% artifact identification accuracy.

Lightweight convolutional neural network (CNN) on tiny embedded platforms can offer energy efficient solution for today's IoT devices. However, CNN implementation on embedded system faces processing bottleneck in convolutional layers and memory storage issues in fully connected layers. In past years, heterogeneous acceleration, where compute intensive tasks are performed on kernel specific cores, has gained attention. In this paper we propose, a domain specific and programmable accelerator “PACENet”-Programmable many-core ACcElerator for convolution neural Network architecture. It consists of neural network kernel specific instruction set architecture such as convolution, maxpool and relu. To demonstrate efficiency of the proposed PACENet, we implemented ResNet-20 for CIFAR-10 dataset, where PACENet performs convolution layer, Relu activations, Maxpool layer, and fully-connected layer. We also implemented ResNet-20 for CIFAR-10 dataset on NVIDIA TX1 mobile GPU platform using Tensorflow and cuDNN libraries. Compared to NVIDIA TX1 platform implementation PACENet platform implementation performs 1.4× to 4.5× faster and saves 2.8× to 9× energy consumption respectively. PACENet achieves 2.9× to 9.3× higher throughput per watt as compared to TX1 platform implementation.

Deep neural networks have been shown to outperform prior state-of-the-art solutions that often relied heavily on hand-engineered feature extraction techniques coupled with simple classification algorithms. In particular, deep convolutional neural networks have been shown to dominate on several popular public benchmarks such as the ImageNet database. Unfortunately, the benefits of deep networks have yet to be fully exploited in embedded, resource-bound settings that have strict power and area budgets. Graphical processing unit (GPU) have been shown to improve throughput and energy-efficiency over central processing unit (CPU) due to their highly parallel architecture yet still impose a significant power burden. In a similar fashion, field programmable gate array (FPGA) can be used to improve performance while further allowing more fine-grained control over implementation to improve efficiency. In order to reduce power and area while still achieving required throughput, classification-efficient network architectures are required in addition to optimal deployment on efficient hardware. In this work, we target both of these enterprises. For the first objective, we analyze simple, biologically inspired reduction strategies that are applied both before and after training. The central theme of the techniques is the introduction of sparsification to help dissolve away the dense connectivity that is often found at different levels in convolutional neural networks. The sparsification techniques include feature compression partition , structured filter pruning , and dynamic feature pruning . Additionally, we explore filter factorization and filter quantization approximation techniques to further reduce the complexity of convolutional layers. In the second contribution, we propose SPARCNet, a hardware accelerator for efficient deployment of SPAR se C onvolutional NET works. The accelerator looks to enable deploying networks in such resource-bound settings by both exploiting efficient forms of parallelism inherent in convolutional layers and by exploiting the sparsification and approximation techniques proposed. To demonstrate both contributions, modern deep convolutional network architectures containing millions of parameters are explored within the context of the computer vision dataset CIFAR. Utilizing the reduction techniques, we demonstrate the ability to reduce computation and memory by 60% and 93% with less than 0.03% impact on accuracy when compared to the best baseline network with 93.47% accuracy. The SPARCNet accelerator with different numbers of processing engines is implemented on a low-power Artix-7 FPGA platform. Additionally, the same networks are optimally implemented on a number of embedded commercial-off-the-shelf platforms including NVIDIAs CPU+GPU SoCs TK1 and TX1 and Intel Edison. Compared to NVIDIAs TK1 and TX1, the FPGA-based accelerator obtains 11.8 × and 7.5 × improvement in energy efficiency while maintaining a classification throughput of 72 images/s. When further compared to a number of recent FPGA-based accelerators, SPARCNet is able to achieve up to 15 × improvement in energy efficiency while consuming less than 2W of total board power at 100MHz. In addition to improving efficiency, the accelerator has built-in support for sparsification techniques and ability to perform in-place rectified linear unit (ReLU) activation function, max-pooling, and batch normalization.

The four V's in Big data sets, Volume, Velocity, Variety, and Veracity, provides challenges in many different aspects of real-time systems. Out of these areas securing big data sets, reduction in processing time and communication bandwidth are of utmost importance. In this paper we adopt Compressive Sensing (CS) based framework to address all three issues. We implement compressive Sensing using Deterministic Random Matrix (DRM) on Artix-7 FPGA, and CS reconstruction using Orthogonal Matching Pursuit (OMP) algorithm on Virtex-7 FPGA. The results show that our implementations for CS sampling and reconstruction are 183x and 2.7x respectively faster when compared to previously published work. We also perform case study of two different applications i.e. multi-channel Seizure Detection and Image processing to demonstrate the efficiency of our proposed CS-based framework. CS-based framework allows us to reduce communication transfers up to 75% while achieving satisfactory range of quality. The results show that our proposed framework is 290x faster and has 7.9x less resource utilization as compared to previously published AES based encryption.

Deep neural networks have been shown to outperform prior state-of-the-art solutions that often relied heavily on hand-engineered feature extraction techniques coupled with simple classification algorithms. In particular, deep max-pooling convolutional neural networks (MPCNN) have been shown to dominate on several popular public benchmarks. Unfortunately, the benefits of deep networks have yet to be exploited in embedded, resource-bound settings that have strict power and area budgets. GPUs have been shown to improve throughput and energy-efficiency over CPUs due to their parallel architecture. In a similar fashion, FPGAs can improve performance while allowing more fine control over implementation. In order to meet power, area, and latency constraints, it is necessary to develop network reduction strategies in addition to optimal mapping. This work looks at two specific reduction techniques including limited precision for both fixed-point and floating-point formats, and performing weight matrix truncation using singular value decomposition. An FPGA-based framework is also proposed and used to deploy the trained networks. To demonstrate, a handful of public computer vision datasets including MNIST, CIFAR-10, and SVHN are fully implemented on a low-power Xilinx Artix-7 FPGA. Experimental results show that all networks are able to achieve a classification throughput of 16 img/sec and consume less than 700 mW when running at 200 MHz. In addition, the reduced networks are able to, on average, reduce power and area utilization by 37% and 44%, respectively, while only incurring less than 0.20% decrease in accuracy.

Almost 90% of the data available today was created within the last couple of years, thus Big Data set processing is of utmost importance. Many solutions have been investigated to increase processing speed and memory capacity, however I/O bottleneck is still a critical issue. To tackle this issue we adopt Sketching technique to reduce data communications. Reconstruction of the sketched matrix is performed using Orthogonal Matching Pursuit (OMP). Additionally we propose Gradient Descent OMP (GD-OMP) algorithm to reduce hardware complexity. Big data processing at real-time imposes rigid constraints on sketching kernel, hence to further reduce hardware overhead both algorithms are implemented on a low power domain specific many-core platform called Power Efficient Nano Clusters (PENC). GD-OMP algorithm is evaluated for image reconstruction accuracy and the PENC many-core architecture. Implementation results show that for large matrix sizes GD-OMP algorithm is 1.3x faster and consumes 1.4x less energy than OMP algorithm implementations. Compared to GPU and Quad-Core CPU implementations the PENC many-core reconstructs 5.4x and 9.8x faster respectively for large signal sizes with higher sparsity.

Wearable personal health monitoring systems can offer a cost effective solution for human healthcare. These systems must provide both highly accurate, secured and quick processing and delivery of vast amount of data. In addition, wearable biomedical devices are used in inpatient, outpatient, and at home e-Patient care that must constantly monitor the patient's biomedical and physiological signals 24/7. These biomedical applications require sampling and processing multiple streams of physiological signals with strict power and area footprint. The processing typically consists of feature extraction, data fusion, and classification stages that require a large number of digital signal processing and machine learning kernels. In response to these requirements, in this paper, a low-power, domain-specific many-core accelerator named Power Efficient Nano Clusters (PENC) is proposed to map and execute the kernels of these applications. Experimental results show that the manycore is able to reduce energy consumption by up to 80% and 14% for DSP and machine learning kernels, respectively, when optimally parallelized. The performance of the proposed PENC manycore when acting as a coprocessor to an Intel Atom processor is compared with existing commercial off-the-shelf embedded processing platforms including Intel Atom, Xilinx Artix-7 FPGA, and NVIDIA TK1 ARM-A15 with GPU SoC. The results show that the PENC manycore architecture reduces the energy by as much as 10X while outperforming all off-the-shelf embedded processing platforms across all studied machine learning classifiers.

Multi-Sensor health monitoring systems are used to predict near future events of our health system. Each sensor generates humongous amount of data per second and needs to be processed in real-time. At the same time health monitoring systems are battery operated, thus they have rigid constraints on power and area of processing platform. Additionally, health monitoring systems should be accurate, thus we adapt machine learning techniques to improve detection accuracy. We propose a programmable Big Data Processing framework to reduce on-chip communications and computations, thus reducing energy of the processing. We integrate a low-overhead sketching framework with a low-power programmable PENC many-core platform. The sketching technique reduces the data communications and computations, additionally processing time is scaled down by parallel processing on the many-core platform. For demonstration we show seizure detection application with 22-channel of electroencephalograph (EEG), each channel generates 256 samples per second requiring total of 88 Kbps data rate. The computations are reduced by 16× while energy consumption of processing is reduced up to 68%. For compression rates of 2-16×, the seizure detection performance for sensitivity and specificity is degraded by 2.07% and 2.97%, respectively for Logistic Regression classifier.

The ability to accurately and robustly detect seizures in an ambulatory setting using scalp-based EEG has been the focus of much research over the last several decades. However, its numerous challenges and obstacles have impeded the realization of a definitive solution. This work aims to build upon our existing research and apply newer advanced machine learning and hardware techniques to this issue. The novelty proposed is two-fold. First, we utilize max-pooling convolutional neural networks (MPCNN) to perform end-to-end learning. Second, we demonstrate that transfer-learning can be used to teach MPCNNs generalized features of both normal and epileptiform patterns from a pool of subjects' raw EEG data. Using this hybrid approach, the system is able to detect all 184 seizure onsets from 24 cases with average latency of 1.47 seconds and 3.2 false-alarms/day. To demonstrate the full system, the entire design is efficiently implemented onto a highly parallel, highly granular embedded SoC (NVIDIA Jetson TK1). When utilizing the GPU, the system is able to classify 15-second segments in 308 μs and last over 80 hours.

In this article, we propose a real-time anomaly detection framework for an NoC-based many-core architecture. We assume that processing cores and memories are safe and anomaly is included through a communication medium (i.e., router). The article targets three different attacks, namely, traffic diversion, route looping, and core address spoofing attacks. The attacks are detected by using machine-learning techniques. Comprehensive analysis on machine-learning algorithms suggests that Support Vector Machine (SVM) and K-Nearest Neighbor (K-NN) have better attack detection efficiency. It has been observed that both algorithms have accuracy in the range of 94% to 97%. Additional hardware complexity analysis advocates SVM to be implemented on hardware. To test the framework, we implement a condition-based attack insertion module; attacks are performed intra- and intercluster. The proposed real-time anomaly detection framework is fully placed and routed on Xilinx Virtex-7 FPGA. Postplace and -route implementation results show that SVM has 12% to 2% area overhead and 3% to 1% power overhead for the quad-core and 16-core implementation, respectively. It is also observed that it takes 25% to 18% of the total execution time to detect an anomaly in transferred packets for quad-core and 16-core, respectively. The proposed framework achieves 65% reduction in area overhead and is 3 times faster compared to previous published work.

Compressive Sensing (CS) signal reconstruction can be implemented using convex relaxation, non-convex, or local optimization algorithms. Though the reconstruction using convex optimization, such as the Iterative Hard Thresholding algorithm, is more accurate than matching pursuit algorithms, most researchers focus on matching pursuit algorithms because they are less computationally complex. Orthogonal Matching Pursuit (OMP) is a greedy algorithm, which solves the problem by choosing the most significant variable to reduce the least square error. In this paper, we propose an efficient parallel architecture for OMP CS reconstruction. For architecture implementation, we perform measurement and sparsity analysis to reduce the complexity. The proposed architecture is platform independent and is implemented on 7 different platforms including general purpose CPUs, GPUs, a Virtex-7 FPGA and a domain specific many-core. The implementation results indicate that reconstruction time on FPGA is improved by 3× compared to previous FPGA implementation, whereas GPU implementation is 4× faster than the previously proposed GPU-based OMP architecture. The CPU implementation is 6× faster, compared with previous CPU-based implementation. The domain specific many-core acheives 24 times faster reconstruction time when compared to both GPU and CPU implementations.

Unavailability of functional units is a major performance bottleneck in general-purpose processors (GPP). In a GPP with limited number of functional units while a functional unit may be heavily utilized at times, creating a performance bottleneck, the other functional units might be under-utilized. We propose a novel idea for adapting functional units in GPP architecture in order to overcome this challenge. For this purpose, a selected set of complex functional units that might be under-utilized such as multiplier and divider, are realized using a programmable look up table-based fabric. This allows for run-time adaptation of functional units to improving performance. The programmable look up tables are realized using magnetic tunnel junction (MTJ) based memories that dissipate near zero leakage and are CMOS compatible. We have applied this idea to a dual issue architecture. The results show that compared to a design with all CMOS functional units a performance improvement of 18%, on average is achieved for standard benchmarks. This comes with 4.1% power increase in integer benchmarks and 2.3% power decrease in floating point benchmarks, compared to a CMOS design.

Compressive Sensing (CS) is a novel scheme, in which a signal that is sparse in a known transform domain can be reconstructed using fewer samples. However, the signal reconstruction techniques are computationally intensive and power consuming, which make them impractical for embedded applications. This work presents a parallel and reconfigurable architecture for Orthogonal Matching Pursuit (OMP) algorithm, one of the most popular CS reconstruction algorithms. In this paper, we are proposing the first reconfigurable OMP CS reconstruction architecture which can take different image sizes with sparsity up to 32. The aim is to minimize the hardware complexity, area and power consumption, and improve the reconstruction latency while meeting the reconstruction accuracy. First, the accuracy of reconstructed images is analyzed for different sparsity values and fixed point word length reduction. Next, efficient parallelization techniques are applied to reconstruct signals with variant signal lengths of N. The OMP algorithm is mainly divided into three kernels, where each kernel is parallelized to reduce execution time, and efficient reuse of the matrix operators allows us to reduce area. The proposed architecture can reconstruct images of different sizes and measurements and is implemented on a Xilinx Virtex 7 FPGA. The results indicate that, for a 128x128 image reconstruction, the proposed reconfigurable architecture is 2.67x to 1.8x faster than the previous non-reconfigurable work which is less complex and uses much smaller sparsity.

The variability of deep-submicron technologies creates systems with asymmetric cores from a frequency and leakage power viewpoint, which makes an opportunity for performance-power optimization. In particular, process variation can transform a homogeneous many-core platform into a heterogeneous system where the task mapping is NP-hard problem. In this paper, we propose a mapping algorithm that selects the appropriate task mapping along with voltage and frequency assignment for a cluster of cores. The mapping algorithm, which is based on simulated annealing, determines cluster voltages and core frequencies to minimize energy consumption and EDP under process variation. We examine the effectiveness of our proposed algorithm on a fully placed and routed 128-core biomedical accelerator in 45 nm when running various applications including compressive sensing, seizure detection and ultrasound spectral Doppler and linear regression. The results indicate that exposing frequency and power variation to the mapping algorithm results in up to 22% (on average 11%) energy saving and 31% (on average19%) EDP improvement.

Pulsed wave (PW) Doppler ultrasound is a common technique used for making non-invasive velocity measurements of blood flow in humans. Most current PW Doppler ultrasound designs rely on fixed signal processing hardware; greatly limiting their versatility. This paper presents a highly efficient and highly versatile FPGA-based PW spectral Doppler ultrasound system. The system is implemented on a Virtex-5 FPGA using Xilinx's ISE design suite. In order to measure the accuracy of the system, a similar design was implemented in MATLAB. Furthermore, the design was also implemented in 65 nm CMOS ASIC design for performance comparisons. The Virtex-5 design requires 1,159 of 17,280 slice resources and consumes 1.089 watts of power when running at its maximum clock speed of 333 megahertz. The ASIC design has an area of .573 mm2and consumes 41 mW of power at a maximum clock speed of 1 GHz.

This paper presents a reconfigurable many-core platform performing fixed point DSP applications supporting up to 64 cores routed in a hierarchical network. To demonstrate an application, electroencephalogram (EEG) seizure detection and analysis is mapped onto the cores. The individual cores are based on a 5 stage RISC pipeline architecture optimized to support communication to other cores on the platform. To reconfigure the platform, programs are loaded onto each of the cores. Communication between cores is implemented using low-area routers that partitions computational cores into hierarchical clusters resulting in a low network diameter. The routers use a packet-switched protocol that minimizes circuitry which further reduces circuit size in comparison to the computational circuitry. A globally asynchronous, locally synchronous (GALS) architecture is implemented to eliminate global clock routing which consumes high levels of power due to long propagation and thus high capacitive loading from many cores. Additionally, cores not configured for an application has its local clock disabled which turns off unused cores. The overall result is a platform with lower power consumption than a traditional single core DSP with the reconfigurability lacking in an ASIC. Applications tested within the mapping include the Fast Fourier Transform (FFT) and Finite Impulse Response (FIR) filter. The seizure detection and analysis algorithm, when mapped onto the many-core platform, takes 5663 cycles to execute in 14.45 µs. The prototype SoC is implemented in 65 nm CMOS which contains 64 cores and occupies 8.41 mm2.

This paper presents a high performance architecture for the reconstruction of compressive sampled signals using Orthogonal Matching Pursuit (OMP) algorithm. Q-R decomposition (QRD) process is used for the matrix inverse core and a new algorithm for finding fast inverse square root of a fixed point number is also implemented to support the QRD process. The optimized architecture takes 256-length input vector and 64 measurement data, and reconstructs a signal of sparsity 8. The design is implemented in 65 nm CMOS which runs at 165 MHz and occupies 0.69 mm2, total reconstruction takes 13.7 μs. The implementation on Xilinx FPGA Virtex-5 takes 27.12 μs to reconstruct a 256-length signal of sparsity 8. The same architecture for 128-length signal of sparsity 5 on Virtex-5 is 2.4 times faster than the state-of-the-art implementation.

An efficient technique for early detection of undecodable blocks during LDPC decoding is introduced. The proposed method avoids unnecessary decoding iterations by predicting decoding failure and therefore results in significant improvement in power and latency in low SNR values. The proposed method which has a low hardware overhead compares the parity checksum against predefined threshold values for three iterations and terminates decoding if a condition is met. A 5.25 mm210GBASE-T Split-Row Threshold decoder is implemented using the proposed technique in 65 nm CMOS. The postlayout results show that at low SNR value of 3.0 dB, the decoder requires 2.3 times fewer decoding iterations which results in 23 pJ/bit energy dissipation. This is 2.4 times lower than the energy dissipation of Split-Row Threshold decoder without the proposed early stopping technique.

We present an improved thresholding LDPC decoding algorithm which outperforms the split-row and original split-row threshold decoders with a small increase in hardware. Simulation results show that the algorithm provides 0.27- 0.50 dB coding gain over split-row, 0.10-0.20 dB over split-row threshold, and is within 0.08-0.13 dB of SPA. Compared with the original threshold algorithm the check node processor's gate count is increased by 3% while total chip area is kept the same.

The recently introduced Split-Row Threshold algorithm significantly improves the error performance when compared to the non- threshold Split-Row algorithm while requiring a very small increase in hardware complexity. The Multi-Split-Row Threshold decoding algorithm presented in this paper enables further reductions in routing complexity for greater throughput and smaller circuit area implementations. Several Multi-Split-Row Threshold decoder designs have been implemented in 65 nm CMOS and the impact of the different levels of partitioning on error performance, wire interconnect complexity, decoder area, and speed are investigated. The Split-Row-16 Threshold decoder occupies 3.8 mm2, runs at 100 MHz, delivers a throughput of 13.8 Gbps at 15 iterations and is only 0.28 dB and 0.22 dB away from SPA and MinSum Normalized.

We propose the "multi-split-row'" LDPC decoding method which allows further reductions in routing complexity, greater throughput, and smaller circuit area implementations compared to the previously proposed split-row decoding method. Multi-split-row is especially useful for regular high row weight LDPC codes. A 2048-bit full parallel decoder is implemented in a 0.18 μm CMOS technology using standard MinSum, split-row-2 and split-row-4 methods. The split-row-4 decoder delivers 7.1 Gbps throughput with 15 decoding iterations, and has 3.2 times smaller circuit area and 5.2 times higher throughput than the standard MinSum decoder.

A robust, scalable, and power efficient dual-clock first-input first-out (FIFO) architecture which is useful for transferring data between modules operating in different clock domains is presented. The architecture supports correct operation in applications where multiple clock cycles of latency exist between the data producer, FIFO, and the data consumer; and with arbitrary clock frequency changes, halting, and restarting in either or both clock domains. The architecture is demonstrated in both a 0.18- mum CMOS full-custom design and a 0.18-mum CMOS standard cell design used in a globally asynchronous locally synchronous array processor. It achieves 580-MHz operation and 10.3-mW power dissipation while performing simultaneous FIFO read and write operations at 1.8 V.

A reduced complexity LDPC decoding method is presented that dramatically reduces wire interconnect complexity, which is a major issue in LDPC decoders. The proposed split-row method makes column processing parallelism easier to exploit, doubles available row processor parallelism, and significantly simplifies row processors - which results in smaller area, higher speeds, and lower energy dissipation. Simulation results over an additive white Gaussian channel show that the error performance of high row-weight codes with split-row decoding is within 0.3-0.6 dB of the min-sum and sum-product decoding algorithms. A full parallel decoder for a (3,6) LDPC code with a code length of 1536 bits is implemented in a 0.18 mum CMOS technology twice: once using the split-row method, and once using the min-sum algorithm for comparison. The split-row decoder operates at 53 MHz and delivers a throughput of 5.4 Gbps with 15 decoding iterations per block. The split-row decoder is about 1.3 times smaller, has an average wire length 1.5 times shorter, and has a throughput 1.6 times higher than the min-sum decoder.