Edge platforms in autonomous systems integrate multiple sensors to interpret their environment. The high-resolution and high-bandwidth pixel arrays of these sensors improve sensing quality but also generate a vast, and arguably unnecessary, volume of real-time data. This challenge, often referred to as the analog data deluge, hinders the deployment of high-quality sensors in resource-constrained environments. This paper discusses the concept of cognitive sensing, which learns to extract low-dimensional features directly from high-dimensional analog signals, thereby reducing both digitization power and generated data volume. First, we discuss design methods for analog-to-feature extraction (AFE) using mixed-signal compute-in-memory. We then present examples of cognitive sensing, incorporating signal processing or machine learning, for various sensing modalities including vision, Radar, and Infrared. Subsequently, we discuss the reliability challenges in cognitive sensing, taking into account hardware and algorithmic properties of AFE. The paper concludes with discussions on future research directions in this emerging field of cognitive sensors.

Modeling the large-scale dynamical systems is a computationally expensive task. This is particularly a problem when the focus is solely on understanding the local behavior or state of the systems. Our primary objective is to determine when the propagation of such local interactions will reach a specific region of interest. Although conventional approaches that reconstruct the states of entire dynamic nodes can be used, they may entail unnecessary computational costs. In this paper, we investigate a Structured Latent space for Localized Prediction (SLLP) for the computationally efficient prediction of local behavior in the dynamical systems. The proposed model comprises a CNN encoder to represent the system in a low-dimensional vector, a LSTM module to learn the dynamics in the vector space, and a MLP decoder to predict the future state of a dynamic node. We evaluate the proposed method in the forest fire and stock market models in the task of predicting the burned state of a tree node and buy state of a investor node in future. We compare the proposed model with general ConvLSTM that reconstructs and predicts the entire systems. The proposed model exhibits similar or slightly worse AUC but significantly reduces computational costs, such as FLOPs (×131) and latency (×4.9), than ConvLSTM when predicting a single dynamic node.

One of the primary challenges of deploying deep neural networks (DNNs) is ensuring their reliable performance in unpredictable edge environments, which are often disrupted by a variety of uncertainties and variations. Estimating uncertainty is crucial in order to understand the reliability of task predictions and prevent system failures. However, quantifying uncertainty stemming from non-ideal properties of processing hardware has not yet been thoroughly studied. To address this, we present Cross-Layer Uncertainty Estimator (CLUE), which quantifies task uncertainty originating from both sensing/processing hardware variations and DNN algorithm uncertainty. Our experimental results demonstrate that CLUE provides uncertainty with up to 80.4% less calibration error and only 12% of energy overheads compared to using task DNN solely. Furthermore, CLUE is able to detect unreliable tasks that stem from processing hardware variations, which prior uncertainty estimators were unable to achieve. Finally, we demonstrate an adaptive control of processing hardware using CLUE, which allows a dynamic trade-off control between task accuracy and energy consumption.

Quantifying model uncertainty of Deep Neural Network (DNN) is important to understand the reliability of the model prediction and avoid risks in safety critical applications. Various approaches, including Bayesian neural networks, Monte-Carlo dropout, and ensembles, are suggested to measure the model uncertainty; but with huge computational cost. We present ModelNet, an Artificial Neural Network (ANN) that can estimate spatial/semantic model uncertainties of a DNN based object detection with less computation overhead. ModelNet is a deterministic ANN that distills the predictive distribution of stochastic DNN. Experimental results show that ModelNet can learn the uncertainty estimation from stochastic DNN in various architectures. ModelNet can perform as a probabilistic object detector with 39x-179x less number of operations, or as an uncertainty assistant to a task network with 1.4x more parameters and 38x less number of operations compared to stochastic DNN. Moreover, a case study of uncertainty driven adaptive sensor using ModelNet is presented.

A key challenge for deployment of artificial intelligence (AI) in real-time safety-critical systems at the edge is to ensure reliable performance even in unreliable environments. This paper will present a broad perspective on how to design AI platforms to achieve this unique goal. First, we will present examples of AI architecture and algorithm that can assist in improving robustness against input perturbations. Next, we will discuss examples of how to make AI platforms robust against hardware induced noise and variation. Finally, we will discuss the concept of using lightweight networks as reliability estimators to generate early warning of potential task failures.

There is a growing interest in deploying complex deep neural networks (DNN) in autonomous systems to extract task-specific information from real-time sensor data and drive critical tasks. The perturbations in sensor data due to noise or environmental conditions can lead to errors in information extraction and degrade reliability of the entire autonomous systems. This paper presents a light-weight deep learning plat-form, WarningNet, that operates on sensor data to estimate potential task failures due to spatiotemporal input perturbations. Experimental results show that WarningNet can provide early warning of the performance degradation of different tasks within a fraction of the time required for the task to complete. As a case-study, we show that the early warning can be leveraged to improve the task reliability under adverse condition using on-demand input pre-processing.

A new trend in system-on-chip (SoC) design is chiplet-based IP reuse using 2.5-D integration. Complete electronic systems can be created through the integration of chiplets on an interposer, rather than through a monolithic flow. This approach expands access to a large catalog of off-the-shelf intellectual properties (IPs), allows reuse of them, and enables heterogeneous integration of blocks in different technologies. In this article, we present a highly integrated design flow that encompasses architecture, circuit, and package to build and simulate heterogeneous 2.5-D designs. Our target design is 64core architecture based on Reduced Instruction Set Computer (RISC)-V processor. We first chipletize each IP by adding logical protocol translators and physical interface modules. We convert a given register transfer level (RTL) for 64-core processor into chiplets, which are enhanced with our centralized network-onchip. Next, we use our tool to obtain physical layouts, which is subsequently used to synthesize chip-to-chip I/O drivers and these chiplets are placed/routed on a silicon interposer. Our package models are used to calculate power, performance, and area (PPA) and reliability of 2.5-D design. Our design space exploration (DSE) study shows that 2.5-D integration incurs 1.29× power and 2.19× area overheads compared with 2-D counterpart. Moreover, we perform DSE studies for power delivery scheme and interposer technology to investigate the tradeoffs in 2.5-D integrated chip (IC) designs.

A new trend in complex SoC design is chiplet-based IP reuse using 2.5D integration. In this paper we present a highly-integrated design flow that encompasses architecture, circuit, and package to build and simulate heterogeneous 2.5D designs. We chipletize each IP by adding logical protocol translators and physical interface modules. These chiplets are placed/routed on a silicon interposer next. Our package models are then used to calculate PPA and signal/power integrity of the overall system. Our design space exploration study using our tool flow shows that 2.5D integration incurs 2.1x PPA overhead compared with 2D SoC counterpart.

Deep learning on an edge device requires energy efficient operation due to ever diminishing power budget. Intentional low quality data during the data acquisition for longer battery life, and natural noise from the low cost sensor degrade the quality of target output which hinders adoption of deep learning on an edge device. To overcome these problems, we propose simple yet efficient mixture of pre-processing experts (MoPE) model to handle various image distortions including low resolution and noisy images. We also propose to use adversarially trained auto encoder as a pre-processing expert for the noisy images. We evaluate our proposed method for various machine learning tasks including object detection on MS-COCO 2014 dataset, multiple object tracking problem on MOT-Challenge dataset, and human activity classification on UCF 101 dataset. Experimental results show that the proposed method achieves better detection, tracking and activity classification accuracies under noise without sacrificing accuracies for the clean images. The overheads of our proposed MoPE are 0.67% and 0.17% in terms of memory and computation compared to the baseline object detection network.