Superpixel generation is a common preprocessing step in vision processing aimed at dividing an image into non-overlapping regions. Simple Linear Iterative Clustering (SLIC) is a commonly used superpixel algorithm that offers a good balance between performance and accuracy. However, the algorithm's high computational and memory bandwidth requirements result in performance and energy efficiency that do not meet the requirements of real-time embedded applications. In this work, we explore the design of an energy-efficient superpixel accelerator for real-time computer vision applications. We propose a novel algorithm, Subsampled SLIC (S-SLIC), that uses pixel subsampling to reduce the memory bandwidth by 1.8×. We integrate S-SLIC into an energy-efficient superpixel accelerator and perform an in-depth design space exploration to optimize the design. We completed a detailed design in a 16nm FinFET technology using commercially-available EDA tools for high-level synthesis to map the design automatically from a C-based representation to a gate-level implementation. The proposed S-SLIC accelerator achieves real-time performance (30 frames per second) with 250× better energy efficiency than an optimized SLIC software implementation running on a mobile GPU.

An energy-efficient speech extraction (SE) processor is proposed for the robust speech recognition in the head-mounted display (HMD) systems. Speech extraction is essential for robust speech recognition in noisy environment. For the low-latency speech extraction, FastSE is proposed to overcome 50x larger complex cICA-based selection process which results in <;2ms SE latency. Moreover, a reinforced-FastSE (RFSE) scheme is proposed to achieve 97.2% accuracy with small on-chip memory size of only 33kB for the low-power HMD applications. Also, Reconfigurable matrix operation accelerator (RMAT) is implemented for energy-efficient acceleration of dominant matrix operation on SE. As a result, the proposed SE processor achieves 1.3x lower latency with 4.24x smaller memory compared to the state-of-the-art work, so that speech recognition in noisy environment becomes possible for mobile HMD applications.

In mobile object recognition (OR) applications, the power consumption of image sensor and data communication between image sensor and digital OR processor becomes crucial as digital OR processor consumes less power in deep sub-micron process. To reduce the amount of data transaction from image sensor to digital OR processor, digital/analog mixed-signal focal-plane processing of Binary Robust Invariant Scalable Keypoints (BRISK) feature extraction in CMOS image sensor (CIS) is proposed. The proposed CIS processor sends BRISK feature vectors instead of the whole image pixel data, resulting in 79% reduction of data communication. In this work, mixed-signal processing of corner detection and successive approximation register (SAR)-based scoring are implemented for BRISK feature point detection. To achieve scale-invariance in object recognition, scale-space is generated and stored in analog line memory. In addition, noise reduction scheme is integrated in column processing chain to remove salt and pepper noise, which degrades recognition accuracy. In a post layout simulation, the proposed system achieves 0.70pW/pixel*frame*feature at 30fps in a 130nm CMOS technology, which is 13.6% lower than the state-of-the-art.

A Feature Reuse Engine (FReE) is proposed to achieve low-power object recognition in video frames. Previous object recognition processors perform redundant processing for repeated features of sequential frames in video. Unlike previous works, proposed FReE reuses 58% of features from previous frame with inter-frame similarity. However, false reuse can degrade the object recognition accuracy. By pixel intensity-based Near Pixel Information (NPI), FReE can decide the reusability with 96% of accuracy. Additionally, to minimize latency and power overhead of FReE, a dedicated integral image generator and NPI generator are proposed. As a result, power consumption of object recognition processor is reduced by 31% with the proposed FReE which consumes only 1.92mW in a 130nm CMOS technology.

We propose an analog Radial-Basis-Function (RBF) circuit that generates 4 different types of RBFs, which are spline, Gaussian, multi-quadratic, and log-like spline curves. Moreover, the proposed RBF circuit is designed to have high tunability on centers, heights, and widths. The proposed RBF circuit is also robust to both temperature variation (-37~87□C) and supply voltage variation (1~2V). The sum of area and power consumption of each RBF from 3 different previous works is 13, 622μm2and 121μW, respectively. On the other hand, the proposed circuit occupies only 1,050μm2and consumes 10.5μW which are only 13% and 11.5%, respectively. For its verification, an analog/digital mixed-mode RBF Neural Network (RBFNN) classifier is designed which adopted the proposed RBF circuit.

In this paper, we propose a 60fps cortical vision processor modeling a hierarchical object classification model (HMAX) based on spatio-temporality in video stream. It is hard to implement a real-time hardware for HMAX operation due to its high computational cost from 2-D template matching. Three components are proposed with our improved algorithms. Class Refinement Structure (CRS) dramatically reduces a dimension of HMAX descriptors by 97.01% compared to the previous works by exploiting spatio-temporal features in video action recognition. Spatio-Temporal Memory Structure (STMS) adopts spatially adaptive window technique, and it reduces the required on-chip data bandwidth and computations per a template in S2 stage. In addition, a dual image buffer structure also reduces the required off-chip network bandwidth for processing complex hierarchical stages and numerous image spaces. As a result, the 10.8 GOPS cortical vision processor implemented in 0.13μm CMOS process achieves 60frames/sec performances for 256×256 video inputs at 200MHz operating frequency.