Today's CPUs are general-purpose processors, which have the von Neumann architecture (including the Harvard architectures) to maximize the generality and programmability. On the other hand, application-specific integrated circuits (ASICs) have domain-specific architectures to optimize the cost-effective performance but show very low generality. The combination of generality and ASIC, which usually seemed to have no contact, is expected to be enabled by deep learning (DL). DL, realized with deep neural networks (DNNs), has changed the paradigm of machine learning (ML) and brought significant progress in vision, speech, language processing, and many other applications. DNNs have special features that can be efficiently implemented with dedicated architectures, ASICs. Sharing their special features, DNNs have a wide variety of network architectures, and even the same network architecture can be used for different applications depending on the weight parameters. This paper aims to provide the necessity, validity, and characteristics of the ML-specific integrated circuits (MSICs) that have a different architecture from the von Neumann architecture. MSICs can avoid the overhead from the complex instruction set, instruction decoder, multilevel caches, and branch prediction of the recent von Neumann architecture processors designed for high generality and programmability. We will also discuss the necessity and validity of a heterogeneous architecture in MSIC, starting from the differences between the visual-type information processing and the vector-type information processing, and show the chip implementation results.

A real-time global matching optical flow estimation (OFE) processor is proposed for action recognition in mobile devices. The global OFE requires a large number of external memory accesses (EMAs) and matrix computations, thus it is incompatible on mobile devices with real-time constraints. For real-time OFE on mobile devices, this paper proposes two key features, both of which to reduce the required memory bandwidth and a number of computations: 1) Tile-based hierarchical OFE enables intermediate data to be processed within 328 KB on-chip memory without external memory access. 2) Background skipping eliminates redundant matrix computation for zero optical flow region. Therefore, the proposed features reduce external memory bandwidth and computation by 99.7 % and 50.7 %, respectively. The proposed 4 mm2OFE processor is implemented in 65 nm CMOS technology and it achieves real-time OFE of 46.1 frames-per-second (fps) throughput for an image resolution of QVGA (320 × 240) and the resulting optical flow can be successfully used for action recognition.

An energy-efficient analog SRAM (A-SRAM) is proposed to eliminate redundant analog-to-digital (A/D) and digital-to-analog (D/A) conversion in mixed-signal systems, such as neuromorphic chips and neural networks. D/A conversion is integrated into the SRAM readout by charge sharing of the proposed split bitline (BL). Also, A/D conversion is integrated into the SRAM write operation with the successive approximation method in the proposed input-output block. Also, a configurable SRAM bitcell array is proposed to allocate the converted digital data without unfilled bitcells. The multirow access decoder selects multiple bitcells in a single column and configures the bitcell array by controlling the BL switches to split BLs. The proposed A-SRAM is implemented using the 65-nm CMOS technology. It achieves 17.5-fJ/bit energy-efficiency and 21-Gbit/s throughput for the analog readout, which are 64% and 1.3 times better than those of the conventional SRAM followed by a digital-to-analog converter (DAC). Also, the area is reduced by 91% compared with the conventional SRAM with analog-to-digital converter (ADC) and DAC.

An energy-efficient analog SRAM (A-SRAM) is proposed to eliminate redundant analog-to-digital (A/D) and digital-to-analog (D/A) conversions in the mixed-signal processing such as a biomedical and a neural network applications. The D/A and the A/D conversion are integrated into the SRAM readout by the charge sharing of the proposed split bit-line (BL) and the SRAM write by the successive approximation method, respectively. And a data structure is newly proposed to allocate each bit of the input data to the binary-weighted bit-cell array. The proposed A-SRAM is implemented using 65 nm CMOS technology. As a result, it achieves 17.5 fJ/bit read energy-efficiency and 21 Gbit/s read throughput, which are 54% lower and 1.3× higher than the conventional SRAM. Also, the area is reduced by 31% compared to the conventional SRAM with ADC and DAC.

A Feature Reuse Engine (FReE) is proposed to achieve low-power object recognition in video frames. Previous object recognition processors perform redundant processing for repeated features of sequential frames in video. Unlike previous works, proposed FReE reuses 58% of features from previous frame with inter-frame similarity. However, false reuse can degrade the object recognition accuracy. By pixel intensity-based Near Pixel Information (NPI), FReE can decide the reusability with 96% of accuracy. Additionally, to minimize latency and power overhead of FReE, a dedicated integral image generator and NPI generator are proposed. As a result, power consumption of object recognition processor is reduced by 31% with the proposed FReE which consumes only 1.92mW in a 130nm CMOS technology.