This paper proposes an ultra-low-power CMOS Image Sensor (CIS) chip based on sensing-with-computing (Senputing) architecture to reduce the power bottleneck of vision system. This Senputing chip achieves BNN 1st-layer convolution in analog domain with ultra-low power consumption. It has two working modes, Normal-Sensor (NS) mode and Direct- Photocurrent-Computation (DPC) mode. The prototype measurement results under 65nm CMOS process on MNIST classification task shows that the power of feature map computation is 2.17μW with 120fps frame rates and 98.1% accuracy. The computation efficiency reaches to 11.49TOPs/W, which is 14.8× higher than state-of-art works.

This paper proposes an in-situ self-powered BNN-based intelligent visual perception system that harvests light energy utilizing the indispensable image sensor itself. The harvested energy is allocated to the low-power BNN computation modules layer by layer, adopting a light-weighted duty-cycling-based energy scheduler. A software-hardware co-design method, which exploits the layer-wise error tolerance of BNN as well as the computing-error and energy consumption characteristics of the computation circuit, is proposed to determine the parameters of the energy scheduler, achieving high energy efficiency for self-powered BNN inference. Simulation results show that with the proposed inference-adaptive energy scheduling method, self-powered MNIST classification task can be performed at a frame rate of 4 fps if the harvesting power is 1μW, while guaranteeing at least 90% inference accuracy using binary LeNet-5 network.

Always-on intelligent visual perception applications are widely deployed in edges in the AIoT era. In order to eliminate power costs of data conversion and transmission, this paper proposes Senputing, an ultra-low-power processing-in-sensor chip that completely fuses sensing and computing together for a BNN-based hierarchical processing system. This chip could operate in two modes. In computation mode, photocurrents are directly utilized for computing without being converted into voltages, and the computation results of 1-st BNN layer are directly sent out to subsequent BNN processors for an always-on coarse classification, eliminating conversion power and storage cost of raw images. Once an interested objected is detected, this chip switches to sensor mode and sends raw images to potential full-precision processors or cloud servers for fine-grained recognition or segmentation. A$32\times 32$prototype is fabricated with 180nm CMOS process. It accomplishes MNIST dataset classification task with the accuracy of 93.76% and the power consumption of 147nW at 156fps, achieving$13.1\times $energy efficiency compared with state-of-the-art work.

In the intelligent IoT edge devices, power consumption is increasing due to the deployment of high-precision algorithms, which greatly limits the working time of the devices. The power of A/D conversion and data transmission has become the bottleneck of traditional visual system. In this paper, a new sensing-with-computing (Senputing) architecture is proposed to reduce this power bottleneck by combining imaging and BNN 1st-layer feature map computation. This Senputing architecture has two working modes, Normal-Sensor mode and Direct-Photocurrent-Computation mode with different resolutions (128×128 and 32×32). An ultra-low-power CMOS image sensor (CIS) chip with Senputing architecture is proposed to verify the feasibility. Our CIS chip is simulated with 180nm CMOS technology, the power of feature map computation is 5.9μW, and the frame rate is 208fps. The computation efficiency reaches to 8.23TOPs/W, which is 10.1 x higher than previous works.

Convolutional neural network (CNN) has been widely deployed in various processors for intelligent visual signal processing. However, the large amount of activations and weights in CNN causes huge power consumption on SRAM access. Data-adaptive SRAM design is a widely studied method to reduce SRAM reading power based on the utilization of data patterns, while current designs only exploit data patterns in a coarse granularity, and have no advantages when faced with randomly distributed weight data. In this article, we propose a hardware–software co-design scheme to reduce SRAM reading power for CNN processing. First, we propose a reconfigurable data-adaptive SRAM architecture with column data segmentation (CDS-RSRAM) to utilize data patterns. Data in one column is partitioned into several segments, and finer-grained data patterns are exploited within each segment for further reading power reduction. Then, a novel training method—minimum segmented neighbor difference (miniSND)—is proposed for enhancing the correlation of weights. MiniSND improves the similarity of weights without classification accuracy degradation, thus weights could benefit from CDS-RSRAM and be read out with less power consumption. Simulation results demonstrate that the co-design scheme saves up to 66%(8b)/89%(2b) power consumption compared with 8T SRAM.

Deploying intelligent visual algorithms in terminal devices for always-on sensing is an attractive trend in the IoT era. In-sensor-processing architecture is proposed to reduce power consumption on A/D conversion and data transmission, which performs pre-processing and only converting low-throughput features. However, current designs still require high energy consumption on photoelectric conversion and analog data movement. In this paper, two methods are proposed to improve the energy efficiency of in-sensor-processing architecture, including direct photocurrent computation and 2D kernel scheduling. Photocurrents are directly involved in computation to avoid data conversion; thus the indispensable imaging power is also utilized for computing. Since the location of the pixel data is fixed, data scheduling is conducted on digital weights to eliminate analog data storage and movement. We implement a prototype chip with an array of 32 × 32 units to calculate the first layer of binarized LeNet-5. The post-simulation shows that the proposed architecture reaches the energy efficiency of 11.49TOPs/W, about 14.8x higher than previous works.