Convolutional neural network (CNN) has demonstrated great success in pattern recognition scenarios at the cost of nearly billions of parameters and consequent convolution operations. Various dedicated hardware designs are proposed to accelerate the CNN computation in more energy-efficient manners. Especially, the bit-serial accelerator (BSA) is one of the most effective approaches on resource-limited platforms by eliminating zero-bit computations. However, the irregular distribution and varying number of effectual (nonzero) bits in weights significantly cause hardware underutilization, impeding further performance improvement of state-of-the-art BSAs. To address this issue, BitCluster, a hardware-friendly quantization method, is proposed to make each weight with the identical number of effectual bits for load-balanced computation. Considering distinct sensitivities to weight precision in different neural layers, layer-level BitCluster is proposed to design further for fine-grained weight quantization. It systematically determines the layerwise quantization configurations, which significantly improve the overall performance with$1.6\times $higher hardware utilization and$3.4\times $speedup on average than state-of-the-art BSAs, with$5\times $better energy efficiency on average.

The mnemonic descent method (MDM) algorithm is the first end-to-end recurrent convolutional system for high-accuracy face alignment. However, the heavy computational complexity and high memory access demands make it difficult to satisfy the requirements of real-time applications. To address this problem, an improved MDM (I-MDM) algorithm is proposed for efficient hardware implementation based on several hardware-oriented optimizations. First, a patch merging mechanism is introduced to dynamically cluster and eliminate redundant landmarks, which significantly reduces computational complexity with minimal accuracy loss. Second, a dedicated convolutional layer is inserted to halve the number of computations and memory access of the subsequent fully connected layer, yielding a 4.42% decrease in the failure rate. Third, a lightweight preprocessing method named dual regressors is proposed to reinitialize face images, which can greatly improve the overall accuracy. Moreover, compared with a similar method, the DR method can reduce computations and memory storage by nearly 99.9%. Overall and compared with the MDM algorithm, I-MDM not only reduces the number of computations by 23.5% but also decreases the failure rate by 17.9% on the 300 W test set. Based on the proposed I-MDM algorithm, an I-MDM-based hardwired accelerator is presented using the TSMC 65 nm CMOS process. First, compared with similar solutions, the gradient calculation operation is rearranged and loaded pixels are reused in the HoG feature extraction to eliminate all division operations and 25% off-chip memory access. Second, patch-independent central activations are used to enable patch-level pipelined operations, yielding a 2× acceleration in the overall process. This accelerator achieves 460 GOPS/W energy efficiency at 330 MHz, which is 38× higher than the most recent face alignment accelerator with the same process.

Although convolutional neural network (CNN) models have greatly enhanced the development of many fields, the untenable number of parameters and computations in these models yield significant performance and energy challenges in hardware implementations. Transferred filter-based methods, as very promising techniques that have not yet been explored in the architecture domain, can substantially compress CNN models. However, their straightforward hardware implementation inherently incurs massive redundant computations, causing significant energy and time consumption. In this work, a highly efficient transferred filter-based engine (TFE) is developed to alleviate this deficiency, with CNN models compressed and accelerated. First, the filters of CNN models are flexibly transferred according to specific tasks to reduce the model size. Then, two hardware-friendly mechanisms are proposed in the TFE to remove duplicate computations caused by transferred filters, which can further accelerate transferred CNN models. The first mechanism exploits the shared weights hidden in each row of transferred filters and reuses the corresponding same partial sums, reducing at least 25% of repetitive computations in each row. The second mechanism can intelligently schedule and access the memory system to reuse the repetitive partial sums among different rows of the transferred filters with at least 25% of computations eliminated. Furthermore, an efficient hardware architecture is proposed in the TFE to fully reap the benefits of the two proposed mechanisms such that different types of networks are flexibly supported. To achieve high energy efficiency, the sub-array-based filter mapping method (SAFM) is proposed, where the process element (PE) subarray is used as the elementary computational unit to support various filters. Therein, input data can be efficiently broadcast in each PE sub-array and the load can be stripped from each PE and intensively alleviated, which can dramatically reduce the area and power consumption. Excluding MobileNet-like networks that adopt depth-wise convolution, most mainstream networks can be compressed and accelerated by the proposed TFE. Two state-of-the-art transferred filter-based methods, i.e., doubly CNN and symmetry CNN are implemented by exploiting the TFE. Compared with Eyeriss, average speedup improvements of 2.93× and 3.17× are achieved in the convolutional layers of various modern CNNs. The overall energy efficiency can be improved by 12.66× and 13.31× on average. Compared with other state-of-the-art related works, the TFE can maximally achieve a parameter reduction of 4.0×, a speedup of 2.72× and an energy efficiency improvement of 10.74× on VGGNet.

Face detection and alignment are two fundamental tasks for facial applications and the corresponding accelerators have been designed to enable energy-efficient acceleration. However, these dedicated accelerators are always designed separately, thereby ignoring the inherent correlation between face detection and alignment and causing additional communication and area overhead. Based on this motivation, a multi-task cascaded convolutional networks (MTCNN) algorithm-based accelerator is presented in this work to support both face detection and alignment for multiple faces. First, multiply-accumulate (MAC) operations and memory access of the magnification process in the resize module are reduced by 22.8% and 24.8% on average, respectively, when compared with those of similar methods. Second, clustering non-maximum suppression (C-NMS) is proposed to significantly reduce the intersection over union computation and eliminate the hardware-inference sorting process in NMS, yielding a 16.0% speedup in the overall process. Third, an efficient pipeline architecture is proposed to implement a complexity- and memory-intensive proposal network of MTCNN in a more computationally efficient manner, with 38.3% less memory capacity than a similar solution. Meanwhile, only approximately half of the multipliers are needed to achieve the same throughput with high pipeline utilization. Fourth, considering the variable number of faces in each input, a batch schedule mechanism is proposed to improve the fully-connected layer hardware utilization by 16.7% on average in the batch process. Based on a simulation with the TSMC 28 nm CMOS process, this accelerator consumes only 10.9ms at 400 MHz to simultaneously process 5 faces. The power efficiency reaches 4.80 TOPS/W, which is$227.4\times $higher than that of the state-of-the-art solution.

A binary-weight hourglass network (B-HG) accelerator for landmark detection, built on the proposed look-up-table (LUT) based multi-level prediction-correction approach, is enabled for high-speed and energy-efficient processing on IoT edge devices. First, LUT with a unified mode is adopted to support convolutional neural network with fully variable weight bit precision to minimize operations of B-HG, which achieves 1.33×-1.50× speedup on multi-bit weight CNN relative to the similar solution. Second, multi-level prediction-correction model is proposed to achieve computational-efficient convolution with adaptive precision. The operations saved can be increase by about 30% than the two-stage model. Besides, nearly 77.4% of the operations in B-HG can be saved by using the combination of these two methods, yielding a 2.3× inference speedup. Third, block computing based pipeline is designed to improve the residual block deficiency in B-HG. It can not only reduce about 66.2% off-chip memory access than the baseline, but also save 60% and 31% on-chip memory space and access compared to the similar fused-layer accelerator. The proposed B-HG accelerator achieves 450 fps at 500MHz based on the simulation in TSMC 28 nm process. Meanwhile, the power efficiency is up to 8.5 TOPS/W, which is two orders of magnitude higher than the dedicated face landmark detection accelerator.

Face detection and alignment are highly-correlated, computation-intensive tasks, without being flexibly supported by any facial-oriented accelerator yet. This work proposes the first unified accelerator for multi-face detection and alignment, along with the optimizations on multi-task cascaded convolutional networks algorithm, to implement both multi-face detection and alignment. First, the clustering non-maximum suppression is proposed to significantly reduce intersection over union computation and eliminate the hardware-interfer-ence sorting process, bringing 16.0% speed-up without any loss. Second, a new pipeline architecture is presented to implement the proposal network in more computation-efficient manner, with 41.7% less multiplier usage and 38.3% decrease in memory capacity compared with the similar method. Third, a batch schedule mechanism is proposed to improve hardware utilization of fully-connected layer by 16.7% on average with variable input number in batch process. Based on the TSMC 28 nm CMOS process, this accelerator only consumes 6.7ms at 400 MHz to simultaneously process 5 faces for each image and achieves 1.17 TOPS/W power efficiency, which is 54.8× higher than the state-of-the-art solution.

The coarse-to-fine shape searching (CFSS) framework is a recently developed algorithm that achieves relatively high accuracy in face alignment by alleviating the poor initialization problem facing traditional cascaded regression approaches. However, its high computational complexity and memory access demands make it difficult for CFSS to satisfy the requirements of real-time processing. To address this issue, a fast shape searching face alignment (F-SSFA) accelerator is presented based on the optimization of the CFSS algorithm and an efficient hardware implementation. First, the learning-based low-dimensional speeded-up robust features method, based on the correlations between the SURF features and the regression targets, is introduced to distill the feature set down to the only most distinct features to reduce the computing load. Second, the partial keypoints Euclidean distance and shape affine transformation are introduced to replace feature extraction and support vector machine classification, thereby accelerating the shape searching process. Compared with CFSS, F-SSFA achieves a $5.8\times $ speedup while achieving similar accuracy. Moreover, a VLSI architecture is proposed to realize the fixed-point F-SSFA algorithm. Multiple descriptors located in adjacent regions are simultaneously generated in a single access to the corresponding image data. Therefore, repeated memory access operations are avoided. The optimal parameter configuration for hardware implementation is also exploited based on a tradeoff between accuracy and hardware performance. Simulated with TSMC 65-nm 1P8M technology within a 3.6 mm2area, a post-layout simulation shows that 700 fps can be achieved while consuming 300 mW at 200 MHz.

Face alignment is a critical task in many multimedia and vision applications that use face-based algorithms. Recent research has focused on achieving efficient initialization to improve performance; however, the use of facial attributes and the extent of their correlation with initialization have not been fully exploited. This paper presents a lightweight method called expression- and pose-based adaptive initialization (EXPAI), in which facial attributes, that is, expression and pose information, are used as priors. This approach can significantly improve the face alignment performance. In addition, reliable expression and head pose information can be derived simultaneously in the same framework. First, an expression- and pose-based template dictionary is formed by augmenting the mean shape across three degrees of freedom, thereby substantially improving the robustness of the initial shape with respect to large head pose variations. Second, each the template corresponds to an image of interest, which is jointly determined using a shape-constrained multiclass classifier and binary classifiers, and is assigned a pretrained confidence coefficient. The initial shape that is thus generated for subsequent cascaded regression is more adaptive and enables higher accuracy. Furthermore, EXPAI enables initialization with significantly increased computational efficiency because of its independence from the original dataset. The experimental results obtained on the widely used 300-W dataset show that our method achieves very competitive performance compared with that of state-of-the-art methods. In particular, for the challenging subset of 300-W, EXPAI reduces errors by more than 14% compared with coarse-to-fine shape searching (CFSS), which currently exhibits the best performance among regression-based approaches. Furthermore, a speed increase of more than 10 times compared with CFSS is achieved.

In this work, a fast shape searching face alignment (F-SSFA) algorithm based accelerator is proposed to achieve real-time processing. Firstly, a learning based low-dimensional SURF feature is introduced to reduce the computation cost in the cascaded regression. Then the Euclidean distance and shape affine transformation are utilized to accelerate the shape searching procedure. F-SSFA therefore greatly reduces the computational complexity while keeping the same accuracy. Also, a fixed-point F-SSFA based VLSI architecture is designed with approximately 80% decrease in the data transmission traffic. The throughput of this accelerator achieves 700 fps, which is especially suitable for high-speed facial-related applications.