Driving at night is more challenging and dangerous than driving during the day. Modeling driver eye movement and attention allocation during night driving can help guide unmanned intelligent vehicles and improve safety during similar situations. However, until now, few studies have modeled a drivers’ true fixations and attention allocation in specific night circumstance. Therefore, we collected an eye tracking dataset from 30 experienced drivers while they viewed night driving videos under a hypothetical driving condition, termed Driver Fixation Dataset in night (DrFixD(night)). Based on DrFixD(night) which includes multiple drivers’ attention allocation, we proposed a spatio-temporal dual-encoder network model, named as STDE-Net, to improve saliency detection in night driving condition. The model includes three modules: i) spatio-temporal dual encoding module, ii) fusion module based on attention mechanism, and iii) decoding module. A convolutional LSTM is employed to learn the time connection of video sequences, and a convolution neural network combined pyramid dilated convolution is adopted to extract spatial features in the spatio-temporal dual encoding module. The attention mechanism is exploited to fuse the temporal and spatial features together and selectively highlight the significant features in night traffic scene. We compared the proposed model with other traditional methods and deep learning models, both qualitatively and quantitatively, and found that the proposed model can predict driver’s fixation more accurately. Specifically, the proposed model not only predicts the main goals, but also predicts the important sub goals, such as pedestrians, bicycles and so on, showing excellent prediction of dimly lit targets at night.

Convolutional neural networks (CNNs) have been widely deployed in computer vision tasks. However, the computation and resource intensive characteristics of CNN bring obstacles to its application on embedded systems. This article proposes an efficient inference accelerator on Field Programmable Gate Array (FPGA) for CNNs with depthwise separable convolutions. To improve the accelerator efficiency, we make four contributions: (1) an efficient convolution engine with multiple strategies for exploiting parallelism and a configurable adder tree are designed to support three types of convolution operations; (2) a dedicated architecture combined with input buffers is designed for the bottleneck network structure to reduce data transmission time; (3) a hardware padding scheme to eliminate invalid padding operations is proposed; and (4) a hardware-assisted pruning method is developed to support online tradeoff between model accuracy and power consumption. Experimental results show that for MobileNetV2 the accelerator achieves 10× and 6× energy efficiency improvement over the CPU and GPU implementation, and 302.3 frames per second and 181.8 GOPS performance that is the best among several existing single-engine accelerators on FPGAs. The proposed hardware-assisted pruning method can effectively reduce 59.7% power consumption at the accuracy loss within 5%.

The post-training compression with quantization is a common technology to improve the efficiency of embedded neural network accelerators. In this paper, a Dynamic Quantization in Inference (DQI) method is proposed to solve the severe quantization overflow problem that may occur in CNN inference process. Based on analysis of quantization errors of activation values in convolutional layers, efficient quantization overflow detection and quantization parameters dynamic update are designed and implemented in CNN accelerator. The evaluation result on VGG16 and MobileNetV2 models demonstrates that DQI can improve the inference accuracy of by up to 11.59% in high overflow scenarios, while the overhead in hardware resources and runtime is acceptable.

Convolutional neural networks (CNNs) have been widely deployed in computer vision tasks. However, the computation and resource intensive characteristics of CNN bring obstacles to its application on embedded systems. MobileNet, as a representative of compact models, can reduce the amount of parameters and computation. A high-performance inference accelerator on FPGA for MobileNet is proposed in this paper. With respect to the three types of convolution operations, multiple parallel strategies are exploited and the corresponding hardware structures such as input buffer and configurable adder tree are designed. With respect to the bottleneck block, a dedicated architecture is proposed to reduce data transmission time. In addition, a hardware padding scheme to improve the efficiency of padding is proposed. The accelerator implemented on Virtex-7 FPGA reaches 70.8% Top-1 accuracy under 8-bit quantization. The accelerator achieves 302.3 FPS and 181.8 GOPS, which obtains 22.7x, 3.9x and 1.4x speedup compared to the implementations in Snapdragon 821 CPU, i7-6700HQ CPU and GTX 960M GPU, respectively.