To ensure resilient neural network processing on even unreliable hardware, comprehensive reliability analysis against various hardware faults is generally required before the neural network models are deployed, and efficient fault injection tools are highly demanded. However, many existing fault injection tools remain limited to basic fault injection and fail to provide fine-grained vulnerability analysis capability. In addition, many of the fault injection tools also need to change the neural network models and make the fault injection closely coupled with normal neural network processing, which complicates the use of these tools and slows down the fault simulation. The various fault injection implementations and error metrics make the comparison between different fault-tolerant studies difficult. To this end, we propose MRFI, a highly configurable multiresolution fault injection tool for deep neural networks. It enables users to modify an independent fault configuration file rather than neural network models for fault injection and vulnerability analysis. Particularly, it integrates extensive fault analysis functionalities from different perspectives and enables multiresolution investigation of the vulnerability of neural networks. In addition, it does not modify the major neural network computing framework of PyTorch. Hence, it allows parallel processing on GPUs naturally and exhibits fast fault simulation according to our experiments. Moreover, we also have the fault injection calibrated with fault simulation with architectural details and validate the accuracy of the proposed fault injection. Finally, MRFI is also open-sourced on GitHub (MRFI https://github.com/fffasttime/MRFI).

Soft errors in large VLSI circuits have a significant impact on computing- and memory-intensive neural network (NN) processing. Understanding the influence of soft errors on NNs is critical to protect against soft errors for reliable NN processing. Prior work mainly relies on fault simulation to analyze the influence of soft errors on NN processing. They are accurate but usually specific to limited configurations of errors and NN models due to the prohibitively slow simulation speed especially for large NN models and datasets. With the observation that the influence of soft errors propagates across a large number of neurons and accumulates as well, we propose to characterize the soft error-induced data disturbance on each neuron with a normal distribution model using the central limit theorem and develop a series of statistical models to analyze the behavior of NN models under soft errors in general. The statistical models reveal not only the correlation between soft errors and the accuracy of NN models but also how NN parameters, such as quantization and architecture affect the reliability of NNs. The proposed models are compared with fault simulations and verified comprehensively. In addition, we observe that the statistical models that characterize the soft error influence can also be utilized to predict fault simulation results in many cases and we explore the use of the proposed statistical models to accelerate fault simulations of NNs. Our experiments show that the proposed accelerated fault simulation provides almost two orders of magnitude speedup with negligible loss of simulation accuracy compared to the baseline fault simulations.

Winograd is generally utilized to optimize convolution performance and computational efficiency because of the reduced multiplication operations, but the reliability issues brought by winograd are usually overlooked. In this work, we observe the great potential of winograd convolution (WG-Conv) in improving neural network (NN) fault tolerance. Based on the observation, we evaluate WG-Conv fault tolerance comprehensively from different granularities ranging from models, layers, and operation types for the first time. Then, we explore the use of inherent fault tolerance of WG-Conv for cost-effective NN protection against soft errors. Specifically, we mainly investigate how WG-Conv can be effectively incorporated with classical fault-tolerant design approaches including triple modular redundancy (TMR), fault-aware retraining, and constrained activation functions. According to our experiments, WG-Conv can reduce the fault-tolerant design overhead by 55.77% on average without any accuracy loss compared to standard convolution (ST-Conv), and further reduce the computing overhead by 17.24% when the inherent fault tolerance of WG-Conv is considered. When it is applied on fault-tolerant NNs enhanced with fault-aware retraining and constrained activation functions, the resulting model accuracy generally shows significant improvement in the presence of various faults.

Vision transformers (ViTs) that leverage self-attention mechanism have shown superior performance on many classical vision tasks compared to convolutional neural networks (CNNs) and gain increasing popularity recently. Existing ViTs’ works mainly optimize performance and accuracy, but ViTs’ reliability issues induced by soft errors in large-scale VLSI designs have generally been overlooked. In this work, we mainly study the reliability of ViTs and investigate the vulnerability from different architecture granularities ranging from models, layers, modules, and patches for the first time. The investigation reveals that ViTs with the self-attention mechanism are generally more resilient on linear computing including general matrix–matrix multiplication (GEMM) and full connection (FC) and show a relatively even vulnerability distribution across the patches. ViTs involve more fragile non-linear computing such as softmax and GELU compared to typical CNNs. With the above observations, we propose a lightweight block-wise algorithm-based fault-tolerance (LB-ABFT) approach to protect the linear computing implemented with distinct sizes of GEMM and apply a range-based protection scheme to mitigate soft errors in non-linear computing. According to our experiments, the proposed fault-tolerant approaches enhance ViTs’ accuracy significantly with minor computing overhead in the presence of various soft errors.

Winograd convolution is originally proposed to reduce the computing overhead by converting multiplication in neural network (NN) with addition via linear transformation. Other than the computing efficiency, we observe its great potential in improving NN fault tolerance and evaluate its fault tolerance comprehensively for the first time. Then, we explore the use of fault tolerance of winograd convolution for either fault-tolerant or energy-efficient NN processing. According to our experiments, winograd convolution can be utilized to reduce fault-tolerant design overhead by 27.49% or energy consumption by 7.19% without any accuracy loss compared to that without being aware of the fault tolerance.