Sparsity, a widely recognized path to curbing the computational needs of deep neural networks (DNNs), still suffers a number of roadblocks in practice, despite a decade of intensive research on sparse neural networks. The extant structured sparsity patterns often fail to attain significant model compression, while the hardware challenges posed by unstructured sparsity are yet to be fully overcome. As algorithmic and hardware innovations individually deliver limited benefits, a synergistic approach is necessary to unleash the potential of sparse DNNs. This work proposes a tightly integrated design methodology for the sparsity patterns and associated hardware platforms to reach the highest model compression goals while simultaneously facilitating efficient hardware processing. We demonstrate that novel complementary sparsity patterns can offer utmost expressiveness levels with inherent hardware exploitable regularity. Our novel dynamic training method converts the expressiveness of such sparsity configurations into highly accurate and compact sparse neural networks. Complementary sparsity is represented in a dense format, and when synergistically coupled with minimal yet strategic hardware modifications, can be processed in close concordance with the conventional dataflow of the dense matrix operations. We thus demonstrate that there is ample room for innovation beyond conventional techniques and immense practical potential for sparse neural networks through the synergistic design of sparsity patterns and hardware architectures.

The efficiency of machine intelligence techniques has improved noticeably in the embedded application domains thanks to the dedicated hardware accelerators for deep neural networks (DNNs). Despite the economic criticality of yield and reliability problems in advanced semiconductor nodes, these concerns have attracted limited attention in the context of embedded machine intelligence devices. The micro-architectural features of deep learning accelerators, when paired with the algorithmic characteristics of DNNs, unlock novel opportunities to tackle semiconductor reliability problems in embedded deep learning devices. While the fine-grained bypassing of the faulty processing elements reins the computational impact of hardware defects, a one-time training of DNNs with Hardware-Aware Dropout/Dropconnect techniques boosts model decentralization and facilitates accurate neural network inference in the degraded computational fabrics. Furthermore, on-device calibration methods can improve resilience even further without necessitating expensive defect compensation methods such as device-specific training. Our work confirms the potential for improving the yield, reliability, and operational lifetime of embedded machine intelligence devices through a highly practical co-design of DNNs and configurable hardware architectures.

Sparse deep learning models are known to be more accurate than their dense counterparts for equal parameter and computational budgets. Unstructured model pruning can deliver dramatic compression rates, yet the consequent irregular sparsity patterns lead to severe computational challenges for modern computational hardware. Our work introduces a set of complementary sparsity patterns to construct both highly expressive and inherently regular sparse neural network layers. We propose a novel training approach to evolve inherently regular sparsity configurations and transform the expressive power of the proposed layers into a competitive classification accuracy even under extreme sparsity constraints. The structure of the introduced sparsity patterns engenders optimal compression of the layer parameters into a dense representation. Moreover, the constructed layers can be processed in the compressed format with full-hardware utilization in minimally modified non-sparse computational hardware. The experimental results demonstrate superior compression rates and remarkable performance improvements in sparse neural network inference in systolic arrays.

As deep learning algorithms are widely adopted, an increasing number of them are positioned in embedded application domains with strict reliability constraints. The expenditure of significant resources to satisfy performance requirements in deep neural network accelerators has thinned out the margins for delivering safety in embedded deep learning applications, thus precluding the adoption of conventional fault tolerance methods. The potential of exploiting the inherent resilience characteristics of deep neural networks remains though unexplored, offering a promising low-cost path towards safety in embedded deep learning applications. This work demonstrates the possibility of such exploitation by juxtaposing the reduction of the vulnerability surface through the proper design of the quantization schemes with shaping the parameter distributions at each layer through the guidance offered by appropriate training methods, thus delivering deep neural networks of high resilience merely through algorithmic modifications. Unequaled error resilience characteristics can be thus injected into safety-critical deep learning applications to tolerate bit error rates of up to at absolutely zero hardware, energy, and performance costs while improving the error-free model accuracy even further.

The abundant usage of deep neural networks in safety-critical domains such as autonomous driving raises concerns regarding the impact of hardware-level faults on deep neural network computations. As a failure can prove to be disastrous, low-cost safety mechanisms are needed to check the integrity of the deep neural network computations. We embed safety checksums into deep neural networks by introducing a custom regularization term in the network training. We partition the outputs of each network layer into two groups and guide the network to balance the summation of these groups through an additional penalty term in the cost function. The proposed approach delivers twin benefits. While the embedded checksums deliver low-cost detection of computation errors upon violations of the trained equilibrium during network inference, the regularization term enables the network to generalize better during training by preventing overfitting, thus leading to significantly higher network accuracy.

DNNs are abundantly employed in a variety of applications, including real-time systems with strict safety constraints. The consequences of errors prove disastrous in safety-critical systems, such as autonomous driving, healthcare, and industrial applications. DNNs are resilient to limited numerical perturbations yet fragile under large deviations in weights and activations. The traditional error tolerance measures fail to meet the tight design constraints of DNN processing systems due to extensive overheads or limited advantages in abundant error conditions. The algorithmic particularities of DNNs though create novel opportunities to deal with errors more effectively and economically. We revisit the two fundamental tasks in fault-tolerant system design, namely, error detection and correction, and demonstrate that the precise versions of these operations could be replaced by approximated counterparts in DNNs to deliver an extensive bit-error resilience even at high error rates while necessitating no information redundancy. We first maintain DNN accuracy even under extreme error rates by suppressing the numerical contributions of anomalous activations, eliminating any reliance on precise error correction. We tackle the problem of no redundancy error detection by establishing in training numerical associations among activations, and employing them for anomaly detection. Anomalous feature detection and suppression, performed efficiently at inference with minimal resources in a DNN accelerator, is shown to deliver significant resilience boosts while imposing neither information redundancy nor perceptible overheads.

Deep learning techniques have enjoyed wide adoption in real life, including in various safety-critical embedded applications. While neural network computations require protection against hardware errors, the substantial overheads of conventional error-tolerance techniques limit their use on embedded platforms, which carry out demanding deep neural network computations with limited resources. The utilization of conventional techniques is further constrained in high error rate scenarios, increasingly prevalent under aggressive energy and performance optimizations. To resolve this conundrum, we introduce a novel median feature selection technique to filter the impact of bit errors prior to the execution of each layer. While our technique can be deemed as a fine-grained modular redundancy scheme, its construction purely out of the inherent redundancy of the network necessitates neither additional parameters nor extra multiply-accumulate operations, squashing the inordinate overheads typically associated with such techniques. Median feature selection can be efficiently performed in hardware and seamlessly integrated into embedded deep learning accelerators as a modular plug-in. Deep learning models can be trained with standard tools and techniques to ensure a graceful operational interface with the feature selection stages. The proposed technique allows the system to perform accurately even at high error rates by improving its resilience up to four orders of magnitude, yet incurs negligible 0.19%–0.48% area and 0.07%–0.19% power overheads for the required operations.

The widespread usage of deep neural networks in autonomous driving necessitates a consideration of the safety arguments against hardware-level faults. This study confirms the possible catastrophic impact of hardware-level faults on DNN accuracy; the consequent need for low-cost fault tolerance methods can be met through a rigorous exploration of the mathematical properties of the associated computations. We propose Sanity-Check, which makes use of the linearity property and employs spatial and temporal checksums to protect fully-connected and convolutional layers in deep neural networks. Sanity-Check can be purely implemented on software and deployed on different execution platforms with no additional modification. We also propose Sanity-Check hardware which integrates seamlessly with modern DNN accelerators and neutralizes the small performance overhead in pure software implementations. Sanity-Check delivers perfect error-caused misprediction coverage in our experiments, which makes it a promising candidate for boosting the reliability of safety-critical deep neural network applications.