Deep Neural Networks (DNN) are computationally intensive to train. It consists of a large number of multidimensional dot products between many weights and input vectors. However, there can be significant similarities among input vectors. If one input vector is similar to another, its computations with the weights are similar to those of the other and, therefore, can be skipped by reusing the already-computed results. We propose a novel scheme, called MERCURY, to exploit input similarity during DNN training in a hardware accelerator. MERCURY uses Random Projection with Quantization (RPQ) to convert an input vector to a bit sequence, called Signature. A cache (MCACHE) stores signatures of recent input vectors along with the computed results. If the Signature of a new input vector matches that of an already existing vector in the MCACHE, the two vectors are found to have similarities. Therefore, the already-computed result is reused for the new vector. To the best of our knowledge, MERCURY is the first work that exploits input similarity using RPQ for accelerating DNN training in hardware. The paper presents a detailed design, workflow, and implementation of the MERCURY. Our experimental evaluation with twelve different deep learning models shows that MERCURY saves a significant number of computations and speeds up the model training by an average of 1.97× with an accuracy similar to the baseline system.

Neural network training is inherently sequential where the layers finish the forward propagation in succession, followed by the calculation and back-propagation of gradients (based on a loss function) starting from the last layer. The sequential computations significantly slow down neural network training, especially the deeper ones. Prediction has been successfully used in many areas of computer architecture to speed up sequential processing. Therefore, we propose ADA-GP, which uses gradient prediction adaptively to speed up deep neural network (DNN) training while maintaining accuracy. ADA-GP works by incorporating a small neural network to predict gradients for different layers of a DNN model. ADA-GP uses a novel tensor reorganization method to make it feasible to predict a large number of gradients. ADA-GP alternates between DNN training using backpropagated gradients and DNN training using predicted gradients. ADA-GP adaptively adjusts when and for how long gradient prediction is used to strike a balance between accuracy and performance. Last but not least, we provide a detailed hardware extension in a typical DNN accelerator to realize the speed up potential from gradient prediction. Our extensive experiments with fifteen DNN models show that ADA-GP can achieve an average speed up of 1.47 × with similar or even higher accuracy than the baseline models. Moreover, it consumes, on average, 34% less energy due to reduced off-chip memory accesses compared to the baseline accelerator.

Approximate computing has significant potential to improve the efficiency of a computing system. Numerous techniques have been proposed in literature. Virtually, all of them require programmers to either experiment with every instance of a specific type of code region exhaustively to find approximable code regions or annotate such regions manually. Both approaches are error-prone and can lead to missed opportunities. Therefore, we propose XMeter to automatically find and quantify approximable code regions. XMeter, first, analyzes the application code statically using a novel algorithm based on memory location updates. Also, XMeter provides a deep learning-based predictor to predict the accuracy of the application when different code regions are approximated. Our proposed scheme does not require the programmer to experiment exhaustively for all possible error rates and types of approximation techniques. Moreover, the scheme does not require any domain knowledge and is not specific to any approximation technique. Therefore, it is general enough to be applicable for any approximation technique. We developed XMeter using LLVM and experimented with 10 applications. We analyzed 43 approximable functions and found 21 to be highly tolerant of errors. We validated our results using 4 well-known approximation techniques and showed that XMeter can predict an application's accuracy accurately.