This study investigated how straggler mitigation affects accuracy during distributed training. While distributed training is one promising way to shorten training time, we cannot obtain a maximum performance improvement if some workers become stragglers due to a slowdown. We can avoid the performance impact from stragglers by excluding them from training. However, we face another problem of decreasing accuracy during training because gradients computed by excluded stragglers are not reflected in the training. To achieve both straggler tolerance and accuracy awareness, we should implement distributed training so that excluded stragglers return to the training once they recover from slowdown. Although some implementations supporting auto-scaling have been created, they have not evaluated the training accuracy during scale-in to exclude stragglers from training and scale-out to return them to training. Therefore, we conducted experiments to see how excluding stragglers causes accuracy degradation. We also discussed how the exclusion time and training data assignment can have an impact on accuracy during training.

Distributed deep learning using large mini-batches is a key strategy to perform the deep learning as fast as possible, but it represents a great challenge as it is difficult to achieve high scaling efficiency when using large clusters without compromising accuracy. The particular problem in this challenge is decreasing the number of model update iterations in whole of training. Thus, we need a technique which can converge the validation accuracy with a small number of iterations to address this challenge. In this paper, we introduce a novel technique, Final Polishing. This technique adjusts the means and variances in the batch normalization and mitigates the difference of normalization between validation datasets and augmented training datasets. By applying the technique, we achieved top-1 validation accuracy of 75.08% with mini-batch size of 81,920, with 2,048 GPUs and completed the training of ResNet-50 in 74.7 seconds.In addition, targeting top-1 validation accuracy of 75.9% or more, we tried additional parameters tuning. Then, we adjusted the number of GPUs and hyperparameters of DNNs with Final Polishing, and we also achieved top-1 validation accuracy of 75.97% with mini-batch size of 86,016, with 3,072 GPUs and completed the training of ResNet-50 in 62.1 seconds.

MPI non-blocking collective operations offer a high level interface to MPI library users, and potentially allow communication to be overlapped with calculation. Progression, which controls communications running in the background of the calculation, is the key factor to achieve an efficient overlap. The most commonly used progression method is manual progression, in which a progression function is called in the main calculation. In manual progression, MPI library users have to estimate the communication timing to maximize the overlap effect and thus to manage the complex communication optimization. An alternative approach for progression is the use of separate communication threads. By using communication threads, communication calculation overlap can be achieved simply. However, context switches between the calculation thread and the communication thread cause lower performance in the frequent case where all cores are used for calculation. In this paper, we propose a novel threaded progression method using Hyper-Threading to maximize the overlap effect of non-blocking collective operations. We apply MONITOR/MWAIT instructions to the communication thread on Hyper-Threading so as not to degrade the calculation thread due to shared core resource conflict. Evaluation on 8-node Infini Band connected IA server clustered systems confirmed that the latency is suppressed to a small level and that our approach has an advantage over manual progression in terms of communication-calculation overlap. Using a real application of CG benchmark, our method achieved 32% reduction in execution time compared to using blocking collective operation, and that is nearly perfect overlap. Although manual progression also achieved perfect overlap, our method has the advantage that no communication timing tuning is required for each application.

To enable overlap of computation and communication with non-blocking collective communication, it is required to progress asynchronously a sequence of communications. One of the naive implementation is to use a separate thread for communitation and run it in back of computation thread. However if the total number of threads is greater than the number of physical cores, context switches cause performance degradation of the computation thread. Simultaneous MultiThread (SMT) can be used to avoid this problem. However, commonly-used busy polling for incoming message detection also causes performance degradation of the computation thread. In this paper, we propose incoming message detection method using MONITOR/MWAIT instructions to reduce the performance degradation. Experiment results show that the performance of computation thread is improved largely compared to the busy polling method while latency is suppressed by a small increase.

A cerebellar model arithmetic computer (CMAC) is a neural network whose advantage is fast learning. It is, however, difficult to decide on the various parameters of a CMAC in advance. The pseudo-bacterial genetic algorithm (PBGA) is an evolutionary algorithm that is efficient in local searching. This paper proposes a "PBGA/CMAC" system that decides the positions of partitions, which are the main parameters of a CMAC, using the PBGA. The PBGA/CMAC hardware is implemented for acceleration, because PBGA/CMAC needs a large amount of computation time. An efficient learning method using pipelining is also presented. It is found that the accuracy with the proposed learning method is almost the same as that of the conventional CMAC's learning. The PBGA/CMAC hardware is 140 times faster than that of the equivalent PBGA software.