Synchronous stochastic gradient descent (S-SGD) with data parallelism has become a de-facto approach in training large-scale deep neural networks (DNNs) on multi-GPU systems. However, S-SGD requires iteratively synchronizing gradients from all workers, which incurs excessive communication costs and limits the scaling efficiency of GPU clusters. Gradient sparsification like top-k sparsification techniques has been shown to be potentially effective in reducing the communication volume. Yet, existing approaches with top-k sparsification still suffer from high communication complexity in that they often need a very low density to achieve better performance while easily sacrificing the model accuracy. To this end, we propose a novel sparse communication approach called TopKA2A, which integrates top-k sparsification with AlltoAll communication to exchange sparse tensors among GPUs, resulting in a notable reduction in communication complexity. With rigorous theoretical analysis on the conditions that TopKA2A can be applied, we design a simple yet effective tensor fusion algorithm based on binary search. We perform in-depth analysis and evaluation of communication efficiency by comparing TopKA2A with state-of-the-art solutions over popular models without compromising model accuracy. Experimental results demonstrate that TopKA2A yields substantial communication efficiency gains and runs up to 73% faster than existing algorithms on a 32-GPU cluster.

Most face-inpainting methods perform well in face repair. However, these methods can only complete a single face image per input. Although existing various image-inpainting methods can achieve pluralistic image inpainting, they typically produce faces with distorted structures or the same texture. To resolve these shortcomings and achieve high-quality diverse face inpainting, we propose PFTANet, a two-stage pluralistic face-inpainting network that transforms attribute information. In the first stage, the face-parsing network is fine-tuned to obtain semantic facial region information. In the second stage, a generator consisting of SNBlock, CF_ShiftBlocks, and CF_MergeBlock, which ensures that high-quality pluralistic face results are generated, is used. Specifically, CF_ShiftBlocks completes pluralistic face generation by transforming the attribute information from the conditional face extracted by the attribute extractor and ensuring the consistency of the attribute information between the conditional and generated faces. CF_MergeBlock ensures structural consistency between the masked and background regions of the generated face using facial region semantic information. A multi-patch discriminator is used to enhance facial detail generation. Experimental results for the CelebA and CelebA-HQ datasets indicated that PFTANet achieved pluralistic and visually realistic face inpainting.

For patients with dilated cardiomyopathy (DCM), fast and accurate diagnosis is important to save lives. MRI is a non-invasive, effective medical imaging method that allows doctors to diagnose DCM. However, manual and semi-automatic segmentation is subjective, non-reproducible and time-consuming task. In this paper, a new attention-based convolutional encoder-decoder network is proposed to automatically segment my-ocardium in DCM, which assisting the doctor to quickly diagnose. In the proposed method, the attention mechanism module is used, which is able to fully highlight useful features that facilitate segmentation while suppress useless features that are not conducive to segmentation. Combining with the multi-scale convolution, our encoder-decoder network can accurately segment the my-ocardium in DCM. We verified our approach on 1155 myocardial MRI. Our network achieves the most advanced segmentation performance on the cardiac DCM dataset. Experiment results demonstrate the effectiveness of the proposed method.