Distributed deep neural networks (DNN) training systems deployed across workers have been widely used in various domains, while the communication overhead among workers for synchronizing gradient tensors often becomes the performance bottleneck. To optimize communication efficiency, state-of-the-art studies often apply both two techniques: i) gradient sparsification compression, which truncates the gradient to its largest elements to reduce the communication traffic, and ii) tensor fusion, which merges multiple gradient tensors within a fusion buffer to transmit them together to reduce the communication startup overhead. However, we find that existing studies often apply gradient sparsification after tensor fusion (we call sparsification-behind tensor fusion), which leads to a fact that a lot of fused gradient tensors are missed after the sparsification, thus impairing the convergence performance.

Data-parallel deep neural networks (DNN) training systems deployed across nodes have been widely used in various domains, while the system performance is often bottlenecked by the communication overhead among workers for synchronizing gradients. Top-k sparsification compression is the de facto approach to alleviate the communication bottleneck, which truncates the gradient to its largest k elements before sending it to other nodes.

In this paper, we are interested in exploring the problem of full-resolution image segmentation, with the focus placed on learning full-resolution representations for biomedicine images. We divide the original resolution image into patches of different sizes in different stages and then extracte local features from large to small patches using efficient and flexible components in modern convolutional neural networks (CNN). Meanwhile, a multilayer perceptron (MLP) block intended for modeling long-range dependencies between patches is designed to compensate for the inherent inductive bias caused by convolution operations. In addition, we perform multi-scale fusion and receive representation information from parallel paths at each stage, resulting in a rich full-resolution representation. We evaluate the proposed method on different biomedical image segmentation tasks and it achieves a competitive performance compared to the latest deep learning segmentation methods. It is hoped that this method will serve as a useful alternative to biomedical image segmentation and provide an improved idea for the research based on full-resolution representation.