Distributed training improves the accuracy of large models by scaling parameter sizes. However, this scalability introduces communication bottlenecks between nodes. Gradient compression can alleviate these bottlenecks, but most existing methods are lossy, potentially compromising convergence and accuracy. This paper proposes a novel lossless compression algorithm for the floating-point gradient data, significantly improving compression efficiency compared to traditional lossless float compression algorithms while overcoming the shortcomings of lossy compression. First, new distance-run-length encoding and exponent prediction encoding using dynamic Huffman coding (D.Huff.) are designed based on the distinct statistical characteristics of zero and non-zero gradient data, effectively representing data values and zero locations. Second, an innovative hybrid Huffman coding (H.Huff.) is introduced, combining the advantages of dynamic and static Huffman coding. It enhances compression efficiency via local data adaptability, and simultaneously improves computational efficiency and hardware implementation friendliness with multiple fixed Huffman code tables. We also present a K-means-based probability vector clustering algorithm to effectively construct multiple prebuilt Huffman code tables. To evaluate the effectiveness of the proposed compression method, we selected two different deep neural network (DNN) learning tasks: (1) image classification, using ResNet50 and VGG16 for CIFAR-100 training, and ViT for ImageNet; and (2) language modeling, employing BERT on the IMDB dataset. The results are presented in Table 1. Furthermore, we tested mixed-precision training compression using the BF16 format. Experimental results demonstrate that our method achieves a lossless compression ratio of 0.4931 on typical gradient data, representing a significant 16.61% improvement over the state-of-the-art lossless float data compression approach, ndzip.

Asymmetric Numeral Systems (ANS) play a significant role in learned image codecs (LIC) because of their high coding efficiency. However, it constitutes a substantial portion of inference time, making it the main bottleneck in real-time LIC due to its high computational demands, complex control logic, and serial execution flow. To address these challenges, this paper introduces a hardware-oriented ANS algorithm hANS that reduces complex calculations for state encoding and state-symbol decoding. Furthermore, hANS employs fixed-latency calculation to eliminate control logic, which often causes inconsistent delays. To further enhance throughput, we propose a hardware architecture of ANS for LIC (ANS-LIC), introducing a novel hardware parallelism scheme that incorporates pipeline execution and multi-bin parallelism for encoding, along with multi-stream parallelism for decoding. Additionally, by optimizing the execution order, we achieve a reduction in hardware resource utilization during the decoding process. The proposed ANS-LIC hardware is implemented in RTL and synthesized using TSMC 65nm technology and the Alveo U250 Data Center Accelerator Card. We evaluate ANS-LIC on the Kodak and DIV2K LIC datasets, achieving a 1.17% compression ratio improvement over the SOTA method, Recoil. The implementation results and comparison with other works are presented in Table 1. The synthesis indicates that ANS-LIC requires only 385.5/393.0k gates for encoding and decoding, without SRAM. ANS-LIC achieves throughput improvements of 13.29×/1.47× for encoding and decoding over Recoil. In summary, the proposed ANS-LIC demonstrates substantial advantages.