With the advancement of deep learning, network communication has become the most critical factor in model training. Especially, the all-reduce operation can comprise over 70% of the cumulative training duration as a pivotal component within data parallelism. However, existing all-reduce algorithms often perform poorly in complex network topologies, and there is currently no universal and straightforward all-reduce algorithm that can effectively adapt to diverse topological structures.In this work, we propose a topology decoupled all-reduce algorithm. We decouple the network into multiple tree substructures, select the trees with the smallest heights, and then split the data to perform aggregate communication within these selected structures. This approach significantly reduces the number of communications and enhances efficiency. Experimental results show that our topology decoupled all-reduce algorithm reduces communication time compared to NCCL’s by 42.9% and enhances end-to-end training efficiency by 11.7%.

Ring all-reduce is currently the most commonly used collective communication technique in the fields of data parallel and distributed computing. It consists of three phases: communication establishment, data transmission, and data processing at each step. However, this method may suffer from increased communication latency as the number of computation nodes increases, excessive communication steps and data processing procedures can lead to insufficient bandwidth utilization. To address this issue, this article proposes an Interleaved Bidirectional Ring (IBing) all-reduce method, which uses specially crafted communication operations to improve communication efficiency by reducing the effects of both communication establishment and data processing time. IBing reduces the number of communication steps by half compared to the Ring all-reduce. The results of extensive experiments indicate that the proposed IBing design can reduce total communication consumption by an average of 8.49% and up to 49.73%.

In the context of distributed deep learning, computational clusters place greater emphasis on static all-reduce communication latency while also needing to support large-scale networking. However, the communication efficacy of current all-reduce algorithms within specialized network topologies requires enhancement. Existing all-reduce communication algorithms inadequately exploit cluster bandwidth, leading to considerable bandwidth idleness. The optimization of communication algorithms becomes imperative to fully utilize the available bandwidth. Addressing this concern, we propose an innovative approach: a topology-aware interleaved all-reduce algorithm for Dragonfly networks (TIAD). Leveraging the inherent characteristics of the Dragonfly network, TIAD employs an interleaved communication mechanism for both intra- and inter-group data collection, significantly augmenting communication efficiency. Moreover, we refine the Dragonfly network with minimal adjustments, aligning it with the theoretical structure of interleaved communication. We also proposed an all-reduce communication method to complement the TIAD algorithm, specifically for scenarios where only a subset of nodes in the Dragonfly network participate in the communication task. Our experiments demonstrate that TIAD exhibits the shortest communication time across diverse node sizes and bandwidth conditions. Notably, our algorithm reduces communication time by up to 23.4% during the collection communication phase in comparison to the PAARD algorithm.

With the rapid development of blockchain technology, its network size has increased by hundreds or thousands of times. For large-scale blockchain networks, consensus algorithms need to be used to achieve consensus among dispersed nodes. Only blocks approved by more than half of the nodes can truly achieve blockchain. Therefore, communication efficiency in blockchain networks is crucial, and the excessive communication delay generated as the number of nodes increases can seriously affect the processing efficiency of blockchain systems. Therefore, we have designed a consensus mechanism based on grouping for the structural characteristics of P2P communication networks in blockchain systems, and a deep learning based grouping algorithm for the decentralized P2P networks. Through experiments, it can be seen that our algorithm can improve the communication efficiency of 74.3%-76.1% compared to the unoptimized decentralized P2P network nodes.

An interconnected distributed computing cloud, comprising network-linked computational nodes, facilitates the allocation of computational resources across diverse locations. The strategic offloading of computationally demanding tasks from resource-constrained nodes to proximate nodes endowed with ample resources can enhance the resource utilization within the computing cloud. Given the interdependencies among segmented tasks, improper task partitioning may engender supplementary overheads. Furthermore, the process of task offloading introduces additional communication overhead. Consequently, the judicious division of task components and the selection of appropriate offloading destinations assume paramount significance in mitigating collaborative computing complexity and communication overhead. To address these challenges, we propose a communication-efficient collaborative task offloading methodology, which entails the segmentation and offloading of computation-intensive tasks from the source node to one or more high-performance nodes to fully leverage device resources. Additionally, we devise a subtask placement strategy predicated on minimizing communication distances, whereby multiple subtasks are allocated to distinct nodes based on the calculated communication latencies between the source node and various computing nodes. Experiments demonstrate that our task offloading method exhibits the lowest overall loss.

Oracle Bone Inscriptions (OBI) is one of the oldest scripts in the world. The rejoining of Oracle Bone (OB) fragments is of vital importance to the research of ancient scripts and history. Although significant progress has been achieved in the past decades, the rejoining work still heavily relies on domain knowledge and manual work, thus remains a low efficient and time-consuming process Therefore, an automatic and practical algorithm/system for OB rejoining is of great value to the OBI community. To this end, we collect a real-world dataset for rejoining Oracle Bone fragments, namely OB-Rejoin, which consists of 998 OB rubbing images that suffer from low quality image problems, due to intrinsic underground eroding over time and extrinsic imaging conditions in the past. Moreover, a practical Self-Supervised Splicing Network, S3-Net, is proposed to rejoin the OB fragments based on shape similarity of their borderlines. Specifically, we first transform the manually annotated borderline strokes of OB images into times series style shape representations, which are fed as input to a Generative Adversarial Network for augmenting positive pairs of rejoinable OBs for each OB fragment that does not have rejoinable counterparts. A Siamese network is trained on such augmented data in a contrastive learning manner to retrieve the matching OB fragments of an unseen query from an OB fragment gallery. Experiments on the OB-Rejoin benchmark show that our data-driven approach outperforms two recent methods for time-series analysis. In order to demonstrate its practical potential, we deploy the proposed S3-Net method in real tests and ultimately discover dozens of new rejoinings missed by domain experts for decades.

Oracle Bone Inscriptions (OBI) research is very meaningful for both history and literature. In this paper, we introduce our contributions in AI-Powered Oracle Bone (OB) fragments rejoining and OBI recognition. (1) We build a real-world dataset OB-Rejoin, and propose an effective OB rejoining algorithm which yields a top-10 accuracy of 98.39%. (2) We design a practical annotation software to facilitate OBI annotation, and build OracleBone-8000, a large-scale dataset with character-level annotations. We adopt deep learning based scene text detection algorithms for OBI localization, which yield an F-score of 89.7%. We propose a novel deep template matching algorithm for OBI recognition which achieves an overall accuracy of 80.9%. Since we have been cooperating closely with OBI domain experts, our effort above helps advance their research. The resources of this work are available at https://github.com/chongshengzhang/OracleBone.