• LUGH is a new buffer architecture for Dragonfly Interconnection Networks. • LUGH achieves high performance and fairness in Dragonfly networks. • A theoretical model to compare the fairness of buffer architectures has been obtained. • Theoretical throughput calculations align with simulation results. • LUGH is a cost-effective solution which outperforms other VC mechanisms in most scenarios. Dragonfly networks offer a viable solution for large-scale supercomputers and datacenters. However, developing efficient routing mechanisms for these networks presents significant challenges. Current solutions often lead to unstable network behavior due to congestion and fairness issues, exacerbating performance variability and the tail-latency problem. An analysis of the topology and its standard deadlock avoidance mechanisms reveals that server access to global network links varies based on their location in the network, resulting in throughput unfairness. To address this issue, this paper introduces a novel switch buffer architecture which reduces head-of-line blocking and enhances fairness, to significantly improve overall network performance. Despite offering comparable cost to existing solutions, the proposed buffer architecture proves superior performance. Real-world synthetic simulations scenarios further confirm these findings, showing performance improvements between 10 % and 47 % against conventional solutions in medium sized Dragonflies.

High-radix, low-diameter networks like HyperX and Dragonfly use a Full-mesh core, and rely on multiple virtual channels (VCs) to avoid packet deadlocks in adaptive routing. However, VCs introduce significant overhead in the switch in terms of area, power, and design complexity, limiting the switch scalability. This paper starts by revisiting VC-less routing through link ordering schemes in Full-mesh networks, which offer implementation simplicity but suffer from performance degradation under adversarial traffic. Thus, to overcome these challenges, we propose TERA (Topology-Embedded Routing Algorithm), a novel routing algorithm which employs an embedded physical subnetwork to provide deadlock-free non-minimal paths without using VCs. In a Full-mesh network, TERA outperforms link ordering routing algorithms by 80% when dealing with adversarial traffic, and up to 100% in application kernels. Furthermore, compared to other VC-based approaches, it reduces buffer requirements by 50%, while maintaining comparable latency and throughput. Lastly, early results from a 2D-HyperX evaluation show that TERA outperforms state-of-the-art algorithms that use the same number of VCs, achieving performance improvements of up to 32%. Index Terms-Deadlock-free routing, Full-mesh, virtual channels, adaptive routing.

This work presents CAMINOS, a new interconnection network simulator focusing on router microarchitecture. It was developed in Rust, a novel programming language with a syntax similar to C/C++ and strong memory protection. The architecture of CAMINOS emphasizes the composition of components. This allows new designs to be defined in a configuration file without modifying source code, greatly reducing effort and time. In addition to simulation functionality, CAMINOS assists in managing a collection of simulations as an experiment. This includes integration with SLURM to support executing batches of simulations and generating PDFs with results and diagnostics. We show that CAMINOS makes good use of computing resources. Its memory usage is dominated by in-flight messages, showing low overhead in memory usage. We attest that CAMINOS can effectively use CPU time, as scenarios with little contention execute faster.

Abstract Since today’s HPC and data center systems can comprise hundreds of thousands of servers and beyond, it is crucial to equip them with a network that provides high performance. New topologies proposed to achieve such performance need to be evaluated under different traffic conditions, aiming to closely replicate real-world scenarios. While most optimizations should be guided by common traffic patterns, it is essential to ensure that no pathological traffic pattern can compromise the entire system. Determining synthetic adversarial traffic patterns for a network typically relies on a thorough understanding of its topology and routing. In this paper, we address the problem of identifying a generic adversarial traffic pattern for low-diameter direct interconnection networks. We first focus on Random Regular Graphs (RRGs), which represent a typical case for these networks. Moreover, RRGs have been proposed as topologies for interconnection networks due to their superior scalability and expandability, among other advantages. We introduce Ant Mill, an adversarial traffic pattern for RRGs when using routes of minimal length. Secondly, we demonstrate that the Ant Mill traffic pattern is also adversarial in other low-diameter direct interconnection networks such as Slimfly, Dragonfly, and Projective networks. Ant Mill is thoroughly motivated and evaluated, enabling future studies of low-diameter direct interconnection networks to leverage its findings.

To interconnect their growing number of servers, current supercomputers and data centers are starting to adopt low-diameter networks, such as HyperX, Dragonfly and Dragonfly+. These emergent topologies require balancing the load over their links and finding suitable non-minimal routing mechanisms for them becomes particularly challenging. The Valiant load balancing scheme is a very popular choice for non-minimal routing. Evolved adaptive routing mechanisms implemented in real systems are based on this Valiant scheme. All these low-diameter networks are deadlock-prone when non-minimal routing is employed. Routing deadlocks occur when packets cannot progress due to cyclic dependencies. Therefore, developing efficient deadlock-free packet routing mechanisms is critical for the progress of these emergent networks. The routing function includes the routing algorithm for path selection and the buffers management policy that dictates how packets allocate the buffers of the switches on their paths. For the same routing algorithm, a different buffer management mechanism can lead to a very different performance. Moreover, certain mechanisms considered efficient for avoiding deadlocks, may still suffer from hard to pinpoint instabilities that make erratic the network response. This paper focuses on exploring the impact of these buffers management policies on the performance of current interconnection networks, showing a 90% of performance drop if an incorrect buffers management policy is used. Moreover, this study not only characterizes some of these undesirable scenarios but also proposes practicable solutions.

Supercomputer and datacenter networks can comprise hundreds of thousands of severs. Focusing on direct networks, different topologies have been proposed to attain such a high scalability, from Flattened Butterfly and Dragonfly to the most disruptive approach represented by Jellyfish, which is based on a random interconnection pattern. The routing problem on such networks remains a challenge that can be tackled as a topology aware solution, or with an agnostic approach. The case of random networks is a very special one because of the lack of an acceptable routing algorithm for them since no a priori topological clues can be exploited. In this paper, we introduce the Polarized Routing Algorithm, an adaptive non-minimal hop-by-hop mechanism for direct networks that can be used in a number of topologies, including Jellyfish. Polarized routing was conceived following two design criteria: a source-destination symmetry in the routes to enable load-balancing and to avoid undoing previously taken hops. A thorough experimentation shows Polarized routing constitutes an efficient and versatile solution, attaining the highest performance both in benign scenarios under uniform traffic patterns and in adverse ones on the tested networks. Interestingly, this algorithm provides important performance gains of more than 30% in the Jellyfish topology, for different traffic patterns, when compared to the state of the art solutions.

Large computer systems, like those in the TOP 500 ranking, comprise about hundreds of thousands cores. Simulating application execution in these systems is very complex and costly. This article explores the option of using application skeletons, together with an analytic simulator, to study the performance of these large systems. With this aim, the Dimemas simulator has been enhanced with the capability of simulating application skeletons. This enhancement allows simulating the skeleton of Lulesh, an application with 90k processes in a single day. In addition, it also generates traces, which is of great value to validate skeletons and simulations.

Big scale, high performance and fault-tolerance, low-cost and graceful expandability are pursued features in current datacenter networks (DCN). Although there have been many proposals for DCNs, most modern installations are equipped with classical folded Clos networks. Recently, regular random topologies, as the Jellyfish, have been proposed for DCNs. However, their completely unstructured nature entails serious design problems. In this paper we propose Random Folded Clos (RFC) and Hydra networks in which the interconnection between certain switches levels is made randomly. Both RFCs and Hydras preserve important properties of Clos networks that provide a straightforward deadlock-free multi-path routing. The proposed networks leverage randomness to be gracefully expandable, thereby allowing for fine grain upgrading. RFCs and Hydras are compared in the paper, in topological and cost terms, against fat-trees, orthogonal fat-trees and random regular networks. Also, experiments are carried out to simulate their performance under synthetic traffic patterns emulating common loads present in warehouse scale computers. These theoretical and empirical studies reveal the interest of these topologies, concluding that Hydra constitutes a practicable alternative to current datacenter networks since it appropriately balance all the main design requirements. Moreover, Hydras perform better than the fat-trees, their natural competitor, being able to connect the same or more computing nodes with significant lower cost and latency while exhibiting comparable throughput.

In datacenter networks, big scale, high performance and faulttolerance, low-cost, and graceful expandability are pursued features. Recently, random regular networks, as the Jellyfish, have been proposed for satisfying these stringent requirements. However, their completely unstructured design entails several drawbacks. As a related alternative, in this paper we propose Random Folded Clos (RFC) networks. They constitute a compromise between total randomness and maintaining some topological structure. As it will be shown, RFCs preserve important properties of Clos networks that provide a straightforward deadlock-free equal-cost multi-path routing and enough randomness to gracefully expanding. These networks are minutely compared, in topological and cost terms, against fat-trees, orthogonal fat-trees and random regular graphs. Also, experiments are carried out to simulate their performance under synthetic traffics that emulate common loads in datacenters. It is shown that RFCs constitute an interesting alternative to currently deployed networks since they appropriately balance all the important design requirements. Moreover, they do that at much lower cost than the fat-tree, their natural competitor. Being able up to connect the same number of compute nodes, saving up to 95% of the cost, and giving similar performance.

The interconnection network comprises a significant portion of the cost of large parallel computers, both in economic terms and power consumption. Several previous proposals exploit large-radix routers to build scalable low-distance topologies with the aim of minimizing these costs. However, they fail to consider potential unbalance in the network utilization, which in some cases results in suboptimal designs. Based on an appropriate cost model, this paper advocates the use of networks based on incidence graphs of projective planes, broadly denoted as Projective Networks. Projective Networks rely on generalized Moore graphs with uniform link utilization and encompass several proposed direct (PN and demi-PN) and indirect (OFT) topologies under a common mathematical framework. Compared to other proposals with average distance between 2 and 3 hops, these networks provide very high scalability while preserving a balanced network utilization, resulting in low network costs.

A construction of two-quasi-perfect Lee codes is given over the space ℤnpfor p prime, p ≡ ±5 (mod 12), and n = 2[p/4]. It is known that there are infinitely many such primes. Golomb and Welch conjectured that perfect codes for the Lee metric do not exist for dimension n ≥ 3 and radius r ≥ 2. This conjecture was proved to be true for large radii as well as for low dimensions. The codes found are very close to be perfect, which exhibits the hardness of the conjecture. A series of computations show that related graphs are Ramanujan, which could provide further connections between coding and graph theories.

In the late years many different interconnection networks have been used with two main tendencies. One is characterized by the use of high-degree routers with long wires while the other uses routers of much smaller degree. The latter rely on two-dimensional mesh and torus topologies with shorter local links. This paper focuses on doubling the degree of common 2D meshes and tori while still preserving an attractive layout for VLSI design. By adding a set of diagonal links in one direction, diagonal networks are obtained. By adding a second set of links, networks of degree eight are built, named king networks. This research presents a comprehensive study of these networks which includes a topological analysis, the proposal of appropriate routing procedures and an empirical evaluation. King networks exhibit a number of attractive characteristics which translate to reduced execution times of parallel applications. For example, the execution times NPB suite are reduced up to a 30 percent. In addition, this work reveals other properties of king networks such as perfect partitioning that deserves further attention for its convenient exploitation in forthcoming high-performance parallel systems.

Dragonfly networks have a two-level hierarchical arrangement of the network routers, and allow for a competitive cost-performance solution in large systems. Non-minimal adaptive routing is employed to fully exploit the path diversity and increase the performance under adversarial traffic patterns. Throughput unfairness prevents a balanced use of the resources across the network nodes and degrades severely the performance of any application running on an affected node. Previous works have demonstrated the presence of throughput unfairness in Dragonflies under certain adversarial traffic patterns, and proposed different alternatives to effectively combat such effect. In this paper we introduce a new traffic pattern denoted adversarial consecutive (ADVc), which portrays a real use case, and evaluate its impact on network performance and throughput fairness. This traffic pattern is the most adversarial in terms of network fairness. Our evaluations, both with or without transit-over-injection priority, show that global misrouting policies do not properly alleviate this problem. Therefore, explicit fairness mechanisms are required for these networks.

Torus networks of moderate degree have been widely used in the supercomputer industry. Tori are superb when used for executing applications that require near-neighbor communications. Nevertheless, they are not so good when dealing with global communications. Hence, typical 3D implementations have evolved to 5D networks, among other reasons, to reduce network distances. Most of these big systems are mixed-radix tori, which are not the best option for minimizing distances and efficiently using network resources. This paper is focused on improving the topological properties of this kind of networks. By using integral matrices to deal with Cayley graphs over Abelian groups, we have been able to propose and analyze a family of high-dimensional mesh-based interconnection networks. As they are built over n-dimensional grids that induce a regular tiling of space, these topologies have been denoted lattice graphs. Higher dimensional networks can be composed over these graphs by means of a lift operation, which is also introduced in the paper. Easy network partitioning and minimal routing algorithm are also provided for these topologies based on this new network operation. Later we focus on cubic crystal lattices for modeling symmetric 3D networks and to show how lattice graphs can help in the design of twisted interconnection networks. In all cases, the networks obtained are better, in topological terms, than their standard tori counterparts. Finally, some practical issues such as implementability and preliminary performance evaluations have been addressed at the end of this work.

Current High-Performance Computing (HPC) and data center networks rely on large-radix routers. Hamming graphs (Cartesian products of complete graphs) and dragonflies (two-level direct networks with nodes organized in groups) are some direct topologies proposed for such networks. The original definition of the dragonfly topology is very loose, with several degrees of freedom, such as the inter- and intragroup topology, the specific global connectivity, and the number of parallel links between groups (or trunking level). This work provides a comprehensive analysis of the topological properties of the dragonfly network, providing balancing conditions for network dimensioning, as well as introducing and classifying several alternatives for the global connectivity and trunking level. From a topological study of the network, it is noted that a Hamming graph can be seen as a canonical dragonfly topology with a high level of trunking. Based on this observation and by carefully selecting the global connectivity, the Dimension Order Routing (DOR) mechanism safely used in Hamming graphs is adapted to dragonfly networks with trunking. The resulting routing algorithms approximate the performance of minimal, nonminimal, and adaptive routings typically used in dragonflies but without requiring virtual channels to avoid packet deadlock, thus allowing for lower cost router implementations. This is obtained by properly selecting the link to route between groups based on a graph coloring of network routers. Evaluations show that the proposed mechanisms are competitive with traditional solutions when using the same number of virtual channels and enable for simpler implementations with lower cost. Finally, multilevel dragonflies are discussed, considering how the proposed mechanisms could be adapted to them.

Many current VLSI on-chip multiprocessors and systems-on-chip employ point-to-point switched interconnection networks. Rings and 2D-meshes are among the most popular interconnection topologies for these increasingly important onchip networks. Nevertheless, rings cannot scale beyond dozens of nodes and meshes are asymmetric. Two of the key features of square 2D-tori are their scalability and symmetry. As higher scalability is demanded by the increasing number of cores (or specialized units) integrated on a chip and symmetry is critical for high-performance and load balancing, we concentrate on 2D-tori. However, most popular deadlock-free routing mechanisms are based on Dimension Order Routing (DOR) which breaks the torus symmetry when managing adversarial traffic patterns. This paper analyzes this problem and its consequences. After that, it proposes a new deadlock-free fully adaptive minimal routing, denoted as σDOR, that preserves tori symmetry under any load. It uses just two virtual channels to avoid DOR-induced asymmetry, the same as in previous competitive proposals. σDOR exhibits better behavior than any of previous solutions as it allows packets to dynamically adapt to local congestion. Experimental results evidence the superior performance of our mechanism, confirming the negative impact of DOR asymmetry.

A complete family of Cayley graphs of degree four, denoted as L-networks, is considered in this paper. L-networks are 2D mesh-based topologies with wrap-around connections. L-networks constitute a graph-based model which englobe many previously proposed 2D interconnection networks. Some of them have been extensively used in the industry as the underlying topology for parallel and distributed computers of different scales. Tori, twisted and doubly twisted tori, toroidal diagonal meshes, chordal rings, and circulant graphs are, among others, members of the L-network family. Therefore, many results obtained in previous studies on these networks can be deduced from the general framework presented in this work. In addition, the network model presented in this work allows for new results on the domain of low-degree interconnection networks. Particularly, closed expressions for the graph distance properties have been derived and an optimal routing algorithm of constant complexity is provided. Since symmetry has a big impact on network performance, we have also identified which L-networks are symmetric by studying their group of automorphisms. Finally, a very simple model that predicts the performance of L-networks is also presented. Such model has been contrasted with empirical evaluation.

The problem of searching for perfect codes has attracted great attention since the paper by Golomb and Welch, in which the existence of these codes over Lee metric spaces was considered. Since perfect codes are not very common, the problem of searching for quasi-perfect codes is also of great interest. In this aspect, also quasi-perfect Lee codes have been considered for 2-D and 3-D Lee metric spaces. In this paper, constructive methods for obtaining quasi-perfect codes over metric spaces modeled by means of Gaussian and Eisenstein-Jacobi integers are given. The obtained codes form ideals of the integer ring thus preserving the property of being geometrically uniform codes. Moreover, they are able to correct more error patterns than the perfect codes which may properly be used in asymmetric channels. Therefore, the results in this paper complement the constructions of perfect codes previously done for the same integer rings. Finally, decoding algorithms for the quasi-perfect codes obtained in this paper are provided and the relationship of the codes and the Lee metric ones is investigated.

Dragonfly networks have been recently proposed for the interconnection network of forthcoming exascale supercomputers. Relying on large-radix routers, they build a topology with low diameter and high throughput, divided into multiple groups of routers. While minimal routing is appropriate for uniform traffic patterns, adversarial traffic patterns can saturate inter-group links and degrade the obtained performance. Such traffic patterns occur in typical communication patterns used by many HPC applications, such as neighbor data exchanges in multi-dimensional space decompositions. Non-minimal traffic routing is employed to handle such cases. Adaptive policies have been designed to select between minimal and nonminimal routing to handle variable traffic patterns. However, previous papers have not taken into account the effect of saturation of intra-group (local) links. This paper studies how local link saturation can be common in these networks, and shows that it can largely reduce the performance. The solution to this problem is to use nonminimal paths that avoid those saturated local links. However, this extends the maximum path length, and since all previous routing proposals prevent deadlock by relying on an ascending order of virtual channels, it would imply unaffordable cost and complexity in the network routers. In this paper we introduce a novel routing/flow-control scheme that decouples the routing and the deadlock avoidance mechanisms. Our model does not impose any dependencies between virtual channels, allowing for on-the-fly (in-transit) adaptive routing of packets. To prevent deadlock we employ a deadlock-free escape sub network based on injection restriction. Simulations show that our model obtains lower latency, higher throughput, and faster adaptation to transient traffic, because it dynamically exploits a higher path diversity to avoid saturated links. Notably, our proposal consumes traffic bursts 43% faster than previous ones.

In this paper we consider perfect codes over two dimensional QAM-type constellations of any cardinal. Such constellations are going to be modeled by L-graphs, which are the two-dimensional family of multidimensional circulants, defined. We show that Gaussian graphs, Lee graphs and the Kronecker product of two cycles are included in this family. Therefore, our method to obtain perfect codes over these lattice subsets is a generalization of the techniques for searching perfect two-dimensional Lee codes and perfect codes over the Kronecker products of two cycles. In addition, we introduce some previously unreported perfect codes.

The SMILE project attempts to build efficient lowcost clusters based on FPGA boards using their reconfigurability capabilities. A real parallel application of Content-Based Information Retrieval over the SMILE cluster is presented. Using this application the SMILE cluster’s performance is evaluated and compared in terms of time and power consumption with traditional cluster architecture.

The SMILE project accelerates scientific and industrial applications by means of a cluster of low-cost FPGA boards. With this approach the intensive calculation tasks are accelerated using the FPGA logic, while the communication patterns of the applications remains unchanged by using a Message Passing Library over Linux. This paper explains the cluster architecture: the SMILE nodes and the developed high-speed communication network for the FPGA RocketIO interfaces. A SystemC model developed to simulate the cluster is also detailed. In order to show the potential of the SMILE proposal a Content-Based Information Retrieval parallel application has been developed and compared with a HP cluster architecture in terms of response time andpower consumption.

In order to propose a new metric over QAM constellations, diagonal Gaussian graphs defined over quotients of the Gaussian integers are introduced in this paper. Distance properties of the constellations are detailed by means of the vertex-to-vertex distribution of this family of graphs. Moreover, perfect codes for this metric are considered. Finally, notable subgraphs of diagonal Gaussian graphs are studied which leads to relate the proposed metric to other well-known graph-based metrics such as the Lee distance.