Supporting such an all-to-all traffic matrix is challenging as it can easily lead to congestion. Scheduling patterns are designed to avoid such congestion by spreading the communications over time. The time is divided in phases and communications are spread across the phases. However, current scheduling algorithms are not fault-tolerant. In this paper we propose a fault-adaptive congestion-free scheduling to support an all-to-all exchange in fat tree topology. Our approach consist in the computation of the minimum number of communication phases required to support the all-to-all exchange with the available links, and of the scheduling of the communications on these phases. It enables to recover from failures and makes optimal use of the remaining bandwidth. We show that our scheduling approach provides better performance than the most common approach which is the Linear-shift scheduling. The throughput is improved by roughly 80% with our approach, for as little as one link failure.

When high throughput and utilization of fabric at close-to-the-link capacity are most needed in a cluster, Ethernet is a potential candidate, rivaling traditional HPC interconnects. The distributed real-time data acquisition at particle physics experiments presents an interesting use case. This article evaluates possible Ethernet-based solutions for aggregating data from hundreds of data sources at a throughput of dozens of Tb/s. This leads us to many-to-one data exchanges where we strive for a cost-optimized setup sustaining more than 80 % of the theoretical link-load. We investigate possible Ethernet-based traffic patterns to handle data acquisition on large multi-source apparatuses. Different numbers of producers and receivers and different link speeds are allowed in a large-scale network. Performance tests were conducted using customized benchmarks and evaluation test benches. The article presents tested scenarios and problems encountered in practice. We describe how our findings influenced the design of a large production system at CERN. We also present relevant general conclusions for a broader range of applications of Ethernet in HPC.

The ATLAS detector at CERN records particle collision "events" delivered by the Large Hadron Collider. Its data-acquisition system identifies, selects, and stores interesting events in near real-time, with an aggregate throughput of several 10 GB/s. It is a distributed software system executed on a farm of roughly 2000 commodity worker nodes communicating via TCP/IP on an Ethernet network. Event data fragments are received from the many detector readout channels and are buffered, collected together, analyzed and either stored permanently or discarded. This system, and data-acquisition systems in general, are sensitive to the latency of the data transfer from the readout buffers to the worker nodes. Challenges affecting this transfer include the many-to-one communication pattern and the inherently bursty nature of the traffic. In this paper we introduce the main performance issues brought about by this workload, focusing in particular on the so-called TCP incast pathology. Since performing systematic studies of these issues is often impeded by operational constraints related to the mission-critical nature of these systems, we focus instead on the development of a simulation model of the ATLAS data-acquisition system, used as a case study. The simulation is based on the well-established OMNeT++ framework. Its results are compared with existing measurements of the system's behavior. The successful reproduction of the measurements by the simulations validates the modeling approach. We share some of the preliminary findings obtained from the simulation, as an example of the additional possibilities it enables, and outline the planned future investigations.