When high throughput and utilization of fabric at close-to-the-link capacity are most needed in a cluster, Ethernet is a potential candidate, rivaling traditional HPC interconnects. The distributed real-time data acquisition at particle physics experiments presents an interesting use case. This article evaluates possible Ethernet-based solutions for aggregating data from hundreds of data sources at a throughput of dozens of Tb/s. This leads us to many-to-one data exchanges where we strive for a cost-optimized setup sustaining more than 80 % of the theoretical link-load. We investigate possible Ethernet-based traffic patterns to handle data acquisition on large multi-source apparatuses. Different numbers of producers and receivers and different link speeds are allowed in a large-scale network. Performance tests were conducted using customized benchmarks and evaluation test benches. The article presents tested scenarios and problems encountered in practice. We describe how our findings influenced the design of a large production system at CERN. We also present relevant general conclusions for a broader range of applications of Ethernet in HPC.

The upgrade of the LHCb experiment at CERN envisions a Data Acquisition and Event Filtering system that captures 100% of the data generated by the various sub-detectors, which measure with great precision the 40 million collisions per second of protons in CERN's Large Hadron Collider. The sensor readings result in about 40 Tbit/s of data, which need to be processed on a large computer farm. Since the computation on CPUs, as it is currently done, does not scale well, it is necessary to accelerate a good portion of the code to meet the computational demands of the proposed system. We are therefore looking for means to accelerate the most time-consuming parts of the event-filtering code. The Ring Imaging Cherenkov (RICH) detectors are one of the component detectors of the overall LHCb experiment. The Cherenkov photon that hits the detector are processed to determine the track of the original particle that caused these photons. The particle velocity and mass, derived from the Cherenkov angle, is used to identify the particle. The entire RICH photon reconstruction algorithm accounts for 50% of the second High Level Trigger (HLT) process and Cherenkov angle reconstruction comprises about 20% of the RICH and is a good candidate for acceleration. An OpenCL implementation of Cherenkov angle reconstruction algorithm that calculates the trajectory of Photons in the RICH detector was developed. The paper looks at the results of the OpenCL implementation of the algorithm on the Nallatech 385 card with Altera Stratix V FPGA, Nvidia GeForce GTX 690 GPU card and the Intel Xeon processor for comparison. While the two GPUs are 3.6× faster than a single FPGA, the FPGA is 3.4× better than two GPUs and 6.6× better than two multicore CPUs when energy efficiency is factored. Although significant speedup of computation was achieved on all the above architectures by using OpenCL, a good portion of the gain was lost due to the overhead of data transfer and parallelism. Different strategies are put forth for improving the speedup. Some optimizations currently possible, low latency links that can replace PCIe and some possible changes to the OpenCL execution model itself are discussed.