The APEnet project is an established initiative aimed at designing interconnection boards based on FPGAs and tailored for use in HPC clusters whose nodes are arranged in a 3D toroidal network topology. Based on a custom communication protocol and network IPs, deployments of an APEnet Network Interface Card (NIC) can be configured for different operating environments. In this work, we describe APEnetX, the latest version of network architecture in the APEnet family, developed on Alveo U200 boards which leverage the 16 nm technology devices in the Ultrascale+ line by Xilinx. APEnetX features a PCIe Gen3×16 interface driven by the Queue Direct Memory Access (QDMA) Xilinx IP and adheres to the RDMA semantics, ensuring efficient high bandwidth data transfer without the involvement of the host operating system. The communication between adjacent nodes is handled through a proprietary lightweight protocol and enabled by the high-speed serial embedded transceivers of the FPGA, while a custom full crossbar switch and an optimized router allow for the implementation of a direct torus topology. Together with a proprietary software driver, a low level communication library and a dedicated MPI library, APEnetX lines up with current application standards in HPC systems. In addition, the release of an APEnetX network simulator allows for testing network functionalities at large scale. The achievement of this work is twofold: we enhance the performance of the previous APE net cards reaching 2 µs user-level latency over the PCIe for small packet transmission between topologically adjacent host nodes; we define the requirements of the communication generated by a reference spiking neural network simulator (NEST) to drive the co-design of future APE net generation.

The TEXTAROSSA project aims to bridge the technology gaps that exascale computing systems will face in the near future in order to overcome their performance and energy efficiency challenges. This project provides solutions for improved energy efficiency and thermal control, seamless integration of heterogeneous accelerators in HPC multi-node platforms, and new arithmetic methods. Challenges are tacked through a co-design approach to heterogeneous HPC solutions, supported by the integration and extension of HW and SW IPs, programming models, and tools derived from European research.

In order to enable Exascale computing, next generation interconnection networks must scale to hundreds of thousands of nodes, and must provide features to also allow the HPC, HPDA, and AI applications to reach Exascale, while benefiting from new hardware and software trends. RED-SEA will pave the way to the next generation of European Exascale interconnects, including the next generation of BXI, as follows: (i) specify the new architecture using hardware-software co-design and a set of applications representative of the new terrain of converging HPC, HPDA, and AI; (ii) test, evaluate, and/or implement the new architectural features at multiple levels, according to the nature of each of them, ranging from mathematical analysis and modeling, to simulation, or to emulation or implementation on FPGA testbeds; (iii) enable seamless communication within and between resource clusters, and therefore development of a high-performance low latency gateway, bridging seamlessly with Ethernet; (iv) add efficient network resource management, thus improving congestion resiliency, virtualization, adaptive routing, collective operations; (v) open the interconnect to new kinds of applications and hardware, with enhancements for end-to-end network services - from programming models to reliability, security, low- latency, and new processors; (vi) leverage open standards and compatible APIs to develop innovative reusable libraries and Fabrics management solutions.

To achieve high performance and high energy efficiency on near-future exascale computing systems, three key technology gaps needs to be bridged. These gaps include: energy efficiency and thermal control; extreme computation efficiency via HW acceleration and new arithmetics; methods and tools for seamless integration of reconfigurable accelerators in heterogeneous HPC multi-node platforms. TEXTAROSSA aims at tackling this gap through a co-design approach to heterogeneous HPC solutions, supported by the integration and extension of HW and SW IPs, programming models and tools derived from European research.

FPGA-based devices are gaining attention in the High-Performance Computing research field because of their reconfigurability, which allows to implement highly optimized domain specific architectures. With the availability of High-Level Synthesis tools, it is far easier now to implement acceleration cores on FPGAs, but the standard CPU+accelerator paradigm, which is the dominant model for GPGPU, falls short exploiting one of the key feature these devices provide, which is the high-throughput and low-latency network capability. In this paper we show a model for leveraging these network capabilities inside a custom accelerator block, allowing to build a net of interconnected computing blocks, which are able to share data directly, without host participation.

We profile the impact of computation and inter-processor communication on the energy consumption and on the scaling of cortical simulations approaching the real-time regime on distributed computing platforms. Also, the speed and energy consumption of processor architectures typical of standard HPC and embedded platforms are compared. We demonstrate the importance of the design of low-latency interconnect for speed and energy consumption. The cost of cortical simulations is quantified using the Joule per synaptic event metric on both architectures. Reaching efficient real-time on large scale cortical simulations is of increasing relevance for both future bio-inspired artificial intelligence applications and for understanding the cognitive functions of the brain, a scientific quest that will require to embed large scale simulations into highly complex virtual or real worlds. This work stands at the crossroads between the WaveScalES experiment in the Human Brain Project (HBP), which includes the objective of large scale thalamo-cortical simulations of brain states and their transitions, and the ExaNeSt and EuroExa projects, that investigate the design of an ARM-based, low-power High Performance Computing (HPC) architecture with a dedicated interconnect scalable to million of cores; simulation of deep sleep Slow Wave Activity (SWA) and Asynchronous aWake (AW) regimes expressed by thalamo-cortical models are among their benchmarks.

We measured the impact of long-range exponentially decaying intra-areal lateral connectivity on the scaling and memory occupation of a distributed spiking neural network simulator compared to that of short-range Gaussian decays. Indeed, while previous studies adopted short-range connectivity, recent experimental neurosciences studies are pointing out the role of longer-range intra-areal connectivity with implications on neural simulation platforms. Two- dimensional grids of cortical columns composed by up to 11 M point-like spiking neurons with spike frequency adaption were connected by up to 30 G synapses using short- and long-range connectivity models. The MPI processes composing the distributed simulator were run on up to 1024 hardware cores, hosted on a 64 nodes server platform. The hardware platform was a cluster of IBM NX360 M5 16-core compute nodes, each one containing two Intel Xeon Haswell 8-core E5-2630 v3 processors, with a clock of 2.40G Hz, interconnected through an InfiniBand network, equipped with 4 QDR switches.

The ExaNeSt project started on December 2015 and is funded by EU H2020 research framework (call H2020-FETHPC-2014, n. 671553) to study the adoption of low-cost, Linux-based power-efficient 64-bit ARM processors clusters for Exascale-class systems. The ExaNeSt consortium pools partners with industrial and academic research expertise in storage, interconnects and applications that share a vision of an Euro-pean Exascale-class supercomputer. Their goal is designing and implementing a physical rack prototype together with its cooling system, the storage non-volatile memory (NVM) architecture and a low-latency interconnect able to test different options for interconnection and storage. Furthermore, the consortium is to provide real HPC applications to validate the system. Herein we provide a status report of the project initial developments.