Clusters with GPUs are mainstream in HPC as shown by the last edition of the Top500 list, increasing the demand for GPU capable scientific computing software. Programming large scale GPU systems in an efficient and future-proof way present numerous challenges, such as optimizations for a variety of GPUs and interconnect hardware, hiding communication overhead with computation and efficient domain partitioning. We present an improvement to the CUDA-based communication of stencil applications in the WALBERLA framework, achieving scalability while supporting different GPUs and communication infrastructures. We utilize the lattice Boltzmann Method for fluid flows as a representative of stencil-based scientific computing and implement a communication hiding strategy that is capable of adjusting to a system's computing and communication capabilities. We compare the use of CUDAMemCopy with the use of customized pack/unpack kernels and show that packing achieves almost linear weak scaling behavior in the Santos Dumont supercomputer with up to 128 GPUs. We also show that the proposed approach is not sensitive to the direction of the domain partitioning, one of the biggest challenges when communicating 3D domains in GPU-based stencil simulations.

This article describes the development of automatic program generation technology to create scalable phase-field methods for material science applications. To simulate the formation of microstructures in metal alloys, we employ an advanced, thermodynamically consistent phase-field method. A state-of-the-art large-scale implementation of this model requires extensive, time-consuming, manual code optimization to achieve unprecedented fine mesh resolution. Our new approach starts with an abstract description based on free-energy functionals which is formally transformed into a continuous PDE and discretized automatically to obtain a stencil-based time-stepping scheme. Subsequently, an automatized performance engineering process generates highly optimized, performance-portable code for CPUs and GPUs. We demonstrate the efficiency for real-world simulations on large-scale GPU-based (PizDaint) and CPU-based (SuperMUC-NG) supercomputers. Our technique simplifies program development and optimization for a wide class of models.

Microstructures forming during ternary eutectic directional solidification processes have significant influence on the macroscopic mechanical properties of metal alloys. For a realistic simulation, we use the well established thermodynamically consistent phase-field method and improve it with a new grand potential formulation to couple the concentration evolution. This extension is very compute intensive due to a temperature dependent diffusive concentration. We significantly extend previous simulations that have used simpler phase-field models or were performed on smaller domain sizes. The new method has been implemented within the massively parallel HPC framework waLBerla that is designed to exploit current supercomputers efficiently. We apply various optimization techniques, including buffering techniques, explicit SIMD kernel vectorization, and communication hiding. Simulations utilizing up to 262,144 cores have been run on three different supercomputing architectures and weak scalability results are shown. Additionally, a hierarchical, mesh-based data reduction strategy is developed to keep the I/O problem manageable at scale.

waLBerla is a massively parallel software framework for simulating complex flows with the lattice Boltzmann method (LBM). Performance and scalability results are presented for SuperMUC, the world's fastest x86-based supercomputer ranked number 6 on the Top500 list, and JUQUEEN, a Blue Gene/Q system ranked as number 5.