We report performance results of large astrophysical N-body simulations on supercomputer Fugaku. We use the hybrid TreePM algorithm, in which the short-range force is evaluated by the oct-tree method, and the long-range force is solved by the particle-mesh method. We introduce three innovations that significantly improve the performance on the novel CPU and extremely large system: the near-ultimate optimization of gravity kernel for short-range forces, the domain decomposition with accelerated communications in hardware-oriented ways, and a novel communication algorithm for long-range forces. We demonstrate that our code scales near linearly up to 158,976 nodes (7,630,848 CPU cores) of Fugaku and enables us to integrate about 1.45 trillion particles per second, which is the world fastest time-to-solution. The average performance achieved is 95.82 Pflops, which corresponds to ∼ 10% of the peak in single precision.

We have simulated, for the first time, the long term evolution of the Milky Way Galaxy using 51 billion particles on the Swiss Piz Daint supercomputer with our N-body gravitational tree-code Bonsai. Herein, we describe the scientific motivation and numerical algorithms. The Milky Way model was simulated for 6 billion years, during which the bar structure and spiral arms were fully formed. This improves upon previous simulations by using 1000 times more particles, and provides a wealth of new data that can be directly compared with observations. We also report the scalability on both the Swiss Piz Daint and the US ORNL Titan. On Piz Daint the parallel efficiency of Bonsai was above 95%. The highest performance was achieved with a 242 billion particle Milky Way model using 18600 GPUs on Titan, thereby reaching a sustained GPU and application performance of 33.49 Pflops and 24.77 Pflops respectively.

As an entry for the 2012 Gordon-Bell performance prize, we report performance results of astrophysical N-body simulations of one trillion particles performed on the full system of K computer. This is the first gravitational trillion-body simulation in the world. We describe the scientific motivation, the numerical algorithm, the parallelization strategy, and the performance analysis. Unlike many previous Gordon-Bell prize winners that used the tree algorithm for astrophysical N-body simulations, we used the hybrid TreePM method, for similar level of accuracy in which the short-range force is calculated by the tree algorithm, and the long-range force is solved by the particle-mesh algorithm. We developed a highly-tuned gravity kernel for short-range forces, and a novel communication algorithm for long-range forces. The average performance on 24576 and 82944 nodes of K computer are 1.53 and 4.45 Pflops, which correspond to 49% and 42% of the peak speed.