We have used data learning and low-precision computation to develop an implicit solver that demonstrates high performance up to 152,352 computer nodes (609,408 MPI processes × 12 OpenMP threads = 7,312,896 parallel computation) and conducted an unprecedented ultra-large-scale analysis of ultra-high-fidelity fault-structure systems using nonlinear dynamic finite element analysis on three-dimensional low-order unstructured elements. The developed solver achieved 25.45-fold speedup from the state of the art solver on Fugaku and attained weak scaling efficiency of 93.7% from 9.391 billion [email protected] computer nodes to 1.201 trillion [email protected],984 computer nodes on performance measurement problems. Moreover, a realistic 324 billion DOF application example, which is difficult to obtain performance for, was computed in high performance. Since the developed solver is based on a highly generalizable algorithm, it is expected to contribute not only to earthquake simulation on Fugaku but also to the enhancement of similar applications in other fields and on other supercomputers.

We develop a stochastic finite element method with ultra-large degrees of freedom that discretize probabilistic and physical spaces using unstructured second-order tetrahedral elements with double precision using a mixed-precision implicit iterative solver that scales to the full Fugaku system and enables fast Uncertainty Quantification (UQ). The developed solver designed to attain high performance on a variety of CPU/GPU-based supercomputers enabled solving 37 trillion degrees-of-freedom problem with 19.8% peak FP64 performance on full Fugaku (89.8 PFLOPS) with 87.7% weak scaling efficiency, corresponding to 224-fold speedup over the state of the art solver running on full Summit. This method, which has shown its effectiveness via solving huge (32-trillion degrees-of-freedom) practical problems, is expected to be a breakthrough in damage mitigation, and is expected to facilitate the scientific understanding of earthquake phenomena and have a ripple effect on other fields that similarly require UQ.

We present a scalable high-performance computing implementation of an agent-based economic model using distributed + shared-memory hybrid parallelization paradigms, capable of simulating 1:1 scale models of large economies like the eurozone. Agent-based economic models consist of millions of agents interacting over several graphs, which are either centralized or scale-free in nature. While most of the interactions are bi-directional, the interaction graphs are dense and random and keep evolving as the simulation progresses. These characteristics cause a very large and unknown number of random communications among MPI processes, posing challenges to developing scalable parallel extensions. Further, random access to large volume of data makes the algorithms highly memory-bound, severely degrading computational performance. Adopting various strategies inspired by the real-world functioning of economies, we reduce the large unknown number of communications to a known handful number. Memory-intensive algorithms are improved to make these cache-efficient, and advanced MPI functions are used to minimize communication overhead, thereby attaining higher performance and scalability. Further, an MPI + OpenMP hybrid model is developed to best utilize modern many-core computing nodes with low per-core memory capacity. It is demonstrated that our implementation can simulate a full fledged economic model with 331 million agents within 108 seconds using 128 CPU cores attaining 70 percent strong scalability.

Abstract Despite the significance of tracking human mobility dynamics in a large‐scale earthquake evacuation for an effective first response and disaster relief, the general understanding of evacuation behaviors remains limited. Numerous individual movement trajectories, disaster damages of civil engineering, associated heterogeneous data attributes, as well as complex urban environment all obscure disaster evacuation analysis. Although visualization methods have demonstrated promising performance in emergency evacuation analysis, they cannot effectively identify and deliver the major features like speed or density, as well as the resulting evacuation events like congestion or turn‐back. In this study, we propose a shot design approach to generate customized and narrative animations to track different evacuation features with different exploration purposes of users. Particularly, an intuitive scene feature graph that identifies the most dominating evacuation events is first constructed based on user‐specific regions or their tracking purposes on a certain feature. An optimal camera route, i.e., a storyboard is then calculated based on the previous user‐specific regions or features. For different evacuation events along this route, we employ the corresponding shot design to reveal the underlying feature evolution and its correlation with the environment. Several case studies confirm the efficacy of our system. The feedback from experts and users with different backgrounds suggests that our approach indeed helps them better embrace a comprehensive understanding of the earthquake evacuation.

Many wave-propagation analyses with varying geometries and material properties are expected to be useful for model optimization. Low-order unstructured finite-element methods are suitable for such analyses, as they are capable of modeling multi-material problems with complex geometries; however, the meshing and analysis cost is large. Therefore, in this paper, we developed a fast mesh-generator and analysis method. The robust mesh generator was 17.4-fold faster than a conventional mesh generator, and the predictor algorithm for dynamic implicit finite-element solvers showed a 1.69-fold increase in speed relative to conventional solvers and a 91.3% size-up efficiency on the full Oakforest-PACS system. We demonstrated the usability of the developed meshing and analysis methods via a wave-propagation simulation on a 1.9 billion unstructured tetrahedral-element model using half of the K computer system (41,472 compute nodes).

Many supercomputers are shifting to architectures with low B (byte/s; memory transfer capability) per F (FLOPS capability) ratios. However, utilizing increased F is difficult for applications that inherently require large B. Targeting an implicit unstructured low-order finite-element analysis solver, which typically requires large B, we have developed a concentrated computation algorithm that yields significant performance improvements on low B/F supercomputers. 35.7% peak performance was achieved for a sparse matrix-vector multiplication kernel, and 15.6% peak performance was achieved for the whole solver on the second generation Xeon Phi-based Oakforest-PACS. This is 5.02 times faster than (and 6.90 times the peak performance of) the state-of-the-art solver (the SC14 Gordon Bell finalist solver). On Oakforest-PACS, the proposed solver was approximately 2.42 times faster than the state-of-the-art solver running on the K computer. The proposed approach has implications for systems and applications and is expected to have significant impact on various fields that use finite-element methods for nonlinear time evolution problems.

This paper presents a new heroic computing method for unstructured, low-order, finite-element, implicit nonlinear wave simulation: 1.97 PFLOPS (18.6% of peak) was attained on the full K computer when solving a 1.08T degrees-of-freedom (DOF) and 0.270T-element problem. This is 40.1 times more DOF and elements, a 2.68-fold improvement in peak performance, and 3.67 times faster in time-to-solution compared to the SC14 Gordon Bell finalist's state-of-the-art simulation. The method scales up to the full K computer with 663,552 CPU cores with 96.6% sizeup efficiency, enabling solving of a 1.08T DOF problem in 29.7 s per time step. Using such heroic computing, we solved a practical problem involving an area 23.7 times larger than the state-of-the-art, and conducted a comprehensive earthquake simulation by combining earthquake wave propagation analysis and evacuation analysis. Application at such scale is a groundbreaking accomplishment and is expected to change the quality of earthquake disaster estimation and contribute to society.

With the aim of dramatically improving the reliability of urban earthquake response analyses, we developed an unstructured 3-D finite-element-based MPI-OpenMP hybrid seismic wave amplification simulation code, GAMERA. On the K computer, GAMERA was able to achieve a size-up efficiency of 87.1% up to the full K computer. Next, we applied GAMERA to a physics-based urban earthquake response analysis for Tokyo. Using 294,912 CPU cores of the K computer for 11 h, 32 min, we analyzed the 3-D non-linear ground motion of a 10.7 BlnDOF problem with 30 K time steps. Finally, we analyzed the stochastic response of 13,275 building structures in the domain considering uncertainty in structural parameters using 3 h, 56 min of 80,000 CPU cores of the K computer. Although a large amount of computer resources is needed presently, such analyses can change the quality of disaster estimations and are expected to become standard in the future.