In this work, we develop a coupler, h3-Open-UTIL/MP, as part of the h3-Open-BDEC project. The h3-Open-UTIL/MP coupler has an ensemble coupling capability to run multiple coupled programs in parallel and an interface to enable Python application coupling. In this paper, first, we outlined the features of h3-Open-UTIL/MP and its ensemble coupling capabilities. Then, we reported the details of Python application coupling and a coupling case study of the atmospheric model NICAM using a machine learning (ML) framework. To couple a Python application, we added a Python wrapper to the h3-Open-UTIL/MP interface provided as a Fortran module. Subsequently, we load Fortran routines as a library in the Python application. We used the ML framework PyTorch as a coupling example. We targeted the NICAM cloud physics subroutine and trained it to generate output variables from input variables. The results show a room for improvement in terms of reproducibility. Reproducibility can be improved in several ways: data selection, preprocess improvement, and changes in ML algorithms. In addition, the ML perspective was the bottleneck of execution time. Hence, there is a clear need to improve the computational performance. One effective method to improve computational performance is to run NICAM on a conventional high-performance computer and PyTorch on a GPU machine. We are currently developing a heterogeneous machine coupling program to implement this coupling method, which is briefly described in the final section of this paper.

Double precision(FP64) is generally used in computer science, and an incomplete Cholesky preconditioned conjugate gradient(ICCG) method, which is widely used in computer simulations, is a typical example. Recently, the use of single-precision(FP32) in the ICCG method with well-conditioned problems is discussed for reducing computational time and power consumption. Also, the use of half-precision(FP16) is examined in the field of machine learning, and hardware support of FP16 on GPUs and some CPUs is advancing. Using FP16 can further reduce the amount of memory translation from FP32, so high effects can be expected if it can be used with the ICCG method. When we apply FP16 to the ICCG method, It is difficult to solve the ill-condition problem because of the poor expressiveness of the exponent of FP16. Therefore, in this study, we evaluate the usefulness of FP16 in the ICCG method with several conditions of the problem, and we also proposed the implementation for CPU of and evaluated FP21 and FP42 adaptive precision whose expressiveness is higher than FP16. Additionally, to reduce the calculation time by using low precision, it is necessary to maintain high memory width consumption by an effective SIMDization. Therefore, in this study, we also evaluate the performance of ELLPACK(ELL) and Sell-C-σ storage formats, which can efficiently use SIMDization.

Parallel-in-time approaches for time-dependent PDEs have attracted a great deal of attention in the context of massively parallel environments. One such approach, multigrid reduction in time (MGRIT), extracts temporal parallelism by applying the reduction-based multigrid method for the temporal domain. To make efficient use of this parallelism, some processors need to be idle on a temporal coarse level of MGRIT. This paper proposes an optimization of the implementation that uses idle processors by assigning the spatial redistributed domain on coarse levels. It accelerates coarse-level spatial solvers and promotes early switching to parallelization-in-time by reducing its overhead. Although the spatial redistribution is contrary to the motivation of parallel-in-time approaches, it is expected to be effective when we need to balance spatial and temporal parallelism. This is because parallelization-in-time is already usually switched on when parallelization-in-space starts to saturate, so one can still benefit from additional parallelism. Numerical experiments demonstrate an improvement in runtime, at most, about 16.6% compared with pure MGRIT, which assigns best spatial and temporal parallelism at specific parallelism.

This special issue on Innovative research and development toward the exascale era explores new foundational and translational research toward enabling exascale computing for emerging scientific and societal challenges. Exascale computing is defined as the capability to perform 1018 operations per second. Productively harnessing such a scale of processing, storage, and networking capabilities for diverse domains— including high-performance computing (HPC) simulations, artificial intelligence (AI), and extreme data-driven computing— relies on not only revitalizing existing parallel and distributed computing technologies but also innovating new solutions. Papers in this special issue explore diverse topics in research encompassing parallel, distributed, and heterogeneous systems for exascale, including advances in applications, programming environments, runtimes, libraries, innovative algorithms, domain-specific frameworks, systems architecture, performance analysis, data processing, and networking technologies.

In this paper, we propose a multiplicative Schwartz-type block multi-color Gauss-Seidel (MS-BMC-GS) smoother for algebraic multigrid (AMG) methods. AMG is an excellent solver and one of the most effective preconditioners for Krylov subspace methods such as the conjugate gradient method. The achievable degree of parallelism, convergence ratio, and computational cost of AMG strongly depend on the chosen smoother. As multiple unknowns are relaxed simultaneously, the MS-BMC-GS smoother realizes higher convergence than the existing parallel Gauss-Seidel smoother. Although this increases the amount of computation, the increase in the computational time is mitigated by the high cache hit ratio owing to the novel blocking technique. Numerical experiments demonstrate that MS-BMC-GS outperforms the block multi-color GS smoother by 18%.

The low-precision floating point arithmetic that performs computation by reducing numerical accuracy with narrow bit-width is attracting since it can improve the performance of the numerical programs. Small memory footprint, faster computing speed, and energy saving are expected by performing calculation with low precision data. However, there have not been many studies on how low-precision arithmetics affects power and energy consumption of numerical codes. In this study, we investigate the power efficiency improvement by aggressively using low-precision arithmetics for HPC applications. In our evaluations, we analyze power characteristics of the Poisson's equation and the ground motion simulation programs with double precision and single precision floating point arithmetics. We confirm that energy efficiency improves by using low-precision arithmetics but it is heavily influenced by parameters such as data division and the number of OpenMP threads.

Hierarchical matrices (H-matrices) are an approximation technique for dense matrices, such as the coefficient matrix of the boundary element method (BEM). An H-matrix is expressed by a set of low-rank approximated and small dense sub-matrices, each of which has various ranks. The use of H-matrices reduces the required memory footprint of dense matrices from O(N^2) to O(NlogN) and is suitable for many-core processors that have relatively small memory capacities compared to traditional CPUs. However, existing parallel adaptive cross approximation (ACA) algorithms, which are low-rank approximation algorithms used to construct H-matrices, are not designed to exploit many-core processors in terms of load balancing. In existing parallel algorithms, the ACA process is independently applied to each sub-matrix. The computational load of the ACA process for each sub-matrix depends on the sub-matrix's rank; however, the rank is defined after the ACA process is applied. This makes it difficult to balance the load. We propose load-balancing-aware parallel ACA algorithms for H-matrices that focus on many-core processors. We implemented the proposed algorithms into HACApK, which is an open-source H-matrix library originally developed for CPU-based clusters. The proposed algorithms were evaluated using BEM problems on an NVIDIA Tesla P100 GPU (P100) and an Intel Xeon Broadwell processor. The evaluation results demonstrate the improved performance of the proposed algorithms in all GPU cases. For example, in a case where it is difficult for existing parallel algorithms to balance the load, the proposed algorithms achieved a 12.9 times performance improvement for the P100.

Many wave-propagation analyses with varying geometries and material properties are expected to be useful for model optimization. Low-order unstructured finite-element methods are suitable for such analyses, as they are capable of modeling multi-material problems with complex geometries; however, the meshing and analysis cost is large. Therefore, in this paper, we developed a fast mesh-generator and analysis method. The robust mesh generator was 17.4-fold faster than a conventional mesh generator, and the predictor algorithm for dynamic implicit finite-element solvers showed a 1.69-fold increase in speed relative to conventional solvers and a 91.3% size-up efficiency on the full Oakforest-PACS system. We demonstrated the usability of the developed meshing and analysis methods via a wave-propagation simulation on a 1.9 billion unstructured tetrahedral-element model using half of the K computer system (41,472 compute nodes).

Multi-kernels leverage today's multi-core chips to run multiple operating system (OS) kernels, typically a Light Weight Kernel (LWK) and a Linux kernel, simultaneously. The LWK provides high performance and scalability, while the Linux kernel provides compatibility. Multi-kernels show the promise of being able to meet tomorrow's extreme-scale computing needs while providing strong isolation, yielding high performance and scalability needed by classical HPC applications. McKernel and mOS started as independent research initiatives to explore the above potential. Previous work described their design and architecture advantages. This paper deploys the two LWKs and presents results from running them on a 2,048-node system with Intel Xeon Phi processors (KNL) connected by Intel Omni-Path Fabric. We compare the performance of McKernel, mOS, and Linux. Although the two multi-kernel efforts approached the problem from different angles, the results show a median performance improvement of 9% with some applications as high as 280% validating the efficacy of the multi-kernel approach. We provide insight into the performance improvements and discuss the strengths of the two different multi-kernel approaches.

Many supercomputers are shifting to architectures with low B (byte/s; memory transfer capability) per F (FLOPS capability) ratios. However, utilizing increased F is difficult for applications that inherently require large B. Targeting an implicit unstructured low-order finite-element analysis solver, which typically requires large B, we have developed a concentrated computation algorithm that yields significant performance improvements on low B/F supercomputers. 35.7% peak performance was achieved for a sparse matrix-vector multiplication kernel, and 15.6% peak performance was achieved for the whole solver on the second generation Xeon Phi-based Oakforest-PACS. This is 5.02 times faster than (and 6.90 times the peak performance of) the state-of-the-art solver (the SC14 Gordon Bell finalist solver). On Oakforest-PACS, the proposed solver was approximately 2.42 times faster than the state-of-the-art solver running on the K computer. The proposed approach has implications for systems and applications and is expected to have significant impact on various fields that use finite-element methods for nonlinear time evolution problems.

The parallel multigrid method is expected to play an important role in large-scale scientific computing on post-peta/exa-scale supercomputer systems, and it also includes serial and parallel communication processes which are generally expensive. In the present work, new format for sparse matrix storage based on sliced Ellpack-Itpack (ELL) format is proposed for optimization of serial communication in data transfer through memories, and hierarchical coarse grid aggregation (hCGA) is introduced for optimization of parallel communication by message passing. The proposed methods are implemented for pGW3D-FVM, a parallel code for 3D groundwater flow simulations using the multigrid method, and the robustness and performance of the code was evaluated on up to 4,096 nodes (65,536 cores) of the Fujistu FX10 supercomputer system at the University of Tokyo. The parallel multigrid solver using the sliced ELL format provided performance improvement in both weak scaling (25%-31%) and strong scaling (9%-22%) compared to the code using the original ELL format. Moreover, hCGA provided excellent performance improvement in both weak scaling (1.61 times) and strong scaling (6.27 times) for flat MPI parallel programming model.

To understand the macroscopic function of the human heart based on sub-cellular microscopic events, multiscale analysis is indispensable. Our heart simulator uses the so-called homogenization method, where both the human heart and the myocardial cells are modeled and solved simultaneously by the finite element method. Because the contraction and deformation of each finite element in the heart model are governed by their respective cell model, the NDOF of all the cells becomes prohibitively large. Furthermore, the phenomena are highly nonlinear and transient. This challenging problem has been tackled by our group for many years, and a novel algorithm to accelerate the computation was implemented in the code. We have recently tested its performance using the T2K Open Supercomputer (Tokyo). A pulsation of the heart with a total NDOF of 160 million was successfully simulated using 6144 CPU cores in ten hours. Scalability and other computational performances were measured and are discussed.

In this work, parallel preconditioning methods based on ldquoHierarchical Interface Decomposition (HID)rdquo and hybrid parallel programming models were applied to finite-element based simulations of linear elasticity problems in media with heterogeneous material properties. Reverse Cuthill-McKee reordering with cyclic multicoloring (CM-RCM) was applied for parallelism through OpenMP. The developed code has been tested on the ldquoT2K Open Super Computer (Todai Combined Cluster)rdquo using up to 512 cores. Preconditioners based on HID provide a scalable performance and robustness in comparison to conventional localized block Jacobi preconditioners. Performance of Hybrid 4x4 parallel programming model is competitive with that of Flat MPI.

An efficient parallel iterative method with selective blocking preconditioning has been developed for symmetric multiprocessor (SMP) cluster architectures with vector processors such as the Earth Simulator. This method is based on a three-level hybrid parallel programming model, which includes message passing for inter-SMP node communication, loop directives by OpenMP for intra-SMP node parallelization and vectorization for each processing element (PE). This method provides robust and smooth convergence and excellent vector and parallel performance in 3D geophysical simulations with contact conditions performed on the Earth Simulator. The selective blocking preconditioning is much more efficient than ILU(1) and ILU(2). Performance for the complicated Southwest Japan model with more than 23 M DOF on 10 SMP nodes (80 PEs) of the Earth Simulator was 161.7 GFLOPS, corresponding to 25.3% of the peak performance for hybrid programming model, and 190.4 GFLOPS (29.8% of the peak performance) for flat MPI, respectively.

Abstract A new multigrid‐preconditioned conjugate gradient (MGCG) iterative method for parallel computers is presented. Iterative solvers with preconditioning, such as the incomplete Cholesky or incomplete LU factorization methods, represent some of the most powerful tools for large‐scale scientific computation. However, the number of iterations required for convergence by these methods increases with the size of the problem. In multigrid solvers, the rate of convergence is independent of problem size, and the number of iterations remains fairly constant. Multigrid is also a good preconditioning algorithm for Krylov iterative solvers. In this study, the MGCG method is applied to Poisson equations in the region between two spherical surfaces on semi‐unstructured, adaptively generated prismatic grids, and to grids with local refinement. Computations using this method on a Hitachi SR2201 with up to 128 processors demonstrated good scalability. Copyright © 2002 John Wiley & Sons, Ltd.

Abstract In this paper, an efficient parallel iterative method for unstructured grids developed by the authors for shared memory symmetric multiprocessor (SMP) cluster architectures on the GeoFEM platform is presented. The method is based on a three‐level hybrid parallel programming model, including message passing for inter‐SMP node communication, loop directives for intra‐SMP node parallelization and vectorization for each processing element (PE). Simple 3D elastic linear problems with more than $10^8$ degrees of freedom have been solved by $3\times3$ block ICCG(0) with additive Schwarz domain decomposition and PDJDS/CM‐RCM reordering on 16 SMP nodes of a Hitachi SR8000 parallel computer, achieving a performance of 20 Gflops. The PDJDS/CM‐RCM reordering method provides excellent vector and parallel performance in SMP nodes, and is essential for parallelization of forward/backward substitution in IC/ILU factorization with global data dependency. The method developed was also tested on an NEC SX‐4 and attained 969 Mflops (48.5% of peak performance) using a single processor. The additive Schwarz domain decomposition method provides robustness for the GeoFEM parallel iterative solvers with localized preconditioning. Copyright © 2002 John Wiley & Sons, Ltd.