ENRICO is a coupled application developed under the U.S. Department of Energy's Exascale Computing Project (ECP) targeting the modeling of advanced nuclear reactors. It couples radiation transport with heat and fluid simulation, including the high-fidelity, highresolution Monte-Carlo code Shift and the Computational fluid dynamics code NekRS. NekRS is a highly-performant open-source code for simulation of incompressible and low-Mach fluid flow, heat transfer, and combustion with a particular focus on turbulent flows in complex domains. It is based on rapidly convergent high-order spectral element discretizations that feature minimal numerical dissipation and dispersion. State-of-the-art multilevel preconditioners, efficient high-order time-splitting methods, and runtime-adaptive communication strategies are built on a fast OCCA-based kernel library, libParanumal, to provide scalability and portability across the spectrum of current and future high-performance computing platforms. On Frontier, Nek5000/RS has recently achieved an unprecedented milestone in breaching over 1 billion spectral elements and 350 billion degrees of freedom. Shift has demonstrated the capability to transport upwards of 1 billion particles per second in full core nuclear reactor simulations featuring complete temperature-dependent, continuous-energy physics on Frontier. Shift achieved a weak-scaling efficiency of 97.8% on 8192 nodes of Frontier and calculated 6 reactions in 214,896 fuel pin regions below 1% statistical error yielding first-of-a-kind resolution for a Monte Carlo transport application.

Large-scale simulations conducted on supercomputers such as leadership-class computing facilities allow researchers to simulate and study complex problems with high fidelity, and thus have become indispensable in diverse areas of science and engineering. These high-fidelity simulations generate vast amount of data which is becoming more and more difficult to transform into knowledge using traditional visual analysis approaches. For instance, there are tremendous challenges in analyzing big data produced by high-fidelity simulations in order to gain meaningful insight into complex phenomena such as turbulent two-phase flows. The traditional workflow, which consists in conducting simulations on supercomputers and recording enormous raw simulation data to disk for further post-processing and visualization, is no longer a viable approach due to prohibitive cost of disk access and considerable amount of time spent on data transfer. Visual Analytics approaches for big data have to be researched and employed to address the problem of knowledge discovery from such large-scale simulations. One approach to tackle this issue is to couple a numerical simulation with in-situ visualization so that the post-processing and visualization occurs while the simulation is running. This in-situ approach minimizes data storage by extracting and visualizing important features of the data directly within the simulation without saving the raw data to disk. In addition, in-situ visualization allows users to steer the simulation by adjusting input parameters while the simulation is ongoing. In this paper, we present our approach for in-situ visualization of simulation data generated by massively parallel finite-element computational fluid dynamics solver (PHASTA) instrumented and linked with ParaView Catalyst. We demonstrate our in-situ visualization and simulation steering capability with a fully resolved turbulent flow through 2×2 reactor subchannel complex geometry. In addition, we present results from our in-situ visualization for turbulent flow simulations conducted on the supercomputers Cray XK7 “Titan” at Oak Ridge National Laboratory and IBM BlueGene/Q “Mira” at Argonne National Laboratory up to 32,768 cores and examine the overhead of in-situ visualization and its effect on code performance.