A key trend facing extreme-scale computational science is the widening gap between computational and I/O rates, and the challenge that follows is how to best gain insight from simulation data when it is increasingly impractical to save it to persistent storage for subsequent visual exploration and analysis. One approach to this challenge is centered around the idea of in situ processing, where visualization and analysis processing is performed while data is still resident in memory. This paper examines several key design and performance issues related to the idea of in situ processing at extreme scale on modern platforms: scalability, overhead, performance measurement and analysis, comparison and contrast with a traditional post hoc approach, and interfacing with simulation codes. We illustrate these principles in practice with studies, conducted on large-scale HPC platforms, that include a miniapplication and multiple science application codes, one of which demonstrates in situ methods in use at greater than 1M-way concurrency.

Large-scale simulations conducted on supercomputers such as leadership-class computing facilities allow researchers to simulate and study complex problems with high fidelity, and thus have become indispensable in diverse areas of science and engineering. These high-fidelity simulations generate vast amount of data which is becoming more and more difficult to transform into knowledge using traditional visual analysis approaches. For instance, there are tremendous challenges in analyzing big data produced by high-fidelity simulations in order to gain meaningful insight into complex phenomena such as turbulent two-phase flows. The traditional workflow, which consists in conducting simulations on supercomputers and recording enormous raw simulation data to disk for further post-processing and visualization, is no longer a viable approach due to prohibitive cost of disk access and considerable amount of time spent on data transfer. Visual Analytics approaches for big data have to be researched and employed to address the problem of knowledge discovery from such large-scale simulations. One approach to tackle this issue is to couple a numerical simulation with in-situ visualization so that the post-processing and visualization occurs while the simulation is running. This in-situ approach minimizes data storage by extracting and visualizing important features of the data directly within the simulation without saving the raw data to disk. In addition, in-situ visualization allows users to steer the simulation by adjusting input parameters while the simulation is ongoing. In this paper, we present our approach for in-situ visualization of simulation data generated by massively parallel finite-element computational fluid dynamics solver (PHASTA) instrumented and linked with ParaView Catalyst. We demonstrate our in-situ visualization and simulation steering capability with a fully resolved turbulent flow through 2×2 reactor subchannel complex geometry. In addition, we present results from our in-situ visualization for turbulent flow simulations conducted on the supercomputers Cray XK7 “Titan” at Oak Ridge National Laboratory and IBM BlueGene/Q “Mira” at Argonne National Laboratory up to 32,768 cores and examine the overhead of in-situ visualization and its effect on code performance.