Whereas rapidly increasing heterogeneous compute capabilities continue to facilitate further scalability, modern applications often instead get limited by suboptimal data movement, as more and more data needs to be shipped across different hardware components (i.e., CPUs, GPUs, and other types of accelerators). We posit that understanding and improving data movement in modern use-cases require a holistic understanding of the underlying Hardware usage as well as the Communication patterns within the overall context of the Application, or as we call it, the HAC domain. Collecting and correlating HAC data currently requires interacting with several profiling tools and libraries, resulting in a tedious workflow that is neither scalable nor portable. Furthermore, existing tools for visualizing data movement profiles also focus on these domains individually, rather than offering a holistic view.We present a unified framework for tracking and visualizing data movement trends in large-scale applications deployed on heterogeneous architectures. Our framework has two interoperable components. (1) DMTracker is a lean software layer that provides a simple interface for configurable HAC profiling of GPU-enabled applications and abstracts away the complexity in using several profiling tools, resulting in a streamlined and time-correlated event history across the HAC domains. (2) DMVis is a web-based tool that combines several linked visualizations to offer holistic visual insights into the runtime behavior and resources utilization of applications, proving pivotal in identifying computationally expensive tasks and data transfers across devices. In this paper, we present the design and prototype of our framework, developed in consultation with domain experts and demonstrated on two case studies, including one for a large language model training. Initial impressions from the experts indicate a positive turn in their usual workflow of observing and tuning the performance through improved data movement strategies.

Optimizing the performance of large-scale parallel codes is critical for efficient utilization of computing resources. Code developers often explore various execution parameters, such as hardware configurations, system software choices, and application parameters, and are interested in detecting and understanding bottlenecks in different executions. They often collect hierarchical performance profiles represented as call graphs, which combine performance metrics with their execution contexts. The crucial task of exploring multiple call graphs together is tedious and challenging because of the many structural differences in the execution contexts and significant variability in the collected performance metrics (e.g., execution runtime). In this paper, we present Ensemble CallFlow to support the exploration of ensembles of call graphs using new types of visualizations, analysis, graph operations, and features. We introduce ensemble-Sankey, a new visual design that combines the strengths of resource-flow (Sankey) and box-plot visualization techniques. Whereas the resource-flow visualization can easily and intuitively describe the graphical nature of the call graph, the box plots overlaid on the nodes of Sankey convey the performance variability within the ensemble. Our interactive visual interface provides linked views to help explore ensembles of call graphs, e.g., by facilitating the analysis of structural differences, and identifying similar or distinct call graphs. We demonstrate the effectiveness and usefulness of our design through case studies on large-scale parallel codes.

Calling context trees (CCTs) couple performance metrics with call paths, helping understand the execution and performance of parallel programs. To identify performance bottlenecks, programmers and performance analysts visually explore CCTs to form and validate hypotheses regarding degraded performance. However, due to the complexity of parallel programs, existing visual representations do not scale to applications running on a large number of processors. We present CallFlow, an interactive visual analysis tool that provides a high-level overview of CCTs together with semantic refinement operations to progressively explore CCTs. Using a flow-based metaphor, we visualize a CCT by treating execution time as a resource spent during the call chain, and demonstrate the effectiveness of our design with case studies on large-scale, production simulation codes.

Understanding and tuning the performance of extreme-scale parallel computing systems demands a streaming approach due to the computational cost of applying offline algorithms to vast amounts of performance log data. Analyzing large streaming data is challenging because the rate of receiving data and limited time to comprehend data make it difficult for the analysts to sufficiently examine the data without missing important changes or patterns. To support streaming data analysis, we introduce a visual analytic framework comprising of three modules: data management, analysis, and interactive visualization. The data management module collects various computing and communication performance metrics from the monitored system using streaming data processing techniques and feeds the data to the other two modules. The analysis module automatically identifies important changes and patterns at the required latency. In particular, we introduce a set of online and progressive analysis methods for not only controlling the computational costs but also helping analysts better follow the critical aspects of the analysis results. Finally, the interactive visualization module provides the analysts with a coherent view of the changes and patterns in the continuously captured performance data. Through a multi-faceted case study on performance analysis of parallel discrete-event simulation, we demonstrate the effectiveness of our framework for identifying bottlenecks and locating outliers.