ABSTRACT Application performance is a function of several unknowns stemming from the interactions between the application, runtime, OS, and underlying hardware, making it challenging to model performance using deep learning techniques, especially without a large labeled dataset. Collecting such labeled longitudinal datasets can take weeks. Intuitively, developers could save analysis time during code development by taking a comparative approach between multiple applications. However, the unknown dynamic interactions between applications and execution environments make it difficult for deep learning‐based models to predict the performance of new applications. In this paper, we address these problems by presenting a labeled dataset for the community and taking a comparative analysis approach to explore the source code differences between different correct implementations of the same problem. This paper assesses the feasibility of using purely static information, for example, Abstract Syntax Tree (AST), of applications to predict performance change based on code structure. We evaluate several deep learning‐based representation learning techniques for source code and propose an architecture for the tree‐based Long Short‐Term Memory (LSTM) models to discover latent representations for a source code's hierarchical structure. We demonstrate that our proposed architecture enables feed‐forward predictive models to predict change in performance using source code with up to 84% accuracy.

Performance anomaly in High Performance Computing (HPC) can be defined as run-to-run variation of an application in repeated runs with the same set of configuration parameters. Such variations can occur for myriad reasons, including contention for shared resources such as the network and dynamic data distribution across application processes. Traditionally HPC researchers focus on real-time anomaly detection using different Machine Learning (ML) methods. These popular methods, such as auto-encoder, limit finding anomalous event patterns during training time. On top of that, in HPC, performance data are stored in tabular format. Though gradient-based methods have already proved their significant improvement over classification tasks, they explicitly use feature-feature relationships, ignoring the potential sample-sample relationship. To fill this gap, we build a performance anomaly classification technique leveraging the potential graph-based representation learning. We hypothesize that a meaningful and robust representation considering the sample-sample relationship for the given tabular datasets will improve the downstream anomaly classification technique. We conduct our experiment on 5 HPC datasets and 6 ML datasets. Our empirical study proves that graph-based anomaly classification outperforms the gradient-based approaches in 6 out of 11 experiments. We also explain how anomaly decisions are made inside the performance graph.

Performance anomalies can manifest as irregular execution times or abnormal execution events for many reasons, including network congestion and resource contention. Detecting such anomalies in real-time by analyzing the details of performance traces at scale is impractical due to the sheer volume of data High-Performance Computing (HPC) applications produce. In this paper, we propose formulating HPC performance anomaly detection as a signal-processing problem where anomalies can be treated as noise. We evaluate our proposed method in comparison with two other commonly used anomaly detection techniques of varying complexity based on their detection accuracy and scalability. Since real-time in-situ anomaly detection at a large scale requires lightweight methods that can handle a large volume of streaming data, we find that our proposed method provides the best trade-off. We then implement the proposed method in Chimbuko, the first online, distributed, and scalable workflow-level performance trace analysis framework. We compare our proposed signal-based anomaly detection algorithm with two other methods using a function of their accuracy, F1 score, and detection overhead. Our experiments demonstrate that our proposed approach achieves a 99% improvement for the benchmark datasets and a 93% improvement with Chimbuko traces.

Cost and power efficiency considerations have driven High Performance Computing (HPC) system design inno-vations in accelerator-based heterogeneous computing. Complex interactions between applications and heterogeneous hardware make it difficult for users to extract maximum performance out of these systems. While there is a plethora of performance measurement and analysis tools for CPU s, the same is not the case for GPUs. Existing tools either provide too high-level information or are overly complicated to setup, impeding performance profiling. While NVIDIA's CUPTI profiling library enables basic kernel-level measurements on NVIDIA's GPUs, it does not provide root-causes of performance slowdown. This paper presents a low-overhead, flexible, and user-friendly tool, LIBNV CD, built on top of CUPTI to simplify performance measurement and analysis of NVIDIA GPUs. LIBNVCD simplifies obtaining fine-grained measurements, requiring only three function calls in source, while masking changes and complexities of CUPTI. By automatically discovering performance event groups, LIBNV CD reduces data collection overhead significantly as many events (not all) can be measured at once. This user-friendly multi-GPU performance measurement tool incurs a mean overhead of less than 1% as compared to CUPTI, and has been released publicly.

Performance analysis has always been an afterthought during the application development process, focusing on application correctness first. The learning curve of the existing static and dynamic analysis tools are steep, which requires understanding low-level details to interpret the findings for actionable optimizations. Additionally, application performance is a function of a number of unknowns stemming from the application-, runtime-, and interactions between the OS and underlying hardware, making it difficult to model using any deep learning technique, especially without a large labeled dataset. In this paper, we address both of these problems by presenting a large corpus of a labeled dataset for the community and take a comparative analysis approach to mitigate all unknowns except their source code differences between different correct implementations of the same problem. We put the power of deep learning to the test for automatically extracting information from the hierarchical structure of abstract syntax trees to represent source code. This paper aims to assess the feasibility of using purely static information (e.g., abstract syntax tree or AST) of applications to predict performance change based on the change in code structure. This research will enable performance-aware application development since every version of the application will continue to contribute to the corpora, which will enhance the performance of the model. We evaluate several deep learning-based representation learning techniques for source code. Our results show that tree-based Long Short-Term Memory (LSTM) models can leverage source code's hierarchical structure to discover latent representations. Specifically, LSTM-based predictive models built using a single problem and a combination of multiple problems can correctly predict if a source code will perform better or worse up to 84% and 73% of the time, respectively.