The recent emergence of Large Language Model (LLM)-based tools such as OpenAI’s ChatGPT and Google’s Gemini has sparked excitement across the software development industry, and offered promises to transform the software development process. Despite the enthusiasm, it remains uncertain whether these tools are already good enough at coding to replace the role of software developers. Currently, no studies have provided insights into the performance of LLMs, such as understanding which characteristics of a programming task might affect an LLM's performance, or predicting how an LLM will handle new programming challenges. In this work, we address these challenges by first creating a data collection framework to gather 3,323 programming tasks from Kattis, a widely-used programming challenge platform. We then use OpenAI's ChatGPT to solve these programming tasks. The solutions obtained from ChatGPT are submitted back to Kattis to evaluate their correctness and effectiveness. Next, we use the collected data, including both problem and solution information, to analyze the task characteristics that significantly influence ChatGPT's performance. Building on the analysis, we develop predictive models that can forecast the efficacy of ChatGPT on new programming problems. Our analysis indicates that factors such as the difficulty level of a programming challenge, or the readability complexity of a problem description can significantly affect the efficacy of ChatGPT. Finally, the experimental results show that our predictive model can correctly predict ChatGPT performance with an accuracy of up to 90% for easy problems, and up to 79% for difficult problems.

Summary Coupled scientific simulation workflows are composed of heterogeneous component applications that simulate different aspects of the physical phenomena being modeled and that interact and exchange significant volumes of data at runtime. As the data volumes and generation rates keep growing, the traditional disk I/O–based data movement approach becomes cost prohibitive, and workflow requires more scalable and efficient approach to support the data movement. Moreover, the cost of moving large volume of data over system interconnection network becomes dominating and significantly impacts the workflow execution time. Minimize the amount of network data movement and localize data transfers are critical for reducing such cost. To achieve this, workflow task placement should exploit data locality to the extent possible and move computation closer to data. In this paper, we investigate applying in‐memory data staging and data‐centric task placement to reduce the data movement cost in large‐scale coupled simulation workflows. Specifically, we present a distributed data sharing and task execution framework that (1) co‐locates in‐memory data staging on application compute nodes to store data that needs to be shared or exchanged and (2) uses data‐centric task placement to map computations onto processor cores that a large portion of the data exchanges can be performed using the intra‐node shared memory. We also present the implementation of the framework and its experimental evaluation on Titan Cray XK7 petascale supercomputer.

As applications target extreme scales, data staging and in-situ/in-transit data processing have been proposed to address the data challenges and improve scientific discovery. However, further research is necessary in order to understand how growing data sizes from data intensive simulations coupled with the limited DRAM capacity in High End Computing systems will impact the effectiveness of this approach. In this paper, we explore how we can use deep memory levels for data staging, and develop a multi-tiered data staging method that spans bothDRAM and solid state disks (SSD). This approach allows us to support both code coupling and data management for data intensive simulation workflows. We also show how an adaptive application-aware data placement mechanism can dynamically manage and optimize data placement across the DRAM ands storage levels in this multi-tiered data staging method. We present an experimental evaluation of our approach using wolf resources: an Infiniband cluster (Sith) and a Cray XK7system (Titan), and using combustion (S3D) and fusion (XGC1) simulations.

Data staging and in-situ/in-transit data processing are emerging as attractive approaches for supporting extreme scale scientific workflows. These approaches improve end-to-end performance by enabling runtime data sharing between coupled simulations and data analytics components of the workflow. However, the complex and dynamic data exchange patterns exhibited by the workflows coupled with the varied data access behaviors make efficient data placement within the staging area challenging. In this paper, we present an adaptive data placement approach to address these challenges. Our approach adapts data placement based on application-specific dynamic data access patterns, and applies access pattern-driven and location-aware mechanisms to reduce data access costs and to support efficient data sharing between the multiple workflow components. We experimentally demonstrate the effectiveness of our approach on Titan Cray XK7 using a real combustion-analyses workflow. The evaluation results demonstrate that our approach can effectively improve data access performance and overall efficiency of coupled scientific workflows.

Summary Managing the large volumes of data produced by emerging scientific and engineering simulations running on leadership‐class resources has become a critical challenge. The data have to be extracted off the computing nodes and transported to consumer nodes so that it can be processed, analyzed, visualized, archived, and so on. Several recent research efforts have addressed data‐related challenges at different levels. One attractive approach is to offload expensive input/output operations to a smaller set of dedicated computing nodes known as a staging area. However, even using this approach, the data still have to be moved from the staging area to consumer nodes for processing, which continues to be a bottleneck. In this paper, we investigate an alternate approach, namely moving the data‐processing code to the staging area instead of moving the data to the data‐processing code. Specifically, we describe the ActiveSpaces framework, which provides (1) programming support for defining the data‐processing routines to be downloaded to the staging area and (2) runtime mechanisms for transporting codes associated with these routines to the staging area, executing the routines on the nodes that are part of the staging area, and returning the results. We also present an experimental performance evaluation of ActiveSpaces using applications running on the Cray XT5 at Oak Ridge National Laboratory. Finally, we use a coupled fusion application workflow to explore the trade‐offs between transporting data and transporting the code required for data processing during coupling, and we characterize sweet spots for each option. Copyright © 2014 John Wiley & Sons, Ltd.

Next generation in-situ/in-transit data processing has been proposed for addressing data challenges at extreme scales. However, further research is necessary in order to understand how growing data sizes from data intensive simulations coupled with limited DRAM capacity in High End Computing clusters will impact the effectiveness of this approach. In this work, we propose using deep memory levels for data staging, utilizing a multi-tiered data staging method with both DRAM and solid state disk (SSD). This approach allows us to support both code coupling and data management for data intensive simulations in cluster environment. We also show how an application-aware data placement mechanism can dynamically manage and optimize data placement across DRAM and SSD storage levels in staging method. We present experimental results on Sith - an Infiniband cluster at Oak Ridge, and evaluate its performance using combustion (S3D) and fusion (XGC) simulations.

SUMMARY Complex coupled multiphysics simulations are playing increasingly important roles in scientific and engineering applications such as fusion, combustion, and climate modeling. At the same time, extreme scales, increased levels of concurrency, and the advent of multicores are making programming of high‐end parallel computing systems on which these simulations run challenging. Although partitioned global address space (PGAS) languages attempt to address the problem by providing a shared memory abstraction for parallel processes within a single program, the PGAS model does not easily support data coupling across multiple heterogeneous programs, which is necessary for coupled multiphysics simulations. This paper explores how multiphysics‐coupled simulations can be supported by the PGAS programming model. Specifically, in this paper, we present the design and implementation of the XpressSpace programming system, which extends existing PGAS data sharing and data access models with a semantically specialized shared data space abstraction to enable data coupling across multiple independent PGAS executables. XpressSpace supports a global‐view style programming interface that is consistent with the PGAS memory model, and provides an efficient runtime system that can dynamically capture the data decomposition of global‐view data‐structures such as arrays, and enable fast exchange of these distributed data‐structures between coupled applications. In this paper, we also evaluate the performance and scalability of a prototype implementation of XpressSpace by using different coupling patterns extracted from real world multiphysics simulation scenarios, on the Jaguar Cray XT5 system at Oak Ridge National Laboratory. Copyright © 2013 John Wiley & Sons, Ltd.

As system scales and application complexity grow, managing and processing simulation data has become a significant challenge. While recent approaches based on data staging and in-situ/in-transit data processing are promising, dynamic data volumes and distributions, such as those occurring in AMR-based simulations, make the efficient use of these techniques challenging. In this paper we propose cross-layer adaptations that address these challenges and respond at runtime to dynamic data management requirements. Specifically we explore (1) adaptations of the spatial resolution at which the data is processed, (2) dynamic placement and scheduling of data processing kernels, and (3) dynamic allocation of in-transit resources. We also exploit coordinated approaches that dynamically combine these adaptations at the different layers. We evaluate the performance of our adaptive cross-layer management approach on the Intrepid IBM-BlueGene/P and Titan Cray-XK7 systems using Chombo-based AMR applications, and demonstrate its effectiveness in improving overall time-to-solution and increasing resource efficiency.

As scientific research becomes more data intensive, there is an increasing need for scalable, reliable, and high performance storage systems. Such data repositories must provide both data archival services and rich metadata, and cleanly integrate with large scale computing resources. ROARS is a hybrid approach to distributed storage that provides both large, robust, scalable storage and efficient rich metadata queries for scientific applications. In this paper, we demonstrate that ROARS is capable of importing and exporting large quantities of data, migrating data to new storage nodes, providing robust fault tolerance, and generating materialized views based on metadata queries. Our experimental results demonstrate that ROARS' aggregate throughput scales with the number of concurrent clients while providing fault-tolerant data access. ROARS is currently being used to store 5.1TB of data in our local biometrics repository.

Quality of data plays a very important role in any scientific research. In this paper we present some of the challenges that we face in managing and maintaining data quality for a terabyte scale biometrics repository. We have developed a step by step model to capture, ingest, validate, and prepare data for biometrics research. During these processes, there are many hidden errors which can be introduced into the data. Those errors can affect the overall quality of data, and thus can skew the results of biometrics research. We discuss necessary steps we have taken to reduce and eliminate the errors. Steps such as data replication, automated data validation, and logging metadata changes are both necessary and crucial to improve the quality and reliability of our data.

Today, campus grids provide users with easy access to thousands of CPUs. However, it is not always easy for nonexpert users to harness these systems effectively. A large workload composed in what seems to be the obvious way by a naive user may accidentally abuse shared resources and achieve very poor performance. To address this problem, we argue that campus grids should provide end users with high-level abstractions that allow for the easy expression and efficient execution of data-intensive workloads. We present one example of an abstraction—All-Pairs—that fits the needs of several applications in biometrics, bioinformatics, and data mining. We demonstrate that an optimized All-Pairs abstraction is both easier to use than the underlying system, achieve performance orders of magnitude better than the obvious but naive approach, and is both faster and more efficient than a tuned conventional approach. This abstraction has been in production use for one year on a 500 CPU campus grid at the University of Notre Dame and has been used to carry out a groundbreaking analysis of biometric data.

Research in the field of biometrics depends on the effective management of large amounts of data and computation. Current research projects in biometrics acquire many terabytes of images and video of subjects in many different modes and situations, annotated with detailed metadata. To study the effectiveness of new algorithms for identifying people, researchers must exhaustively compare large numbers of measurements with a variety of custom functions. The quality of the end results is often dependent upon the sheer amount of data marshalled to support it.To address these challenges, we are constructing BXGrid, an end-to-end computing system for conducting biometrics research. BXGrid assists with the entire research process from data acquisition all the way to generating results for publication. Because the entire chain of research is kept consistently within one system, multiple users may easily share tools and results, building off of each other's work. BXGrid also helps to ensure scientific integrity by automating a variety of consistency checks, external data audits, and reproduction of existing results.

Distributed systems such as clusters, clouds and grids remain a difficult platform for executing large data intensive workloads. Even sophisticated users struggle to shape complex workloads into the simple 'assembly language' of file transfer and job submission. To address this, problem, we advocate the use of abstractions, which are simple frameworks for expressing large structured problems. In this talk, we will discuss three examples of abstractions developed at the University of Notre Dame for scientific applications. In each case, we have been able to scale up workloads one to wto orders of magnitude larger than we previously feasible. Through each example, we will address some persistent obstacles in the field of distributed computing.

Network bandwidth is a precious commodity in the information age. One would be hard-pressed to cite an example of an industry, hobby, or effort that could not benefit, in some way, from a communication network. The ability to minimize the allocation time for communication channels within a network is one of the primary concerns of the network administrator. Network resource caching is one technique that may be used to reduce channel allocation time. Therefore, improvements to the standard cache management algorithms are an important component of achieving this goal. This study demonstrates the potential benefit of the time distance indexing factor (TDIF) cache replacement algorithm over a traditional technique, least recently used (LRU), in reducing communication channel allocation time. The benefit of this technique translates into an increase in the effective bandwidth of the network.