Training models with massive inputs is a significant challenge in the development of Deep Learning pipelines to process very large digital image datasets as required by Whole Slide Imaging (WSI) in computational pathology and analysis of brain fMRI images in computational neuroscience. Graphics Processing Units (GPUs) represent the primary workhorse in training and inference of Deep Learning models. In order to use GPUs to run inference or training on a neural network pipeline, state-of-the-art machine learning frameworks like PyTorch and TensorFlow currently require that the collective memory on the GPUs must be larger than the size of the activations at any stage in the pipeline. Therefore, existing Deep Learning pipelines for these use cases have been forced to develop sub-optimal "patch-based" modeling approaches, where images are processed in small segments of an image. In this paper, we present a solution to this problem by employing tiling in conjunction with check-pointing, thereby enabling arbitrarily large images to be directly processed, irrespective of the size of global memory on a GPU and the number of available GPUs. Experimental results using PyTorch demonstrate enhanced functionality/performance over existing frameworks.

The ability to monitor power and energy accurately and efficiently is becoming increasingly important in all areas of computing. In this paper, we introduce WattProf, a portable and flexible power monitoring platform that allows for fine-grained monitoring of power and energy consumption in various hardware components of HPC clusters. The WattProf system consist of a programmable monitoring PCIe expansion card, sensors for individual system components, host runtime system, and a high-level application programming interface. We demonstrate the use of WattProf on two benchmarks.

Many factors affect the performance and power characteristics of FPGA designs. Among them are the optimization parameters for synthesis, map, and place-and-route design tools. Choosing the right combination of these parameters can substantially lower power requirements, while still satisfying timing constraints. Finding such an improvement, however, requires significant experimentation by the FPGA designer. Exhaustive search through the parameter space is an automated alternative to experimentation but is intractable because of the large search space and the long build time of each parameter combination. In this paper, we propose a machine-learning-based approach to tune FPGA design parameters. It performs sampling-based reduction of the parameter space and guides the search toward promising parameter configurations. In our experiments, such selective sampling finds parameter configurations that meet the timing constraints and are within 0.2% of the optimal power consumption. Furthermore, these configurations are found in an order of magnitude less time compared with exhaustive search. Such speedups can substantially alleviate bottlenecks in the FPGA design process.

Complex parallel applications can often be modeled as directed acyclic graphs of coarse-grained application tasks with dependences. These applications exhibit both task and data parallelism, and combining these two (also called mixed parallelism) has been shown to be an effective model for their execution. In this paper, we present an algorithm to compute the appropriate mix of task and data parallelism required to minimize the parallel completion time (makespan) of these applications. In other words, our algorithm determines the set of tasks that should be run concurrently and the number of processors to be allocated to each task. The processor allocation and scheduling decisions are made in an integrated manner and are based on several factors such as the structure of the task graph, the runtime estimates and scalability characteristics of the tasks, and the intertask data communication volumes. A locality-conscious scheduling strategy is used to improve intertask data reuse. Evaluation through simulations and actual executions of task graphs derived from real applications and synthetic graphs shows that our algorithm consistently generates schedules with a lower makespan as compared to Critical Path Reduction (CPR) and Critical Path and Allocation (CPA), two previously proposed scheduling algorithms. Our algorithm also produces schedules that have a lower makespan than pure task- and data-parallel schedules. For task graphs with known optimal schedules or lower bounds on the makespan, our algorithm generates schedules that are closer to the optima than other scheduling approaches.

Quality of service (QoS) mechanisms allowing users to request for turn-around time guarantees for their jobs have recently generated much interest. In our previous work we had designed a framework, QoPS, to allow for such QoS. This framework provides an admission control mechanism that only accepts jobs whose requested deadlines can be met and, once accepted, guarantees these deadlines. However, the framework is completely blind to the revenue these jobs can fetch for the supercomputer center. By accepting a job, the supercomputer center might relinquish its capability to accept some future arriving (and potentially more expensive) jobs. In other words, while each job pays an explicit price to the system for running it, the system may also be viewed as paying an implicit opportunity cost by accepting the job. Thus, accepting a job is profitable only when the job's price is higher than its opportunity cost. In this paper we analyze the impact such opportunity cost can have on the overall revenue of the supercomputer center and attempt to minimize it through predictive techniques. Specifically, we propose two extensions to QoPS, value-aware QoPS (VQoPS) and dynamic value-aware QoPS (DVQoPS), to provide such capabilities. We present detailed analysis of these schemes and demonstrate using simulation that they not only achieve several factors improvement in system revenue, but also good service differentiation as a much desired side-effect.

This paper examines MPI's ability to support continuous, dynamic load balancing for unbalanced parallel applications. We use an unbalanced tree search benchmark (UTS) to compare two approaches, 1) work sharing using a centralized work queue, and 2) work stealing using explicit polling to handle steal requests. Experiments indicate that in addition to a parameter defining the granularity of load balancing, message-passing paradigms require additional parameters such as polling intervals to manage runtime overhead. Using these additional parameters, we observed an improvement of up to 2times in parallel performance. Overall we found that while work sharing may achieve better peak performance on certain workloads, work stealing achieves comparable if not better performance across a wider range of chunk sizes and workloads.

Complex applications can often be viewed as a collection of coarse-grained data-parallel application components with precedence constraints. It has been shown that combining task and data parallelism (mixed parallelism) can be an effective execution paradigm for these applications. In this paper, we present an algorithm to compute the appropriate mix of task and data parallelism based on the scalability characteristics of the tasks as well as the intertask data communication costs, such that the parallel completion time (makespan) is minimized. The algorithm iteratively reduces the makespan by increasing the degree of data parallelism of tasks on the critical path that have good scalability and a low degree of potential task parallelism. Data communication costs along the critical path are minimized by exploiting parallel transfer mechanisms and use of a locality conscious backfill scheduler. Evaluation using benchmark task graphs derived from real applications as well as synthetic graphs shows that our algorithm consistently performs better than previous scheduling schemes

Computationally complex applications can often be viewed as a collection of coarse-grained data-parallel tasks with precedence constraints. Researchers have shown that combining task and data parallelism (mixed parallelism) can be an effective approach for executing these applications, as compared to pure task or data parallelism. In this paper, we present an approach to determine the appropriate mix of task and data parallelism, i.e., the set of tasks that should be run concurrently and the number of processors to be allocated to each task. An iterative algorithm is proposed that couples processor allocation and scheduling of mixed-parallel applications on compute clusters so as to minimize the parallel completion time (makespan). Our algorithm iteratively reduces the makespan by increasing the degree of data parallelism of tasks on the critical path that have good scalability and a low degree of potential task parallelism. The approach employs a look-ahead technique to escape local minima and uses priority based backfill scheduling to efficiently schedule the parallel tasks onto processors. Evaluation using benchmark task graphs derived from real applications as well as synthetic graphs shows that our algorithm consistently performs better than CPR and CPA, two previously proposed scheduling schemes, as well as pure task and data parallelism

Meta-schedulers such as the LSF MultiCluster scheduler and the Moab meta-scheduler (Silver) are being deployed on clusters interconnected over the grid. A primary goal of such meta-schedulers is to share jobs amongst the individual local sites, balancing the load and improving utilization and turnaround time. This work focuses on current methodologies used in implemented meta-schedulers, such as LSF MultiCluster and Silver. The benefits and disadvantages of both centralized and delegated modes are evaluated. Focusing on the negative impact of meta-schedulers to jobs originating from lightly loaded sites, enhancements are proposed and evaluated via trace-driven simulation. It is shown that it is feasible to reduce the detrimental impact on lightly loaded sites while maintaining excellent overall performance.

Coordinated scheduling across multiple sites over the grid has become a possibility due to grid technologies such as Globus and the Silver meta-scheduler. This would allow user jobs to be transparently executed at remote sites across the grid, instead of a particular local cluster. Previous research has shown this type of job distribution to be beneficial in terms of average metrics such as loss of capacity and turnaround time. This research has sparked interest in implementing such schemes, for example on the Cluster Ohio system. However, an issue that has not been addressed is that of fairness - will jobs at less loaded sites be significantly adversely affected by the distributed scheduling schemes? Trace based simulations show that indeed, there can be considerable unfairness to the less loaded sites when previously proposed distributed scheduling schemes are used. We assess approaches to enhance fairness to jobs at local sites and show that they improve fairness while also providing very good overall performance.

Job scheduling has been a much studied topic over the years. While past research has studied the effect of various scheduling policies using metrics such as turnaround time, slowdown, utilization etc., there has been little research on how fair a nonpreemptive scheduling policy is. We propose an approach to assessing fairness in nonpreemptive job scheduling. Our basic model of fairness is that no later arriving job should delay an earlier arriving job. We quantitatively assess the fairness of several job scheduling strategies and propose a new strategy that seeks to improve fairness.