Summary We present a detailed examination of time agreement characteristics for nodes within extreme‐scale parallel computers. Using a software tool we introduce in this paper, we quantify attributes of clock skew among nodes in three representative high‐performance computers sited at three national laboratories. Our measurements detail the statistical properties of time agreement among nodes and how time agreement drifts over typical application execution durations. We discuss the implications of our measurements, why the current state of the field is inadequate, and propose strategies to address observed shortcomings.

We model an oversubscribed heterogeneous computing system where tasks arrive dynamically and a scheduler maps the tasks to machines for execution. The environment and workloads are based on those being investigated by the Extreme Scale Systems Center at Oak Ridge National Laboratory. Utility functions that are designed based on specifications from the system owner and users are used to create a metric for the performance of resource allocation heuristics. Each task has a time-varying utility (importance) that the enterprise will earn based on when the task successfully completes execution. We design multiple heuristics, which include a technique to drop low utility-earning tasks, to maximize the total utility that can be earned by completing tasks. The heuristics are evaluated using simulation experiments with two levels of oversubscription. The results show the benefit of having fast heuristics that account for the importance of a task and the heterogeneity of the environment when making allocation decisions in an oversubscribed environment. The ability to drop low utility-earning tasks allow the heuristics to tolerate the high oversubscription as well as earn significant utility.

SUMMARY We present a new software‐based clock synchronization scheme that provides high precision time agreement among distributed memory nodes. The technique is designed to minimize variance from a reference chimer during runtime and with minimal time‐request latency. Our scheme permits initial unbounded variations in time and corrects both slow and fast chimers (clock skew). An implementation developed within the context of the message passing interface is described, and time coordination measurements are presented. Among our results, the mean time variance for a set of nodes improved from 20.0 ms under standard Network Time Protocol down to 2.29 μs under our scheme. Copyright © 2012 John Wiley & Sons, Ltd.

We present a new software-based clock synchronization scheme that provides high precision time agreement among distributed memory nodes. The technique is designed to minimize variance from a reference chimer during runtime and with minimal time-request latency. Our scheme permits initial unbounded variations in time and corrects both slow and fast chimers (clock skew). An implementation developed within the context of the MPI message passing interface is described and time coordination measurements are presented. Among our results, the mean time variance among a set of nodes improved from 20.0 milliseconds under standard Network Time Protocol (NTP) to 2.29 μsecs under our scheme.

Grid computing offers a model for solving large-scale scientific problems by uniting computational resources owned by multiple organizations to form a single cohesive resource for the duration of individual jobs. Despite the appeal of using grid computing to solve large problems, its use has been hindered by the challenges involved in developing applications that can run efficiently in grid environments. One substantial obstacle to deploying grid applications across geographically distributed resources is cross-site latency. While certain classes of applications, such as master-slave style or functional decomposition type applications, lend themselves well to running in grid environments due to inherent latency tolerance, other classes of applications, such as tightly-coupled applications in which each processor regularly communicates with its neighboring processors, represent a significant challenge to deployment on grids. In this paper, we present a dynamic load balancing technique for grid applications based on graph partitioning. This technique exploits knowledge of the topology of the grid environment to partition the computation's communication graph in such a way as to reduce the volume of cross-site communication, thus improving the performance of tightly-coupled applications that are co-allocated across distributed resources. Our technique is particularly well suited to codes from disciplines like molecular dynamics or cosmology due to the non-uniform structure of communication in these types of applications. We evaluate the effectiveness of our technique when used to optimize the execution of a tightly-coupled classical molecular dynamics code called LeanMD deployed in a grid environment.

We have observed that supercomputing clusters made up of commodity off-the-shelf computers possess emergent properties that are apparent when these systems are considered as an indivisible entity rather than as a collection of independent nodes. By exploiting predicatable characteristics inherent to supercomputing clusters coupled with these emergent properties, we propose several mechanisms by which cluster security may be enhanced. In this paper, we describe NVisionCC, a cluster security tool that monitors processes across cluster nodes and raises alerts when deviations from a predefined profile of expected processes are noted. Additionally, we demonstrate that the monitoring infrastructure used by NVisionCC incurs a negligible performance penalty on the computational and network resources of the cluster.

Current methods for monitoring the security of large-scale commodity clusters tend to treat these clusters as nothing more than collections of independent nodes. As such, the techniques used to secure these clusters have, for the most part, been adaptations of techniques developed for securing and monitoring enterprise computing environments. We have previously proposed the idea of monitoring the security-state of large-scale commodity clusters by examining their emergent properties, that is, properties that are only visible when one ceases to look at a cluster as a collection of disparate nodes and begins to look at the properties of the cluster as a whole. We show that by correlating the open network ports observed on cluster nodes with other emergent properties - such as active processes and the contents of important system files - security analysts can make insightful observations that can greatly restrict the actions that an attacker can carry out undetected.

Large-scale commodity clusters are used in an increasing number of domains: academic, research, and industrial environments. At the same time, these clusters are exposed to an increasing number of attacks coming from public networks. Therefore, mechanisms for efficiently and flexibly managing security have now become an essential requirement for clusters. However, despite the growing importance of cluster security, this field has been only minimally addressed by contemporary cluster administration techniques. This paper presents a high-level view of existing security challenges related to clusters and proposes a structured approach for handling security in clustered servers. The goal of this paper is to identify various necessarily-distributed security services and their related characteristics as a means of enhancing cluster security.

Large-scale commodity cluster systems are finding increasing deployment in academic, research, and commercial settings. As a direct result of this popularity, cluster systems are also under increasing security threats. Unfortunately, there have been no corresponding improvements in security tools that specifically address the unique needs of cluster security. The paper describes an ongoing research effort at the National Center for Supercomputing Applications to develop tools for managing cluster security via process monitoring. We describe an extensible architecture and present details of a prototype process monitoring tool focused on intrusion detection.

Applications that use high-speed networks to connect geographically distributed supercomputers, databases, and scientific instruments may operate over open networks and access valuable resources. Hence, they can require mechanisms for ensuring integrity and confidentiality of communications and for authenticating both users and resources. Security solutions developed for traditional client-server applications do not provide direct support for the program structures, programming tools, and performance requirements encountered in these applications. We address these requirements via a security-enhanced version of the Nexus communication library, which we use to provide secure versions of parallel libraries and languages, including the Message Passing Interface. These tools permit a fine degree of control over what, where, and when security mechanisms are applied. In particular, a single application can mix secure and nonsecure communication allowing the programmer to make fine-grained security/performance tradeoffs. We present performance results that quantify the performance of our infrastructure.