Burst buffers tolerate I/O spikes in High-Performance Computing environments by using a non-volatile flash technology. Burst buffers are commonly located between parallel file systems and compute nodes, handling bursty I/Os in the middle. In this architecture, burst buffers are shared resources. The performance of an SSD is significantly reduced when it is used excessively because of garbage collection, and we have observed that SSDs in a burst buffer become slow when many users simultaneously use the burst buffer. To mitigate the performance problem, we propose a new user-level I/O isolation framework in a High-Performance Computing environment using a multi-streamed SSD. The multi-streamed SSD allocates the same flash block for I/Os in the same stream. We assign a different stream to each user; thus, the user can use the stream exclusively. To evaluate the performance, we have used open-source supercomputing workloads and I/O traces from real workloads in the Cori supercomputer at the National Energy Research Scientific Computing Center. Via user-level I/O isolation, we have obtained up to a 125% performance improvement in terms of I/O throughput. In addition, our approach reduces the write amplification in the SSDs, leading to improved SSD endurance. This user-level I/O isolation framework could be applied to deployed burst buffers without having to make any user interface changes.

Genome sequence alignment is one of the basic procedure of genome sequencing analysis pipeline and also one of the most time-consuming parts. Including BigBWA, a number of tools were proposed to accelerate genome sequence alignment by parallelizing computation with Hadoop technologies. However, HDFS incurs considerable I/O overhead. In this research, we propose a new sequence alignment tool adopting Hadoop on Lustre. Based on BigBWA, we removed data transfer overhead caused by HDFS and parallelized whole I/O steps. Experimental result shows that our solution is five times faster than original BigBWA in a ten-node Lustre based Hadoop cluster.

Summary In this paper, we conduct a comparative study of relatively simple yet effective scheduling policies for many‐task applications where multiple users with varying numbers of tasks are actively sharing a common system infrastructure. We have implemented three different scheduling mechanisms that can address fairness and user response time respectively in a common middleware stack called HTCaaS, which is a pilot‐job–based multi‐level scheduling system running on top of production‐level clusters. As a representative case of our many‐task applications, we have leveraged the virtual screening application, which is a computational technique used in drug discovery process to select the most promising candidate drugs for in vitro testing from millions of chemical compounds, which typically requires a substantial amount of computing resources and efficient processing of docking simulations. Our comparative experimental results of different scheduling policies show how we can effectively support multiple users in a shared resource environment by balancing between user satisfaction and overall system performance, provide guidelines to improve system utilization, and address additional technical challenges to support various many‐task applications.

Many-Task Computing (MTC) has been a new computing paradigm that aims to bridge the gap between traditional High-Throughput Computing (HTC) and High-Performance Computing (HPC). MTC applications from various scientific domains such as pharmaceuticals, astronomy, physics often consist of a very large number (from thousands to even billions) of data-intensive (tens of MB of I/O per second) tasks with relatively short per task execution times (from seconds to minutes). Each task in MTC applications may require relatively small amount of data processing especially compared to existing Big Data applications typically based on larger data block sizes (e.g. the default block size in Hadoop is 64MB). However, they can consist of much larger numbers of tasks where each task communicates through files instead of message passing interfaces such as MPI in HPC applications. Therefore, MTC can be another type of data-intensive workload where a large number of data processing tasks should be efficiently processed within a relatively short period of time. In this paper, we present design and implementation of MOHA (Many-task computing On HAdoop) which can make an effective convergence of MTC technologies and the existing Big Data platform Hadoop. MOHA is developed as a Hadoop YARN application so that it can transparently co-host existing MTC applications with other Big Data processing frameworks such as MapReduce in a single Hadoop cluster. Our evaluation results based on microbenchmark show that MOHA can substantially reduce the overall execution time of many-task processing with minimal amount of resources compared to an existing Hadoop YARN application. In addition, MOHA can efficiently dispatch a large number of tasks which can be crucial to support challenging MTC applications. MOHA can bring many interesting research issues related to data grouping and declustering on Hadoop Distributed File System (HDFS), scalable job/metadata management, dynamic task load balancing which can ultimately contribute to a new data processing framework in the YARN based Hadoop 2.0 ecosystem.

High-Throughput Computing (HTC) and Many-Task Computing (MTC) paradigms employ loosely coupled applications which consist of a large number, from tens of thousands to even billions, of independent tasks. To support such large-scale applications, a heterogeneous computing system composed of multiple computing platforms with different types such as supercomputers, grids, and clouds can be used. On allocating heterogeneous resources of the system to multiple users, there are three important aspects to consider: fairness among users, efficiency for maximizing the system throughput, and user satisfaction for reducing the average user response time. In this paper, we present three resource allocation policies for multi-user and multi-application workloads in a heterogeneous computing system. These three policies are a fairness policy, a greedy efficiency policy, and a fair efficiency policy. We evaluate and compare the performance of the three resource allocation policies over various settings of a heterogeneous computing system and loosely coupled applications, using simulation based on the trace from real experiments. Our simulation results show that the fair efficiency policy can provide competitive efficiency, with a balanced level of fairness and user satisfaction, compared to the other two resource allocation policies.

Recent emerging applications from a wide range of scientific domains often require a very large number of loosely coupled tasks to be efficiently processed. To support such applications effectively, all the available resources from different types of computing platforms such as supercomputers, grids, and clouds need to be utilized. However, exploiting heterogeneous resources from the platforms for multiple loosely coupled many-task applications is challenging, since the performance of an application can vary significantly depending on which platform is used to run it, and which applications co-run in the same node with it. In this paper, we analyze the platform and co-runner affinities of many-task applications in distributed computing platforms. We perform a comprehensive experimental study using four different platforms, and five many-task applications. We then present a two-level scheduling algorithm, which distributes the resources of different platforms to each application based on the platform affinity in the first level, and maps tasks of the applications to computing nodes based on the co-runner affinity for each platform in the second level. Finally, we evaluate the performance of our scheduling algorithm, using a trace-based simulator. Our simulation results demonstrate that our scheduling algorithm can improve the performance up to 30.0%, compared to a baseline scheduling algorithm.

Traditional High-Throughput Computing (HTC) consists of running many loosely-coupled tasks that are independent but requires a large amount of computing power during significant period of time. However, recent emerging applications requiring millions or even billions of tasks to be processed within a relatively short period of time have expanded the traditional HTC into Many-Task Computing (MTC).In silico drug discovery offers an efficient alternative to reduce the cost of drug development and discovery process. For this purpose, virtual screening is used to select the most promising candidate drugs for in vitro testing from millions of chemical compounds. This process requires a substantial amount of computing resources and high-performance processing of docking simulations, which shows the typical characteristics of MTC applications. As the number of users performing this virtual screening process increases with limited available computing resources, it becomes crucial to devise an effective scheduling policy that can ensure a certain degree of fairness and user satisfaction. In this paper, we present a comparative analysis of scheduling mechanisms for the virtual screening workflow where multiple users in the system are sharing a common service infrastructure. To effectively support these multiple users, the underlying system should be able to consider fairness, user response time and overall system throughput. We have implemented two different scheduling algorithms which can address fairness and user response time respectively in a common middleware stack call edHTCaaS which is a pilot-job based multi-level scheduling system running on top of a dedicated production-level cluster. Throughout comparative analysis of two different scheduling mechanisms targeting different metrics on top of a single H/W and S/W system, we can give an insight to the research community on the design and implementation of a scheduling mechanism that can trade-off user fairness and overall system performance which's crucial to support challenging MTC applications.

A lot of image data has been digitized and preserved in computers. In order to search for an image, an efficient and accurate retrieval method is needed. This paper is concerned with shape retrieval which is one of the searching methods for finding similar images in a database. Shape, the outer form of a picture, is considered the most promising feature to identify entities in an image. The problem of shape retrieval takes a lot of time because an exhaustive search is mainly used in literatures. This paper suggests the use of a clustering method known as dendrogram for shape retrieval. In addition, we proposed the automatic decision of a threshold to determine a number of clusters in dendrogram. Through the experimental result, the proposed method proved fast retrieval preserving almost the same level accuracy.

Abstract This paper addresses performance issues on ARDA Metadata Grid Application (AMGA) and presents new techniques to improve the throughput of AMGA for the WISDOM environment. The first issue is a performance degradation problem when AMGA is used as a metadata service for task retrieval in the WISDOM environment. To deal with the issue, a new AMGA operation designed to reduce the communication overhead required to retrieve a task from AMGA is proposed. According to a performance study conducted with the new operation, the throughput of task retrieval using the proposed operation can be as much as 70 times higher than the throughput when using the existing AMGA operations. The second issue is an AMGA throughput issue in large‐scale grid‐enabled applications such as WISDOM, where it is not uncommon that thousands of jobs running on grid nodes access the AGMA service simultaneously. To address this issue, integration of a load‐balancing technique and a DB connection pool technique into the AMGA are proposed. Test results demonstrate that the performance can be improved linearly in proportion to the number of AMGA servers set up for load balancing; the performance improvement continues until the performance limit of the backend database system is reached. Copyright © 2009 John Wiley & Sons, Ltd.

The coming of the e-science era provides more opportunities to improve the work of scientists by integrating sensors, resources, and services in GRID and Internet environments. We can expect much more improvement in scientists' work if they are supported by an appropriate tool. In this paper, we propose an e-science workbench that helps scientists design and perform their e-science experiments on the GRID or internet environments without the support of programmers. Since the e-science workbench may contain many modules and functions, we adopt a layered architecture in constructing the system. In order to convince potential users of the effectiveness of using the e-science workbench, we developed two domain specific e-Science workbenches. One is a health workbench and the other is a bio workbench. We observed that we could build the u-health and bio workbenches very systematically due to the underlying middle and bottom e-science workbench layers.

DrugScreener-G is an integrated environment for virtual screening with grid computing. It implements basic ideas of grid-enabled large-scale virtual screening into a concrete software suite, especially emphasizing user friendliness and accessibility to grid computing. DrugScreener-G aims at providing users without knowledge of grid computing with an intuitive and easy-to-use integrated environment for grid-enabled large-scale virtual screening. We discuss architecture and design of DrugScreener-G in two aspects, extension and integration of different functions and virtual screening methods with plug-ins and application repository, and virtualization of grid computing services under Web service. DrugScreener-G is easily extensible with new virtual screening methods as it is developed in plug-in architecture. DrugScreener-G virtualizes different grid computing services behind Web service interface to hide technical details of grid computing from users and to provide interoperability among different types of grid computing resources and middleware. It is also discussed how application-level interoperability in DrugScreener-G can be a simple and effective alternative to the recent approaches to grid interoperability, such as SAGA and GIN, which are not yet agreed standards on grid interoperability.

This paper describes a workflow-based problem solving environment (PSE) for supporting collaborative aerospace engineering process on the grid. The focus of our work is on the design of a workflow system that supports computational activities that are to be executed on the grid and manual activities that are to be performed by humans. According to our investigation on the aerospace engineering research process through interviews with aerospace researchers and engineers, it is essential to incorporate the grid-enabled execution of large-scale simulation runs and the support for manual activities into the workflow system. Most existing scientific workflow systems are aware of the importance of the support for the execution of applications on the grid, but not concerned with the support for manual activities. On the other hand, most existing business workflow systems are designed to have human intervention in mind, but not concerned with the grid-enabled execution. Based on the analysis of the e-AIRS system which is about a collaborative PSE for aerospace engineering, we have identified a collaborative research scenario and requirements. As a proof of concept system, we have prototyped a grid-enabled workflow system that realizes the scenario and meets the requirements.

In ubiquitous network, node location information is the most essential data for efficiently managing increased mobility and supporting location based services. Therefore, we propose a next generation location information based network, which uses address containing location information. Finally, its functions, effectiveness, and feasibility will be verified through this demonstration.

In e-science environments, scientists need to execute a scientific application with various parameters multiple times to simulate and experiment complicated problems on the grid. For this, they should write every single job description with distinct parameters even if this is a troublesome task. To provide the flexibility and adaptability for parameter study, we propose an application parameter description language (APDL) and a service oriented parameter study scheme, called a parametric study service (PSS), for parameterized simulations on the grid. The APDL extends the job submission description language (JSDL) to generate parameters for multiple jobs. The proposed PSS provides a unified interface to submit jobs into various middleware platforms such as gLite, Globus, etc. The problem solving environment (PSE) assists a parameter study for their applications and every research fields tend to construct individual own PSE., The proposed PSS can be easily adapted into the specific PSE because of being implemented as Web services. In practice, we apply the APDL and the PSS into aerospace research PSE which carry out the three-dimensional turbulent analysis for compressible flow.

The generic, heterogeneous, and dynamic nature of the grid requires a new from of failure recovery mechanism to address its unique requirements such as support for diverse failure handling strategies, separation of failure handling strategies from application codes, and user-defined exception handling. We here propose a grid workflow system (grid-WFS), a flexible failure handling framework for the grid, which addresses these grid-unique failure recovery requirements. Central to the framework is flexibility by the use of workflow structure as a high-level recovery policy specification. We show how this use of high-level workflow structure allows users to achieve failure recovery in a variety of ways depending on the requirements and constraints of their applications. We also demonstrate that this use of workflow structure enables users to not only rapidly prototype and investigate failure handling strategies, but also easily change them by simply modifying the encompassing workflow structure, while the application code remains intact. Finally, we present an experimental evaluation of our framework using a simulation, demonstrating the value of supporting multiple failure recovery techniques in grid systems to achieve high performance in the presence of failures.