Achieving low and predictable execution time of short jobs in Hadoop clusters has gained a great attention due to their importance on system productivity and user experience. However, one major contributor that makes it challenging is diskI/O interference. We observed that disk writes unintentionally block latency-sensitive short jobs and cause unexpected high latency. Unfortunately, previous research including a disk read bandwidth throttling do not suffice to mitigate such interference. This paper proposes the application-assisted writeback that allows the Hadoop framework to control asynchronous writebacks. We applied the application-assisted writeback to optimize short jobs by preventing asynchronous writebacks when they are expected to interfere with short jobs. Our evaluation resultedin reduction on the average and 99-th percentile execution time of short jobs by 22% and 40%, respectively, without imposing non-acceptable overheads on co-running throughput-oriented batch jobs. In addition, combining the application-assisted writeback with the user-level disk bandwidth throttling can further accelerate short jobs.

With the proliferation of cloud computing, security concerns about confidentiality violations of user data by the privileged domain and system administrators have been growing. This paper proposes secure cloud architecture with a hardware security module, which isolates cloud user data from potentially malicious privileged domains or cloud administrators. Within a securely isolated execution environment, the hardware security module provides essential security functionality with only restricted interfaces exposed to vulnerable management systems or cloud administrators. Such restriction prevents cloud administrators from affecting the security of guest VMs. The proposed architecture not only defends against wide attack vectors but also achieves a small TCB. This paper discusses our hardware and software implementation of the proposed cloud architecture, analyzes its security, and presents its performance results.

Cloud computing based on system virtualization, has been expanding its services to distributed data-intensive platforms such as MapReduce and Hadoop. Such a distributed platform on clouds runs in a virtual cluster consisting of a number of virtual machines. In the virtual cluster, demands on computing resources for each node may fluctuate, due to data locality and task behavior. However, current cloud services use a static cluster configuration, fixing or manually adjusting the computing capability of each virtual machine (VM). The fixed homogeneous VM configuration may not adapt to changing resource demands in individual nodes.

Single system image (SSl) systems have been the mainstay of high-performance computing for many years. SSI requires the integration and aggregation of all types of resources in a cluster to present a single interface to users. We describe a cluster computing environment supporting SSI, constructed through three components: single process space (SPS), process migration, and dynamic load balancing. These components attempt to share all available resources in the cluster among all executing processes, so that the cluster operates like a single node with much more computing power. The most important goal is to combine these constructs in innovative ways for building cluster computing environment for SSI, as well as individually take an approach to improve performance or functionality. Our implementation of process migration has the capability of resolving broken pipe problems and bind errors on server socket reconstruction. We realize SPS based on block PID allocation. We also designed and implemented a dynamic load balancing scheme which resolves the limitations of our previous work by continuously tracing the job resource usage at runtime. The experimental results show that these three constructs for SSI clusters realized scalability, functionality and performance improvement. The cluster computing environment allows these constructs to cooperate implicitly so that they create a synergy effect at the SSI cluster system level and successfully provide a single system image to users and administrators.