In our previous work, we described a resource allocation and scheduling technique for processing an open stream of MapReduce jobs with SLAs (characterized by an earliest start time, an execution time, and a deadline) called the Hadoop Constraint Programming based Resource Management technique (HCP-RM). Since the user-estimated job execution times are used to perform resource allocation and scheduling, error/inaccuracies in the execution times can hinder the ability of HCP-RM from making effective scheduling decisions, leading to a degradation in system performance. This paper focuses on improving the robustness of HCP-RM by introducing a mechanism to handle errors/inaccuracies in user estimates of job execution times that are submitted as part of the job's SLA. A Prescheduling Error Handling technique (PSEH) is devised to adjust the user-estimated execution times of the jobs to make them more accurate before they are used by the resource management algorithm. Results of experiments conducted on a Hadoop cluster deployed on Amazon EC2 demonstrate the effectiveness of the PSEH technique in improving system performance.

Resource allocation and scheduling on clouds are required to harness the power of the underlying resource pool such that the service provider can meet the quality of service requirements of users, which are often captured in service level agreements (SLAs). This paper focuses on resource allocation and scheduling on clouds and clusters that process MapReduce jobs with SLAs. The resource allocation and scheduling problem is modelled as an optimization problem using constraint programming, and a novel MapReduce Constraint Programming based Resource Management algorithm (MRCP-RM) is devised that can effectively process an open stream of MapReduce jobs where each job is characterized by an SLA comprising an earliest start time, a required execution time, and an end-to-end deadline. A detailed performance evaluation of MRCP-RM is conducted for an open system subjected to a stream of job arrivals using both simulation and experimentation on a real system. The experiments on a real system are performed on a Hadoop cluster (deployed on Amazon EC2) that runs our new Hadoop Constraint Programming based Resource Management algorithm (HCP-RM) that incorporates a technique for handling data locality. The results of the performance evaluation demonstrate the effectiveness of MRCP-RM/HCP-RM in generating a schedule that leads to a low proportion of jobs missing their deadlines (P) and also provide insights into system behaviour and performance. In the simulation experiments, it is observed that MRCP-RM achieves on average an 82 percent lower P compared to a technique from the existing literature when processing a synthetic workload from Facebook. Furthermore, in the experiments performed on a Hadoop cluster deployed on Amazon EC2, it is observed that HCP-RM achieved on average a 63 percent lower P compared to an EDF-Scheduler for a wide variety of workload and system parameters experimented with.

A novel MapReduce constraint programming based matchmaking and scheduling algorithm (MRCP) that can handle MapReduce jobs with deadlines and achieve high system performance is devised. The MRCP algorithm is incorporated into Hadoop, which is a widely used open source implementation of the MapReduce programming model, as a new scheduler called the CP-Scheduler. This paper originates from the collaborative research with our industrial partner concerning the engineering of resource management middleware for high performance. It describes our experiences and the challenges that we encountered in designing and implementing the prototype CP-based Hadoop scheduler. A detailed performance evaluation of the CP-Scheduler is conducted on Amazon EC2 to determine the CP-Scheduler's effectiveness as well as to obtain insights into system behaviour and performance. In addition, the CP-Scheduler's performance is also compared with an earliest deadline first (EDF) Hadoop scheduler, which is implemented by extending Hadoop's default FIFO scheduler. The experimental results demonstrate the effectiveness of the CP-Scheduler's ability to handle an open stream of MapReduce jobs with deadlines in a Hadoop cluster.

Clouds that are rapidly gaining in popularity require an effective resource manager that can harness the power of the underlying resource pool, and provide resources on demand to its users. This paper focuses on resource management on clouds for workflow requests characterized by Service Level Agreements (SLAs). Specifically, we devise a novel MapReduce constraint programming based resource manager (MRCP-RM) that can effectively perform matchmaking and scheduling of MapReduce jobs, each characterized by an SLA comprising an earliest start time, execution time, and an end-to-end deadline. Using discrete event simulation a performance evaluation of MRCP-RM is conducted for an open system subjected to a stream of job arrivals. The simulation results demonstrate the effectiveness of the resource manager and provide insights into system behaviour and performance.

This paper focuses on devising efficient resource management techniques used by the resource management middleware in clouds that handle MapReduce jobs with end-to-end service level agreements (SLAs) comprising an earliest start time, execution time, and a deadline. This research and development work, performed in collaboration with our industrial partner, presents the formulation of the matchmaking and scheduling problem for MapReduce jobs as an optimization problem using: Mixed Integer Linear Programming (MILP) and Constraint Programming (CP) techniques. In addition to the formulations devised, our experience in implementing the MILP and CP models using various open source as well as commercial software packages is described. Furthermore, a performance evaluation of the different approaches used to implement the formulations is conducted using a variety of different workloads.

Sending documents containing sensitive information is commonly performed on distributed systems. To protect the sensitive data in the documents, these documents need to be transmitted over a secure channel. However, secure communications can often increase the data transmission time due to the CPU-intensive operations (e.g. encryption/decryption) that need to be applied to the data before the data transfer. This paper focuses on a performance optimization technique called security sieve, which aims to enhance the performance of SSL/TLS-based data transmission. Security sieve reduces the transmission time of transferring classified documents, which may also contain non-sensitive data, by minimizing the time spent performing the CPU-intensive security-related operations. This paper focuses on highlighting the performance optimization principles/performance patterns that were used and includes a discussion of a prototype implementation. In addition, experimental results are presented to demonstrate the performance enhancement provided by security sieve.

Security in a distributed system often comes at the cost of a performance penalty. Due to the CPU time consuming security algorithms used, transferring data using SSL is known to be significantly slow. This paper presents an initial set of research results of a university-industry collaborative research focusing on a performance enhancement technique called security sieve that separates the classified and non-classified components in a document and sends these on a secure and a (faster) non-secure channel respectively. Experimental results presented in the paper demonstrate the effectiveness of the technique.