MapReduce, proposed as a programming model, has been widely adopted in the field of text processing over large datasets with the capability of exploiting the distributed resources and processing the large-scale data. Attributed to its simplicity and scalability, the success seems to have the potential to make Big Data processing by cloud computing available. Nevertheless, such promise is accompanied by the difficulty of fitting the applications into MapReduce. This is because MapReduce is limited to the kind of applications that every input key-value pair is independent of each other. In this paper, we extend the general applicability of MapReduce by allowing the dependence within a set of input key-value pairs but preserving independence among all sets. Such this new modeling paradigm intends MapReduce to shift processing the independent input key-value pairs to processing the independent sets. However, the advancement in the applicability brings the intricate problem of how two-stage processing structure, inherent in MapReduce, handles the dependence within a set of input key-value pairs. To tackle this problem, we propose the design pattern called two-phase data processing. It expresses the application in two phases not only to match the two-stage processing structure but to exploit the power of MapReduce through the cooperation between the mappers and reducers. In addition, we present the design methodology-multiple two-phase data processing-to offer advice on processing the independent sets. The experiment of background subtraction, a part of video surveillance, proves that the new modeling paradigm broadens the possibilities of MapReduce and demonstrates how our design methodology guides the applications to the implementation.

In this paper, we present a distributed file system with storage QoS provision for cloud computing. The objective is to maximize the aggregated disk bandwidth utilization with the assurance of allocated disk bandwidth for each data transfer. First, we implement a distributed file system based on the ECNP model, which avoids matchmaker overloading and the lack of up-to-date information in the original matchmaking model. Next, resource selection policies for resource management are investigated to improve bandwidth utilization. Furthermore, we propose several dynamic replication strategies to balance the workload of resource providers. Experiments are conducted in real systems to study the performance on storage QoS with different resource selection policies and dynamic replication strategies.

In this paper, we present the design and implementation of a negotiation-based resource management system called Mercury. As the number of resources in Grids is increasing rapidly, selecting appropriate resources to execute tasks has become a crucial issue. The proposed system implements the Extended Contract Net Protocol (ECNP), which improves the standard bidding model by integrating a matchmaking technique. Our model addresses the issues of matchmaker overload and the lack of up-to-date resource state information in the original matchmaking model. To ensure that the system is user-friendly, we provide a web interface. By using the provided job templates, users can describe their jobs more easily, and service providers can deploy their services in the system in a simple manner. Furthermore, to improve the interoperability of the system, we adopt open standards, such as the WS-Agreement and JSDL protocols. Our experimental results demonstrate the scalability and efficiency of the system.

In this paper, we present a fast cycle-accurate instruction set simulator (CA-ISS) for system-on-chip development based on QEMU and SystemC. Even though most state-of-the-art commercial tools have tried very hard to provide all the levels of details to satisfy the different requirements of the software designer, the hardware designer, and even the system architect, the hardware/software co-simulation speed is dramatically slow when co-simulating the hardware models at the register-transfer level (RTL) with a full-fledged operating system (OS). Our experimental results show that the combination of QEMU and SystemC can make the co-simulation at the CA level much faster than the conventional RTL simulation, even with a full-fledged operating system up and running. Furthermore, the statistics indicate that with every instruction executed and every memory accessed since power-on traced at the CA level, it takes 28m15.804s on average to boot up a full-fledged Linux kernel, even on a personal computer. Compared to the kernel boot time reported by Xilinx and SiCortex, the proposed CA-ISS is about 6.09 times faster compared to “SystemC without trace” of Xilinx and about 30.32 times faster compared to “SystemC models converted from RTL” of SiCortex. The main contributions of this paper are threefold: 1) a hardware/software co-simulation environment capable of running a full-fledged OS at the early stage of the electronic system level design flow at an acceptable simulation speed is proposed; 2) a virtual platform constructed using the proposed CA-ISS as the processor model can be used to estimate the performance of a target system from system perspective, which all the previous works, such as QEMU-SystemC, do not provide; and 3) such a virtual platform also provides the modeling capability from the transaction level down to the CA level or the other way around.