Traffic prediction, particularly in urban regions, is an important application of tremendous practical value. In this paper, we report a novel and interesting case study of urban traffic prediction in Central, Hong Kong, one of the densest urban areas in the world. The novelty of our study is that we make good second use of inexpensive big data collected from the Hong Kong International Commerce Centre (ICC), a 118-story building in Hong Kong where more than 10,000 people work. As building environment data are much cheaper to obtain than traffic data, we demonstrate that it is highly effective to estimate building occupancy information using building environment data, and then to further use the information on occupancy to provide traffic predictions in the proximate area. Scientifically, we investigate how and to what extent building data can complement traffic data in predicting traffic. In general, this study sheds new light on the development of accurate data mining applications through the second use of inexpensive big data.

Recently, the cloud systems face an increasing number of big data applications. It becomes an important issue for the cloud providers to allocate resources so as to accommodate as many of these big data applications as possible. In current cloud service, e.g., Amazon EMR, a job runs on a fixed cluster. This means that a fixed amount of resources (e.g. CPU, memory) is allocated to the life cycle of this job. We observe that the resources are inefficiently used in such services because of resources usage unbalance. Therefore, we propose a runtime elastic VM approach where the cloud system can increase or decrease the number of CPUs at different time periods for the jobs. There is little change to such services as Amazon EMR, yet the cloud system can accommodate many more jobs. In this paper, we first present a measurement study to show the feasibility and the quantitative impact of adjusting VM configurations dynamically. We then model the task and job completion time of big data applications, which are used for elastic VM adjustment decisions. We validate our models through experiments. We present Tetris, an elastic VM strategy based on cloud system that can better optimize resource utilization to support big data applications. We further implement a Tetris prototype and comprehensively evaluate Tetris on a real private cloud platform using Facebook trace and Wikipedia dataset. We observe that with Tetris, the cloud system can accommodate 31.3% more jobs.

MapReduce-like frameworks have achieved tremendous success for large-scale data processing in data centers. A key feature distinguishing MapReduce from previous parallel models is that it interleaves parallel and sequential computation. Past schemes, and especially their theoretical bounds, on general parallel models are therefore, unlikely to be applied to MapReduce directly. There are many recent studies on MapReduce job and task scheduling. These studies assume that the servers are assigned in advance. In current data centers, multiple MapReduce jobs of different importance levels run together. In this paper, we investigate a schedule problem for MapReduce taking server assignment into consideration as well. We formulate a MapReduce server-job organizer problem (MSJO) and show that it is NP-complete. We develop a 3-approximation algorithm and a fast heuristic design. Moreover, we further propose a novel fine-grained practical algorithm for general MapReduce-like task scheduling problem. Finally, we evaluate our algorithms through both simulations and experiments on Amazon EC2 with an implementation with Hadoop. The results confirm the superiority of our algorithms.

MapReduce has achieved tremendous success for large-scale data processing in data centers. A key feature distinguishing MapReduce from previous parallel models is that it interleaves parallel and sequential computation. Past schemes, and especially their theoretical bounds, on general parallel models are therefore, unlikely to be applied to MapReduce directly. There are many recent studies on MapReduce job and task scheduling. These studies assume that the servers are assigned in advance. In current data centers, multiple MapReduce jobs of different importance levels run together. In this paper, we investigate a schedule problem for MapReduce taking server assignment into consideration as well. We formulate a MapReduce server-job organizer problem (MSJO) and show that it is NP-complete. We develop a 3-approximation algorithm and a fast heuristic. We evaluate our algorithms through both simulations and experiments on Amazon EC2 with an implementation in Hadoop. The results confirm the advantage of our algorithms.

Recent advances in cloud-based big data analysis offers a convenient mean for providing an elastic and cost-efficient exploration of voluminous data sets. Following such a trend, industry leaders as Amazon, Google and IBM deploy various of big data systems on their cloud platforms, aiming to occupy the huge market around the globe. While these cloud systems greatly facilitate the implementation of big data analysis, their real-world applicability remains largely unclear. In this paper, we take the first steps towards a better understanding of the big data system on the cloud platforms. Using the typical MapReduce framework as a case study, we find that its pipeline-based design intergrades the computational-intensive operations (such as mapping/reducing) together with the I/O-intensive operations (such as shuffling). Such computational-intensive and I/O-intensive operations will seriously affect the performance of each other and largely reduces the system efficiency especially on the low-end virtual machines (VMs). To make the matter worse, our measurement also indicates that more than 90 % of the task-lifetime is in the shadow of such interference. This unavoidably reduces the applicability of cloud-based big data processing and makes the overall performance hard to predict. To address this problem, we re-model the resource provisioning problem in the cloud-based big data systems and present an interference-aware solution that smartly allocates the MapReduce jobs to different VMs. Our evaluation result shows that our new model can accurately predict the job completion time across different configurations and significantly improve the user experience for this new generation of data processing service.

We are in an age where people are paying increasing attention to energy conservation around the world. The heating and air-conditioning systems of buildings introduce one of the largest chunks of energy expenses. In this article, we make a key observation that after a meeting or a class ends in a room, the indoor temperature will not immediately increase to the outdoor temperature. We call this phenomenon thermal inertia . Thus, if we arrange subsequent meetings in the same room rather than in a room that has not been used for some time, we can take advantage of such undissipated cool or heated air and conserve energy. Though many existing energy conservation solutions for buildings can intelligently turn off facilities when people are absent, we believe that understanding thermal inertia can lead system designs to go beyond on-and-off-based solutions to a wider realm. We propose a framework for exploring thermal inertia in room management. Our framework contains two components. (1) The energy-temperature correlation model captures the relation between indoor temperature change and energy consumption. (2) The energy-aware scheduling algorithms: given information for the relation between energy and temperature change, energy-aware scheduling algorithms arrange meetings not only based on common restrictions, such as meeting time and room capacity requirement, but also energy consumptions. We identify the interface between these components so further works towards same on direction can make efforts on individual components. We develop a system to verify our framework. First, it has a wireless sensor network to collect indoor, outdoor temperature and electricity expenses of the heating or air-conditioning devices. Second, we build an energy-temperature correlation model for the energy expenses and the corresponding room temperature. Third, we develop room scheduling algorithms. In detail, we first extend the current sensor hardware so that it can record the electricity expenses in re-heating or re-cooling a room. As the sensor network needs to work unattendedly, we develop a hardware board for long-range communications so that the Imote2 can send data to a remote server without a computer relay close by. An efficient two-tiered sensor network is developed with our extended Imote2 and TelosB sensors. We apply laws of thermodynamics and build a correlation model of the energy needed to re-cool a room to a target temperature. Such model requires parameter calibration and uses the data collected from the sensor network for model refinement. Armed with the energy-temperature correlation model, we develop an optimal algorithm for a specified case, and we further develop two fast heuristics for different practical scenarios. Our demo system is validated with real deployment of a sensor network for data collection and thermodynamics model calibration. We conduct a comprehensive evaluation with synthetic room and meeting configurations, as well as real class schedules and classroom topologies of The Hong Kong Polytechnic University, academic calendar year of Spring 2011. We observe 20% energy savings as compared with the current schedules.

We are in an age where people are paying increasing attention to energy conservation around the world. The heating and air-conditioning systems of buildings introduce one of the largest chunk of energy expenses. In this paper, we make a key observation that after a meeting or a class ends in a room, the indoor temperature will not immediately increase to the outdoor temperature. We call this phenomenon Thermal Inertia. Thus, if we arrange subsequent meetings in the same room; than a room that has not been used for some time, we can take advantage of such un-dissipated cool or heated air and conserve energy. We develop a green room management system with three main components. First, it has a wireless sensor network to collect indoor, outdoor temperature and electricity expenses of the air-conditioning devices. Second, we build an energy-temperature correlation model for the energy expenses and the corresponding room temperature. Third, we develop room scheduling algorithms. Our system is validated with real deployment of a sensor network for data collection and thermodynamics model calibration. We conduct a comprehensive evaluation with synthetic room and meeting configurations. We observe a 30% energy saving as compared with the current schedules.