In this work, we present a set of techniques that considerably improve the performance of executing concurrent MapReduce jobs. Our proposed solution relies on proper resource allocation for concurrent Hive jobs based on data dependency, inter-query optimization and modeling of Hadoop cluster load. To the best of our knowledge, this is the first work towards Hive/MapReduce job optimization which takes Hadoop cluster load into consideration. We perform an experimental study that demonstrates 233% reduction in execution time for concurrent vs sequential execution schema. We report up to 40% extra reduction in execution time for concurrent job execution after resource usage optimization. The results reported in this paper were obtained in a pilot project to assess the feasibility of migrating A/B testing from Teradata + SAS analytics infrastructure to Hadoop. This work was performed on eBay production Hadoop cluster.

A major challenge in large-scale process monitoring is to recognize significant transitions in the process conditions and to distinguish them from random fluctuations that do not produce a notable change in the process dynamics. Such transitions should be recognized at the early stages of their development using a minimal "snapshot" of the observable process log. We developed a novel approach to detect notable transitions based on analysis of coherent behavior of frequency components in the process log (coherency portraits). We have found that notable transitions in the process dynamics are characterized by unique coherency portraits, which are also invariant with respect to random process fluctuations. Our experimental study demonstrates significant efficiency of our approach as compared to traditional change detection techniques.

We introduce a novel strategy for data processing in Wireless Sensor Networks (WSNs) in the case of emergency fire evacuation with stringent delay constraints. Such networks should perform distributed emergency assessment, continuous emergency monitoring, and dynamic selection of optimal evacuation strategies. The obvious complexity of these tasks restricts applying existing WSN optimization solutions. Our approach is based on considering the WSN as a complex adaptive system where a complicated task of multi-agent scheduling is factored into a set of smaller subtasks. In this case, decisions made locally by individual sensors can efficiently converge into desirable information processing patterns. A notable feature of our method is its scalability, which allows the sensornet to operate with sufficient quality of service under heavy information loads. We demonstrate the utility of our approach using different fire evacuation scenarios.