In the Internet of Things (IoT) era, with ubiquitous remote sensing devices and other diverse data sources, nearly everything can forward voluminous data continuously, in real-time, which drives demand to perform real-time analytics on uninterrupted IoT data flows. The typical resultant approach is a cloud-centered architecture providing an analytic service for real-time IoT data processing. However, a cloud-centered IoT analytic service cannot guarantee real-time responsiveness has a high-fee pay-as-you-go business model, and opens data privacy concerns. Hence, it becomes rational to shift analytic workloads to the edge and provide a management service for edge analysis. Existing work on providing edge analytics as a service encountered challenges such as lacking a lightweight way to compose IoT applications based on multiple service providers, lacking a flexible and unified way to define domain-specific analytic logic, and maintaining efficiency when processing data on a resource-limited edge. This paper presents EAaaS, a scalable analytic service for enabling real-time edge analytics in IoT scenarios. In this work, we propose a unified rule-based analytic model to ease user's programming efforts in specifying rule-based analytic logic. Moreover, we also designed and implemented a high performance edge engine to apply rule-based analytic on incoming device data streams. To simplify the access to EAaaS service, a group of RESTful web interfaces is also designed for edge analytic management on cloud and flexible composition with external services. EAaaS is implemented as a part of IBM Watson IoT Platform, which is a cloud service for elementary IoT application development on IBM Bluemix cloud announced by IBM recently. We have conducted proof of correctness (PoC) of EAaaS with customers from boat racing in the U.S. and collected valuable feedback from customers for further enhancement of EAaaS's flexibility and usability.

Nowadays vast amounts of data are being produced in continuous ways. They may come from sensors, smart meters, application logs, monitoring software etc. The data need to be processed in realtime to gain actionable insights. Services like smart grid load balancing, cloud platform maintenance, can be carried out in an efficient way. Stream processing is the programming paradigm that answers such demand. When talking about stream processing, we can easily recall several famous open-source software frameworks such as Spark Streaming, Samza, Flink and Storm. Although they provide distributed, robust, low-latency stream processing engines, it's still difficult for an end user to set up a usable stream processing application from scratch. Firstly, users are required to write code to define their business related stream processing logic. Secondly, the submission and update of the stream processing logic require service restart, therefore it may lead to service unavailability for minutes. Thirdly, extra operation effort are required for handling scaling and failover issues. In this paper, we present RTA, a released research service on realtime data processing. The RTA service fills the gap between the stream processing requester and the existing software stacks. It offers a SQL-like stream query language for defining stream processing logic definition over streaming data. It allows users easily define their stream processing logic without programming. In RTA service, stream processing logic is also treated as a type of input, which enables online logic update without service downtime. The RTA service also provides scalability, high availability and resource isolation for serving multiple tenants. In this paper, we also provide a comprehensive evaluation of our service through a case study.

MapReduce and Spark are two very popular open source cluster computing frameworks for large scale data analytics. These frameworks hide the complexity of task parallelism and fault-tolerance, by exposing a simple programming API to users. In this paper, we evaluate the major architectural components in MapReduce and Spark frameworks including: shuffle, execution model, and caching, by using a set of important analytic workloads. To conduct a detailed analysis, we developed two profiling tools: (1) We correlate the task execution plan with the resource utilization for both MapReduce and Spark, and visually present this correlation; (2) We provide a break-down of the task execution time for in-depth analysis. Through detailed experiments, we quantify the performance differences between MapReduce and Spark. Furthermore, we attribute these performance differences to different components which are architected differently in the two frameworks. We further expose the source of these performance differences by using a set of micro-benchmark experiments. Overall, our experiments show that Spark is about 2.5x, 5x, and 5x faster than MapReduce, for Word Count, k-means, and PageRank, respectively. The main causes of these speedups are the efficiency of the hash-based aggregation component for combine, as well as reduced CPU and disk overheads due to RDD caching in Spark. An exception to this is the Sort workload, for which MapReduce is 2x faster than Spark. We show that MapReduce's execution model is more efficient for shuffling data than Spark, thus making Sort run faster on MapReduce.