The performance of distributed streaming systems relies on how even the workload is distributed among their machines. However, data and query workloads are skewed and change rapidly. Therefore, multiple adaptive load-balancing mechanisms have been proposed in the literature to rebalance distributed streaming systems according to the changes in their workloads. This paper introduces a novel attack model that targets adaptive load-balancing mechanisms of distributed streaming systems. The attack reduces the throughput and the availability of the system by making it stay in a continuous state of rebalancing. This paper proposesGuard, a component that detects and blocks attacks that target the adaptive load balancing of distributed streaming systems. Guard uses an unsupervised machine-learning technique to detect malicious users that are involved in the attack. Guard does not block any user unless it detects that the user is malicious. Guard does not depend on a specific application. Experimental evaluation for a high-intensity attack illustrates that Guard improves the throughput and the availability of the system by 85% and 86%, respectively. Moreover, Guard improves the minimum availability that the attacker achieves by 325%.

The wide spread of GPS-enabled devices and the Internet of Things (IoT) has increased the amount of spatial data being generated every second. The current scale of spatial data cannot be handled using centralized systems. This has led to the development of distributed spatial data streaming systems that scale to process in real-time large amounts of streamed spatial data. The performance of distributed streaming systems relies on how even the workload is distributed among their machines. However, it is challenging to estimate the workload of each machine because spatial data and query streams are skewed and rapidly change with time and users' interests. Moreover, a distributed spatial streaming system often does not maintain a global system workload state because it requires high network and processing overheads to be collected from the machines in the system.

Advances in real-world applications require high-throughput processing over large data streams. Micro-batching has been proposed to support the needs of these applications. In micro-batching, the processing and batching of the data are interleaved, where the incoming data tuples are first buffered as data blocks, and then are processed collectively using parallel function constructs (e.g., Map-Reduce). The size of a micro-batch is set to guarantee a certain response-time latency that is to conform to the application's service-level agreement. In contrast to tuple-at-a-time data stream processing, micro-batching has the potential to sustain higher data rates. However, existing micro-batch stream processing systems use basic data-partitioning techniques that do not account for data skew and variable data rates. Load-awareness is necessary to maintain performance and to enhance resource utilization. A new data partitioning scheme termed Prompt is presented that leverages the characteristics of the micro-batch processing model. In the batching phase, a frequency-aware buffering mechanism is introduced that progressively maintains run-time statistics, and provides online key-based sorting as data tuples arrive. Because achieving optimal data partitioning is NP-Hard in this context, a workload-aware greedy algorithm is introduced that partitions the buffered data tuples efficiently for the Map stage. In the processing phase, a load-aware distribution mechanism is presented that balances the size of the input to the Reduce stage without incurring inter-task communication overhead. Moreover, Prompt elastically adapts resource consumption according to workload changes. Experimental results using real and synthetic data sets demonstrate that Prompt is robust against fluctuations in data distribution and arrival rates. Furthermore, Prompt achieves up to 200% improvement in system throughput over state-of-the-art techniques without degradation in latency.

The widespread use of GPS-enabled smartphones along with the popularity of micro-blogging and social networking applications, e.g., Twitter and Facebook, has resulted in the generation of huge streams of geo-tagged textual data. Many applications require real-time processing of these streams. For example, location-based ad-targeting systems enable advertisers to register millions of ads to millions of users based on the users' location and textual profile. Existing streaming systems are either centralized or are not spatial-keyword aware, and hence these systems cannot efficiently support the processing of rapidly arriving spatial-keyword data streams. In this paper, we introduce a two-layered indexing scheme for the distributed processing of spatial-keyword data streams. We realize this indexing scheme in Tornado, a distributed spatial-keyword streaming system. The first layer, termed the routing layer, is used to fairly distribute the workload, and furthermore, co-locate the data objects and the corresponding queries at the same processing units. The routing layer uses the Augmented-Grid, a novel structure that is equipped with an efficient search algorithm for distributing the data objects and queries. The second layer, termed the evaluation layer, resides within each processing unit to reduce the processing overhead. The two-layered index adapts to changes in the workload by applying a cost formula that continuously represents the processing overhead at each processing unit. Extensive experimental evaluation using real Twitter data indicates that Tornado achieves high scalability and more than 2x improvement over the baseline approach in terms of the overall system throughput.

Several space agencies such as NASA are continuously collecting datasets of earth dynamics-e.g., temperature, vegetation, and cloud coverage-through satellites. This data is stored in a publicly available archive for scientists and researchers and is very useful for studying climate, desertification, and land use change. The benefit of this data comes from its richness as it provides an archived history for over 15 years of satellite observations. Unfortunately, the use of such data is very limited due to the huge size of archives (> 500TB) and the limited capabilities of traditional applications. In this demo, we present Shahed, an interactive system which provides an efficient way to index, query, and visualize satellite datasets available in NASA archive. Shahed is composed of four main modules. The uncertainty module resolves data uncertainty imposed by the satellites. The indexing module organizes the data in a novel multi-resolution spatio-temporal index designed for satellite data. The querying module uses the indexes to answer both spatiotemporal selection and aggregate queries provided by the user. The visualization module generates images, videos, and multi-level images which gives an insight of data distribution and dynamics over time. This demo gives users a hands-on experience with Shahed through a map-based web interface in which users can browse the available datasets using the map, issue spatiotemporal queries, and visualize the results as images or videos.

Frame fragmentation into small blocks with dedicated error detection codes per block can reduce the unnecessary retransmission of the correctly received blocks. However, the optimal block size varies based on the wireless channel conditions. Further, blocks within a single frame may have different optimal sizes based on variations in interference patterns. This paper proposes a hybrid interference-resilient frame fragmentation (Hi-Frag) link-layer scheme for wireless sensor networks. It effectively addresses the challenges associated with dynamic partitioning of blocks while accounting for the observed error patterns. Hi-Frag is the first work to introduce an adaptive frame fragmentation scheme with hybrid block sizing, implemented and evaluated on a real WSN testbed. Hi-Frag shows substantial enhancements over fixed-size partial packet recovery protocols, achieving up to 2.5× improvement in throughput when the channel condition is noisy, while reducing network delays by up to 14% of the observed delay. On average, Hi-Frag shows 35% gain in throughput compared to static fragmentation approaches across all channel conditions used in our experiments. Also, Hi-Frag lowers the energy consumed per useful bit by 66% on average compared to conventional protocols, which increases the energy efficiency.

The widespread use of location-aware devices together with the increased popularity of micro-blogging applications (e.g., Twitter) led to the creation of large streams of spatio-textual data. In order to serve real-time applications, the processing of these large-scale spatio-textual streams needs to be distributed. However, existing distributed stream processing systems (e.g., Spark and Storm) are not optimized for spatial/textual content. In this demonstration, we introduce Tornado, a distributed in-memory spatio-textual stream processing server that extends Storm. To efficiently process spatio-textual streams, Tornado introduces a spatio-textual indexing layer to the architecture of Storm. The indexing layer is adaptive, i.e., dynamically re-distributes the processing across the system according to changes in the data distribution and/or query workload. In addition to keywords, higher-level textual concepts are identified and are semantically matched against spatio-textual queries. Tornado provides data deduplication and fusion to eliminate redundant textual data. We demonstrate a prototype of Tornado running against real Twitter streams, where the users can register continuous or snapshot spatio-textual queries using a map-assisted query-interface.

Frame fragmentation techniques aim to achieve higher throughput by reducing retransmissions. Using experiments on a WSN testbed, we show that frame fragmentation also helps to reduce energy consumption. In this paper we propose Green-Frag, a new energy-efficient protocol based on efficient frame fragmentation technique. Green-Frag allows sensor nodes to transmit data with optimal transmit power and frame structure based on environmental conditions. Green-Frag takes into consideration the channel conditions, interference patterns and level, as well as the distance between sender and receiver. The paper discusses various design and implementation considerations for Green-Frag. Using experimental evaluation on a sensor mote testbed, we show that Green-Frag achieves the least energy consumption by choosing the best transmit power according to the channel conditions.