Online analytics, in most advanced scientific, business, and social media applications, rely heavily on the efficient execution of large numbers of Aggregate Continuous Queries (ACQs).ACQscontinuously aggregate streaming data and periodically produce results such asmaxoraverageover a given window of the latest data. It has been shown that it is beneficial to useIncremental Evaluation(IE) for re-using calculations performed over parts of theACQwindow, and to share them inmulti-query(MQ) environments among certain sets ofACQs. In this work, we re-examine how the principle of sharing is applied inIEtechniques as well as inMQoptimizers. We provide an extensive taxonomy ofIEtechniques and a new approach of using the state-of-the-artIEtechniques as part ofMQoptimizers in a way that reduces the execution plan costs by up to 270,000x. We evaluate all of our solutions both theoretically and experimentally using both real and synthetic datasets.

Data stream processing is becoming essential in most current advanced scientific or business applications as data production rates are increasing. Different companies compete to efficiently ingest high velocity data and apply some form of computation in order to make better business decisions. In order to successfully compete in this environment, companies are focusing on the most recent data within a count or time-based window by continuously executing aggregate queries on it. Incremental sliding-window computation is commonly used to avoid the performance implications of re-evaluating the aggregate value of the window from scratch on every update. The state-of-the-art FlatFAT technique executes ACQs with high efficiency but it does not scale well with the increasing workloads. In this paper we propose a novel algorithm, FlatFIT, that accelerates such calculations by intelligently maintaining index structures, leading to higher reuse of intermediate calculations and thus exceptional scalability in systems with heavy workloads. Our theoretical analysis shows that FlatFIT is superior in both time and space complexities compared to FlatFAT, while maintaining the same query generality. Given a window of size n, FlatFIT achieves constant algorithmic complexity compared to O(log(n)) complexity of FlatFAT. We experimentally show that FlatFIT achieves up to a 17x throughput improvement over FlatFAT for the same input workload while using less memory.

Data Stream Management Systems performing on-line analytics rely on the efficient execution of large numbers of Aggregate Continuous Queries (ACQs). The state-of-the-art WeaveShare optimizer uses the Weavability concept in order to selectively combine ACQs for partial aggregation and produce high quality execution plans. However, WeaveShare does not scale well with the number of ACQs. In this paper we propose a novel closed formula, F1, that accelerates Weavability calculations, and thus allows WeaveShare to achieve exceptional scalability in systems with heavy workloads. In general, F1 can reduce the computation time of any technique that combines partial aggregations within composite slides of multiple ACQs. We theoretically analyze the Bit Set approach currently used by WeaveShare and show that F1 is superior in both time and space complexities. We show that F1 performs 1062 times less operations compared to Bit Set to produce the same execution plan for the same input. We experimentally show that F1 executes up to 60,000 times faster and can handle 1,000,000 ACQs in a setting where the limit for the current technique is 550.

The slow intelligence system (SIS) technology is a novel technology for the design of a complex information system that is aware of the environment through multiple sensors and capable of improving its performance over time. In this paper we describe a practical slow intelligence system test bed where super-components can be specified to describe interactions among components. These super-components are automatically transformed into time controllers for components so they can be managed by the SIS test bed. We illustrate the application of this methodology to personal healthcare system design.