In recent years the processing of continuous queries over potentially infinite data streams has attracted a lot of research attention. We observed that the majority of work addresses individual stream operations and system-related issues rather than the development of a general-purpose basis for stream processing systems. Furthermore, example continuous queries are often formulated in some declarative query language without specifying the underlying semantics precisely enough. To overcome these deficiencies, this article presents a consistent and powerful operator algebra for data streams which ensures that continuous queries have well-defined, deterministic results. In analogy to traditional database systems, we distinguish between a logical and a physical operator algebra. While the logical algebra specifies the semantics of the individual operators in a descriptive but concrete way over temporal multisets, the physical algebra provides efficient implementations in the form of stream-to-stream operators. By adapting and enhancing research from temporal databases to meet the challenging requirements in streaming applications, we are able to carry over the conventional transformation rules from relational databases to stream processing. For this reason, our approach not only makes it possible to express continuous queries with a sound semantics, but also provides a solid foundation for query optimization, one of the major research topics in the stream community. Since this article seamlessly explains the steps from query formulation to query execution, it outlines the innovative features and operational functionality implemented in our state-of-the-art stream processing infrastructure.

Our demonstration introduces a novel system architecture which massively facilitates optimization in data stream management systems (DSMS). The basic idea is to decouple optimization from the operative system by means of a secondary optimization system, which bears the burden of determining new query plans. Within the secondary system, which typically runs on a separate machine, we utilize suitable statistical models of the original data streams to simulate them. As the simulation can run at much faster rates, we are able to examine and assess new query plans in a shorter period of time without running the risk of deteriorating the original plan; we only migrate practically approved plans into the operative system. In our demonstration, we will present our prototypical implementation of this optimization architecture. We will demonstrate the interaction between primary and secondary system as well as the key features of the whole optimization process.

Data stream management systems need to control their resources adaptively since stream characteristics as well as query workload vary over time. In this paper we investigate an approach to adaptive resource management for continuous sliding window queries that adjusts window sizes and time granularities to keep resource usage within bounds. These two novel techniques differ from standard load shedding approaches based on sampling as they ensure exact query answers for given user-defined Quality of Service specifications, even under query re-optimization. In order to quantify the effects of both techniques on the various operations in a query plan, we develop an appropriate cost model for estimating operator resource allocation in terms of memory usage and processing costs. A thorough experimental study not only validates the accuracy of our cost model but also demonstrates the efficacy and scalability of the proposed techniques.

In data stream environments, the initial plan of a long-running query may gradually become inefficient due to changes of the data characteristics. In this case, the query optimizer generates a more efficient plan based on the current statistics. The online transition from the old to the new plan is called dynamic plan migration. In addition to correctness, an effective technique for dynamic plan migration should achieve the following objectives: 1) minimize the memory and CPU overhead of the migration, 2) reduce the duration of the transition, and 3) maintain a steady output rate. The only known solutions for this problem are the moving states (MS) and parallel track (PT) strategies, which have some serious shortcomings related to the above objectives. Motivated by these shortcomings, we first propose HybMig, which combines the merits of MS and PT and outperforms both in every aspect. As a second step, we extend PT, MS, and HybMig to the general problem of migration, where both the new and the old plans are treated as black boxes

In order to support continuous queries over data streams, a plethora of suitable techniques as well as prototypes have been developed and evaluated in recent years. In particular, it is of utmost importance to confirm their necessity and feasibility in real-world applications. For that reason, we have successfully coupled our infrastructure for data stream processing (PIPES) with an industrial Production-to-Business software (i-Plant) dedicated to highly automated manufacturing processes.

Adaptivity is a challenging open issue in data stream management. In this paper, we tackle the problem of memory adaptivity inside a system executing temporal sliding window queries over continuous data streams. Two different techniques to control the memory usage at runtime are proposed which refer to changes in window sizes and time granularities. Both techniques differ from standard load shedding approaches based on sampling as they ensure precise query answers for user-defined Quality of Service (QoS) specifications, even under query re-optimization.

PIPES is a flexible and extensible infrastructure providing fundamental building blocks to implement a data stream management system (DSMS). It is seamlessly integrated into the Java library XXL [1, 2, 3] for advanced query processing and extends XXL's scope towards continuous data-driven query processing over autonomous data sources.