Large-scale industrial internet of things (IIoT) applications usually access distributed equipment where high-volume sensor streams are processed. The building of scalable analytic queries and models over such streams could potentially enhance various industrial processes management tasks, e.g., distribution, delivery, and predictive online maintenance. To enable real-time and historical analytics over distributed IIoT applications, we have combined an edge data stream management system (EDSMS), sa.engine, with the highly distributed NoSQL database MongoDB. For supporting advanced analytics and high-volume stream injection into MongoDB, we integrated an extended query processing (EQP) system with sa.engine and MongoDB. This work demonstrates how EQP provides a holistic data management solution for IIoT based on a use case for sound anomaly detection of distributed equipment.

Many business and mission-critical decisions of the Industrial Internet of Things (IIoT) depend on efficient data management of sensor streams. Contemporary distributed IIoT applications consist of large numbers of sensors, producing massive volumes of heterogeneous sensor streams at high rates. The combination of these features of IIoT applications pose substantial challenges for existing Database Management Systems (DBMSs) in providing scalable data analytics. For example, Relational-DBMSs (RDBMSs) exhibit scalability issues, single point of failure, and difficulty in managing heterogeneity due to it’s rigid schemas. In contrast to RDBMSs, distributed NoSQL datastores could provide scalability of heterogeneous data. However, the simple query processing capabilities of NoSQL datastores limit advanced analytics. In this paper, we first compare both approaches, having an RDBMS and NoSQL backend for providing data-management solutions for distributed IIoT applications. Then, we utilize query processing in an in-memory database to integrate edge computing with the NoSQL datastore. By utilizing high-volume streams from a real-world IIoT application of Bosch Rexroth - Hägglund, we show that the proposed approach can potentially overcome the limitations of both RDBMS and NoSQL databases for performing advanced analytics.

With the advent of cyber-physical systems, industrial internet of things (IIoT) and industrial analytics numerous application scenarios have emerged where business and mission-critical decisions depend upon large scale analysis of data in form of sensor streams. However, large volumes of sensor stream data generated at high frequency pose substantial challenges for existing scalable data analysis techniques requiring the use of high-performance distributed datastores. This work covers in-depth performance comparison of three principal categories of distributed state-of the-art NoSQL datastores by evaluating their applicability and efficiency for large-scale analysis of sensor logs from real-world hydraulic power systems. One central datastore is selected from each of the three principal categories of NoSQL datastores: MongoDB from the document store, Cassandra from the column store and Redis from the distributed main memory key-value store to be included in the performance evaluation. Understanding the differences and behavior of this type of systems are crucial for optimizing application performance. Key insights from this work can serve as a basis for an improved understanding of the applicability of NoSQL datastores in systems for large scale data stream analysis. This will be important for supporting data analytics in IIoT applications as found in monitoring and control of Power plants, Smart Cities, Transportation systems, Environmental and Health monitoring, etc.

A stream validation system called SVALI is developed in order to continuously validate correct behavior of industrial equipment. A functional data model allows the user to define meta-data, analyses, and queries about the monitored equipment in terms of types and functions. Two different approaches to validate that sensor readings in a data stream indicate correct equipment behavior are supported: with the model-and-validate approach anomalies are detected based on a physical model, while with learn-and-validate anomalies are detected by comparing streaming data with a model of normal behavior learnt during a training period. Both models are expressed on a high level using the functional data model and query language. The experiments show that parallel stream processing enables SVALI to scale very well with respect to system throughput and response time. The paper is based on a real world application for wheel loader slippage detection at Volvo Construction Equipment implemented in SVALI.

Clustering queries over sliding windows require maintaining cluster memberships that change as windows slide. To address this, the Generic 2-phase Continuous Summarization framework (G2CS) utilizes a generation based window maintenance approach where windows are maintained over different time intervals. It provides algorithm independent and efficient sliding mechanisms for clustering queries where the clustering algorithms are defined in terms of queries over cluster data represented as temporal tables. A particular challenge for real-time detection of a high number of fastly evolving clusters is efficiently supporting smooth re-clustering in real-time, i.e. to minimize the sliding time with increasing window size and decreasing strides. To efficiently support such re-clustering for clustering algorithms where deletion of expired data is not supported, e.g. BIRCH, G2CS includes a novel window maintenance mechanism called Sliding Binary Merge (SBM), which maintains several generations of intermediate window instances and does not require decremental cluster maintenance. To improve real-time sliding performance, G2CS uses generation-based multi-dimensional indexing. Extensive performance evaluation on both synthetic and real data shows that G2CS scales substantially better than related approaches.

A potential problem for persisting large volume of data logs with a conventional relational database is that loading massive logs produced at high rates is not fast enough due to the strong consistency model and high cost of indexing. As a possible alternative, a modern NoSQL data store, which sacrifices transactional consistency to achieve higher performance and scalability, can be utilized. In this paper, we investigate to what degree a state-of-the-art NoSQL database can achieve high performance persisting and fundamental analyses of large-scale data logs from real world applications. For the evaluation, a state-of-the-art NoSQL database, MongoDB, is compared with a relational DBMS from a major commercial vendor and with a popular open source relational DBMS. MongoDB is chosen as it provides both primary and secondary indexing compared to other popular NoSQL systems. These indexing techniques are essential for scalable processing of queries over large scale data logs. To explore the impact of parallelism on query execution, sharding was investigated for MongoDB. Our results revealed that relaxing the consistency did not provide substantial performance enhancement in persisting large-scale data logs for any of the systems. However, for high-performance loading and analysis of data logs, MongoDB is shown to be a viable alternative compared to relational databases for queries where the choice of an optimal execution plan is not critical.

Main-memory database systems (MMDBs) are viable solutions for many scientific applications. Scientific and engineering data often require special indexing methods, and there is a large number of domain specific main memory indexing implementations developed. However, adding an index structure into a database system can be challenging. Mexima (Main-memory External Index Manager) provides an MMDB where new main-memory index structures can be plugged-in without modifying the index implementations. This has allowed to plug-into Mexima complex and highly optimized index structures implemented in C/C++ without code changes. To utilize new user-defined indexes in queries transparently, Mexima automatically transforms query fragments into index operations based on index property tables containing index meta-data. For scalable processing of complex numerical query expressions, Mexima includes an algebraic query transformation mechanism that reasons on numerical expressions to expose potential utilization of indexes. The index property tables furthermore enable validating the correctness of an index implementation by executing automatically generated test queries based on index meta-data. Experiments show that the performance penalty of using an index plugged into Mexima is low compared to using the corresponding stand-alone C/C++ implementation. Substantial performance gains are shown by the index exposing rewrite mechanisms.

Data-intensive applications in e-Science require scalable solutions for storage as well as interactive tools for analysis of scientific data. It is important to be able to query the data in a storage-independent way, and to be able to obtain the results of the data-analysis incrementally (in contrast to traditional batch solutions). We use the RDF data model extended with multidimensional numeric arrays to represent the results, parameters, and other metadata describing scientific experiments, and SciSPARQL, an extension of the SPARQL language, to combine massive numeric array data and metadata in queries. To address the scalability problem we present an architecture that enables the same SciSPARQL queries to be executed on the RDF dataset whether it is stored in a relational DBMS or mapped over a specialized geographically distributed e-Science data store. In order to minimize access and communication costs, we represent the arrays with proxy objects, and retrieve their content lazily. We formulate typical analysis tasks from a computational biology application in terms of SciSPARQL queries, and compare the query processing performance with manually written scripts in MATLAB.

The massive increase in the size of the data provided by e-Science applications requires not only to increase the capabilities of resources, but also to design new strategies for efficient utilization of already available resources. In this paper we present a scalable approach to extend a file-oriented storage system, Chelonia, with geographically distributed databases defined by a generic database schema. The database schema is able to model the data from typical e-Science applications. The system includes web service query service allowing e-Science applications to query the required data.

We have developed a system to process database queries over composed data providing Web services. The queries are transformed into execution plans containing an operator that invokes any Web service for given arguments. A common pattern in these query execution plans is that the output of one Web service call is the input for another, etc. The challenge addressed in this paper is to develop methods to speed up such dependent calls in queries by parallelization. Since Web service calls incur high-latency and message set-up costs, a naive approach making the calls sequentially is time consuming and parallel invocations of the Web service calls should improve the speed. Our approach automatically parallelizes the web service calls by starting separate query processes, each managing a parameterized sub-query, a plan function, for different parameter tuples. For a given query, the query processes are automatically arranged in a multi-level process tree where plan functions are called in parallel. The parallel plan is defined in terms of an algebra operator, FF_APPLYP, to ship in parallel to other query processes the same plan function for different parameters. By using FF_APPLYP we first investigated ways to set up different process trees manually. We concluded from our experiments that the best performing query execution plan is an almost balanced bushy tree. To automatically achieve the optimal process tree we modified FF_APPLYP to an operator AFF_APPLYP that adapts a parallel plan locally in each query process until an optimized performance is achieved. AFF_APPLYP starts with a binary process tree. During execution each query process in the tree makes local decisions to expand or shrink its process sub-tree by comparing the average time to process each incoming tuple. The query execution time obtained with AFF_APPLYP is shown to be close to the best time achieved by manually built query process trees.

A data providing web service returns a set of objects for a given set of parameters without any side effects. We demonstrate a system, WSMED, which provides a web service that can process SQL queries over any data providing web services. A challenge addressed by WSMED is to develop methods to speed up such queries by parallelization. WSMED automatically generates a distributed execution plan that calls web services in parallel. A common pattern in queries over web services is that the output of one web service call is the input for another. To speed-up such queries, WSMED automatically parallelizes the web service calls by starting separate query processes, each managing a parameterized sub-query, a plan function, for different parameters. To automatically achieve the optimal parallel process tree WSMED adapts an initial parallel plan locally in each query process until an optimized performance is achieved. The demonstration is a web interface to a WSMED server. It allow to make SQL queries joining any data providing web services, thus demonstrating that WSMED provides general search of composed web services.

Transportation-related problems, like road congestion, parking, and pollution, are increasing in most cities. In order to reduce traffic, recent work has proposed methods for vehicle sharing, for example for sharing cabs by grouping "closeby" cab requests and thus minimizing transportation cost and utilizing cab space. However, the methods published so far do not scale to large data volumes, which is necessary to facilitate large-scale collective transportation systems, e.g., ride-sharing systems for large cities. This paper presents highly scalable trip grouping algorithms, which generalize previous techniques and support input rates that can be orders of magnitude larger. The following three contributions make the grouping algorithms scalable. First, the basic grouping algorithm is expressed as a continuous stream query in a data stream management system to allow for a very large flow of requests. Second, following the divide-and-conquer paradigm, four space-partitioning policies for dividing the input data stream into sub-streams are developed and implemented using continuous stream queries. Third, using the partitioning policies, parallel implementations of the grouping algorithm in a parallel computing environment are described. Extensive experimental results show that the parallel implementation using simple adaptive partitioning methods can achieve speed-ups of several orders of magnitude without significantly degrading the quality of the grouping.

High energy physics scientists analyze large amounts of data looking for interesting events when particles collide. These analyses are easily expressed using complex queries that filter events. We developed a cost model for aggregation operators and other functions used in such queries and show that it substantially improves performance. However, the query optimizer still produces suboptimal plans because of estimate errors. Furthermore, the optimization is very slow because of the large query size. We improved the optimization by a profiled grouping strategy where the scientific query is first automatically fragmented into subqueries based on application knowledge. Each fragment is then independently profiled on a sample of events to measure real execution cost and cardinality. An optimized fragmented query is shown to execute faster than a query optimized with the cost model alone. Furthermore, the total optimization time, including fragmentation and profiling, is substantially improved.

Abstract Timely and efficient information communication is a key factor in ensuring successful collaboration in engineering collaborative design. This work proposes a database approach to support information communication between distributed and autonomous CAD systems. It provides the designer with an easy and flexible way, a project‐based propagation meta‐table, to specify what parts of a CAD information model should be communicated to other collaborating designers. A CAD peer manager, containing a peer database that stores information to be exchanged with the other collaborators, wraps each participating CAD system. The peer manager identifies changes made to the CAD model by using stored procedures and active rules in the peer database that are automatically generated based on the propagation meta‐table. The identified updates are propagated in a timely manner to other peers via inter‐database message passing, thereby minimizing the volume of necessary information to be exchanged. Furthermore, remote peer designers can flexibly incorporate, filter, or delete received updates by using a propagation control interface, which is also used to issue user's commands to download the data from the CAD system to the peer database and lookup the received messages in the peer database. The approach is applicable on any CAD system having a CORBA interface and can also be applied to other kinds of object‐oriented interfaces. Copyright © 2007 John Wiley & Sons, Ltd.

Abstract The mechanical product development process uses many different software systems to virtually simulate the behaviour of a design. The present work deals with flexible and efficient integration using object‐oriented mediator technology that provides transparent access to distributed engineering systems. The use of mediator technology is investigated for semi‐automatically integrating engineering information resident in computer aided design systems with a Common Object Request Broker Architecture based application programming interface. The purpose is to provide engineering analysis applications access to computer aided design system information and computational methods through a declarative query language. We conclude that the use of a declarative query language for developing engineering applications shows great potential in terms of flexibility, development productivity, performance, and ease of use, compared with using a procedural programming language. The work also shows new use of mediator technology, declarative queries, and active rules within engineering information integration that traditionally is accomplished using procedural programming. Copyright © 2004 John Wiley & Sons, Ltd.

To integrate many data sources we use a peer mediator-framework where views defined in the peers are logically composed in terms of each other A common approach to execute queries over mediators is to treat views in data sources as 'black boxes'. The mediators locally decompose queries into query fragments and submit them to the data sources for processing. Another approach, used in distributed DBMSs, is to treat the views as 'transparent boxes' by importing and fully expanding all views and merge them with the query. The black box approach often leads to inefficient query plans. However, in a peer mediator framework full view expansion (VE) leads to prohibitively long query compilation times when many peers are involved. It also limits peer autonomy since peers must reveal their view definitions. We investigate in a peer mediator framework the tradeoffs between none, partial, and full VE in two different distributed view composition scenarios. We show that it is often favorable with respect to query execution and sometimes even with respect to query compilation time to expand those views having common hidden peer subviews. However, in other cases it is better to use the 'black box' approach, in particular when peer autonomy prohibits view importation. Based on this, a hybrid strategy for VE in peer mediators is proposed.

A system has been developed permitting database queries over data extracted from a CAD system where the query result is returned back to the CAD for visualization and analysis. This has several challenges. First, CAD data representations use complex object-oriented schemas and the query language must be object-oriented too. Second, the query system resides outside the CAD system and must therefore use standardized data exchange formats for interoperability with the CAD. ISO STEP standard exchange formats are used for the exchange. Third, a CAD system cannot import an arbitrary object structure but places restrictions on the imported objects to be acceptable. Therefore, the query system must complement the query results in order to produce an acceptable CAD model, called the model completion of the query. These problems have been solved using an extensible object-relational query processor. The system also supports queries combining CAD data with data from other data sources.

Metadata for the World Wide Web is important, but metadata for Peer-to-Peer (P2P) networks is absolutely crucial. In this paper we discuss the open source project Edutella which builds upon metadata standards defined for the WWW and aims to provide an RDF-based metadata infrastructure for P2P applications, building on the recently announced JXTA Framework. We describe the goals and main services this infrastructure will provide and the architecture to connect Edutella Peers based on exchange of RDF metadata. As the query service is one of the core services of Edutella, upon which other services are built, we specify in detail the Edutella Common Data Model (ECDM) as basis for the Edutella query exchange language (RDF-QEL-i) and format implementing distributed queries over the Edutella network. Finally, we shortly discuss registration and mediation services, and introduce the prototype and application scenario for our current Edutella aware peers.

LH*g (Linear Hashing by grouping) is a high-availability extension of the LH* scalable distributed data structure. An LH*g file scales up with constant key search and insert performance, while surviving any single-site unavailability (failure). We achieve high availability through a new principle of record grouping. A group is a logical structure of up to k records, where k is a file parameter. Every group contains a parity record allowing for the reconstruction of an unavailable member. The basic scheme may be generalized to support the unavailability of any number of sites, at the expense of storage and messaging. Other known high-availability schemes are static, or require more storage, or provide worse search performance.

Abstract Integration of data from autonomous, distributed and heterogeneous data sources poses several technical challenges. This paper overviews the data integration system AMOS II based on the wrapper‐mediator approach. AMOS II consists of: (i) a mediator database engine that can process and execute queries over data stored locally and in several external data sources, and (ii) object‐oriented (OO) multi‐database views for reconciliation of data and schema heterogeneities among sources with various capabilities. The data stored in different types of data sources is translated and integrated using OO mediation primitives, providing the user with a consistent view of the data in all the sources. Through its multi‐database facilities many distributed AMOS II systems can interoperate in a federation. Since most data reside in the data sources, and to achieve high performance, the core of the system is a main‐memory DBMS having a storage manager, query optimizer, transactions, client–server interface, disk backup, etc. The AMOS II data manager is optimized for main‐memory access and is extensible so that new data types and query operators can be added or implemented in some external programming language. The extensibility is essential for providing seamlessaccess to a variety of data sources. Copyright © 2001 John Wiley & Sons, Ltd.

The mediator/wrapper approach is used to integrate data from different databases and other data sources by introducing a middleware virtual database that provides high level abstractions of the integrated data. A framework is presented for querying XML data through such an Object-Oriented (OO) mediator system using an OO query language. The mediator architecture provides the possibility to specify OO queries and views over combinations of data from XML documents, relational databases, and other data sources. In this way interoperability of XML documents and other data sources is provided. The mediator provides OO views of the XML data by inferring the schema of imported XML data from the DTD of the XML documents, if available, using a set of translation rules. A strategy is used for minimizing the number of types (classes) generated in order to simplify the querying. If XML documents not having DTDs are read, or if the DTD is incomplete, the system incrementally infers the OO schema from the XML structure while reading XML data. This requires that the mediator database is capable of dynamically extending and modifying the OO schema. The paper overviews the architecture of the system and describes incremental rules for translating XML documents to OO database structures.

The mediator-wrapper approach to integrate data from heterogeneous data sources has usually been centralized in the sense that a single mediator system is placed between a number of data sources and the applications. As the number of data sources increases, the centralized mediator architecture becomes a bottleneck. This paper presents an architecture for composable and distributed mediator servers, defined in terms of other mediator servers. The modularity of composable mediators allows to build larger systems of distributed mediators integrating many data sources, without the need to maintain a global schema. Composable mediators furthermore provide data independence by allowing locality of changes in both submediators and data sources. However a problem with a distributed and composable mediator architecture is that the query performance may degrade as the number of mediators increases. We describe some challenges for processing queries in this type of environment, and propose a distributed query decomposition algorithm that eliminates some of the overhead of logical mediator composition. For certain mediator compositions it produces distributed query plans whose inter-mediator data flow is optimal with respect to the query but is different from the logical interdependencies between the involved mediators. Experimental results show that this strategy improves the query performance and allows an increase of the number of mediators without query performance degradation.

The concept of object-oriented (OO) views has been a popular approach to data integration. Nevertheless, there have been few reported results on optimization of queries over integrated OO views. In our work, we have developed an OO view system for data integration based on the AMOS database mediator system. The paper describes a system architecture and implementation that takes advantage of query optimization techniques to improve the performance of queries to integrated OO views. The main features of the system are: 1) A passive mediation framework that preserves the autonomy of the data sources. 2) A selective materialization mechanism that minimizes the number of materialized view objects. 3) A predicate based mechanism to guarantee the validity of the materialized view objects as well as the completeness of queries to the view. In order to reduce the overhead of the passive view integration, we use inexpensive calculus based transformations to generate minimal query expressions before the query decomposition and the cost-based algebraic optimization take place. 1.

Terrain-aided navigation is a database application in which an aeroplane locates itself by matching the height trajectory with a terrain-elevation map on board the aircraft.Without an index a matching algorithm has to be applied to the whole map.This paper shows that an indexing technique, termed the IP-index, can efficiently filter out sub-areas of the map to provide starting positions for the matching algorithm (to improve the efficiency).Performance measurements of the IP-index in terrain-aided navigation are given in this paper.The IP-index is a dynamic indexing technique for large 1-D sequences on the value domain.It supports interpolation assumptions on the sequences, i.e., it can be used to query implicit values in addition to explicit values.It can be implemented by regular ordered indexes such as B-trees.

In recent years, research on distributing databases over networks has become increasingly important. In this paper, we concentrate on the issues of the interoperability of heterogeneous DBMSs and enforcing integrity across a multi-database made in this fashion. This has been done through a cooperative project between Aberdeen and Linko/spl uml/ping universities, with database modules distributed between the sites. In the process, we have shown the advantage of using DBMSs based on variants of the functional data model (FDM), which has made it remarkably straightforward to interoperate queries and schema definitions. Further, we have used the constraint transformation facilities of P/FDM (Prolog implementation of FDM) to compile global constraints into active rules installed locally on one or more AMOS (Active Mediators Object System) servers. We present the theory behind this, and the conditions for it to improve performance.

A time sequence is a discrete sequence of vahtes, e.g.temperature measurements, varying over time.Conventional indexes for time sequences are built on the time domain and cannot derd with inverse queries on a time sequence (i.e.computing the times when the values satis~some conditions).To process an inverse query the entire time sequence has to be scanned.This paper presents a dynamic indexing technique on the vahte domain for large time sequences which can be implemented using regular ordered indexing techniques (e.g.B-trees).Our index (termed ZP-irtdex) dramatically improves the query processing time of inverse queries compared to linear scanning.For periodic time sequences that have a limited range and precision on their value domain (most time sequences have this property), the W-index has an upper bound for insertion time and search time.

Presents a difference calculus for determining changes to rule conditions in an active DBMS. The calculus has been used for implementing an algorithm to efficiently monitor rules with complex conditions. The calculus is based on partial differencing of queries derived from rule conditions. For each rule condition, several partially differentiated queries are generated that each considers changes to a single base relation or view that the condition depends on. The calculus considers both insertions and deletions. The algorithm is optimized for deferred rule condition monitoring in transactions with few updates. The calculus allows us to optimize both space and time. Space optimization is achieved since the calculus and the algorithm does not presuppose materialization of monitored conditions to find its previous state. This is achieved by using a breadth-first, bottom-up propagation algorithm and by calculating previous states by doing a logical rollback. Time optimization is achieved through incremental evaluation techniques. The algorithm has been implemented and a performance study is presented at the end of the paper.

Object-oriented database systems (OODBs) have created a demand for relationally complete, extensible, and declarative object-oriented query languages. Until now, the runtime performance of such languages was far behind that of procedural OO interfaces. One reason is the internal use of a relational engine with magnetic disk resident databases. The authors address the processing of the declarative OO language WS-OSQL, provided by the fully operational prototype OODB called WS-IRIS. A WS-IRIS database is main memory (MM) resident. The system architecture, data structures, and optimization techniques are designed accordingly. WS-OSQL queries are compiled into an OO extension of Datalog called ObjectLog, providing for objects, typing, overloading, and foreign predicates for extensibility. Cost-based optimizations in WS-IRIS using ObjectLog are presented. Performance tests show that WS-IRIS is about as fast as current OODBs with procedural interfaces only and is much faster than known relationally complete systems. These results would not be possible for a traditional disk-based implementation. However, MM residency of a database appears to be only a necessary condition for better performance. An efficient optimization is of crucial importance as well.>