Array databases specialize in storage, management, and query processing on massive multidimensional array data such as satellite image time-series, weather forecast models, IoT sensor measurements, medical imaging data, etc. Some of the values in this data may be ”null” for a variety of reasons, such as unknown, known false, etc. The DBMS must handle null values correctly and efficiently. As of today, the effects of different data structures for representing null values on query processing in array databases have not been systematically studied. As a consequence, it is not clear what the optimal way for handling null values is. In this paper, we measure how four different methods for encoding null values perform across most common categories of array operations. The result is a comprehensive overview, publicly available in open source, providing relevant insights into the performance characteristics of these data structures allowing to discern which among them offers the most optimal approach for representing null values in an array processing context.

Arrays are an accepted data structure in database world today, next to sets, graphs, etc. Only relatively late, however, arrays have found acceptance in the database community as a first-class abstraction. Meantime, array databases with their declarative query support on "datacubes" have been built and are successfully applied in science, engineering, fintech, and other domains.In this case study we report about using array databases for the flexible management and analysis of spatio-temporal data for radio networks planning and optimization. We present the use case and how datacubes respond to the challenges of this field. To the best of our knowledge, this has not been attempted before, so it heralds a new field of application for geo array databases, in addition to well-known satellite imagery and climate datacubes.

Arrays as a fundamental data category have found their way into the orchestration of data models supported by databases. While OLAP "datacubes" can be emulated relationally to some extent it was in particular applications in science and engineering that have prompted support for arrays regardless of sparsity and with dedicated powerful array operators supporting n-D Tensor Algebra.

Datacubes form an accepted cornerstone for analysis (and visualization) ready spatio-temporal data offerings. Geo datacubes have been standardized since long under the umbrella concept of coverages, and such data structures are well understood in concept and practice. This, however, is not paired by a similar understanding of coverage analytics.We present a formal model for datacube analytics which is based on Linear Algebra, incorporates space and time semantics, and allows a wide range of common datacube operations, up to, say, the Discrete Fourier Transform. For convenience, the formalism is based on a language allowing expressions of any complexity.The specification is currently in the avanced adoption process of ISO for becoming the future 19123-3 standard.

Server-side extensibility through dynamically linked external code is a common method in relational databases. In the field of Array Databases such User-Defined Functions (UDFs) sometimes even represent the architectural cornerstone for array functionality. On the downside, UDF implementation often suffers from high coding complexity.The rasdaman Array DBMS is a full-stack C++ implementation, so does not rely on some generic UDF mechanism. This allowed designing such an API from scratch, with particular emphasis on UDF coder convenience. The rasdaman UDFs rely on the general C++ client API classes. Based on a straightforward UDF interface definition adapter code is generated automatically. Experimental evaluation shows encouraging results, and the mechanism is going to be used in research and under operational conditions. We present the approach and motivate it through practical use cases.

Machine Learning is increasingly being applied to many different application domains. From cancer detection to weather forecast, a large number of different applications leverage machine learning algorithms to get faster and more accurate results over huge datasets. Although many of these datasets are mainly composed of array data, a vast majority of machine learning applications do not use array databases. This tutorial focuses on the integration of machine learning algorithms and array databases. By implementing machine learning algorithms in array databases users can boost the native efficient array data processing with machine learning methods to perform accurate and fast array data analytics.

Array Databases close a gap in the database ecosystem by adding modelling, storage, and processing support for multi-dimensional arrays. Built into “datacubes” such structures are known since long in OLAP and statistics, but they also appear as spatio-temporal sensor, image, simulation, and statistics data in all science and engineering domains.

The European Copernicus programme generates massive amounts of Earth Observation (EO) data, with the goal of improving our environmental understanding and management on local, regional, and global level. An initiative of this magnitude comes with a set of challenges, especially pertaining to effective service management once the huge volumes of raster data files are distributed across open data centers and commercial companies.The BigDataCube project responds to this challenge through the concept of federated, analysis-ready datacubes, exposed via the open OGC geo standard interfaces for interoperable access and processing, WMS, WCS, and WCPS.Such federated datacube services have been established in the project between the public German Copernicus hub, CODE-DE, and a commercial cloud provider, cloudeo AG, altogether offering access to more than 500 TB. Both data pools are federated in a location-transparent manner, establishing a common information space where users can query and combine datacubes without knowing their location of storage. In this nucleus of a growing federation recently the Alfred Wegener maritime research institute has joined, and further data centers are in the line. The spectrum of functionality available is showcased through several realistic use cases, including a value-adding Sentinel-1 SAR product on sea state and wind speed in the North Sea. The platform used is the pioneer datacube engine and OGC reference implementation, rasdaman. We report on the outcomes of the project, lessons learned, and further work foreseen in this area.

We demonstrate the rasdaman ("raster data manager") scalable datacube engine in a series of multi-dimensional live scenarios of spatio-temporal datacube analytics, distributed processing in federations, as well as simple, rapid construction of datacubes.

With the unprecedented availability of continuously observed and generated data there is a likewise unprecedented potential for new and timely insights; yet, benefits are not fully leveraged as of today. The plethora of formats in combination with heterogeneous services remains is an obstacle - e.g., image services prefer binary formats, SPARQL endpoints like to think in RDF triples, and browsers integrate JSON data smoothly. We propose a model-based multi-encoding approach for overcoming the limitations of individual formats while still supporting their use. Concretely, this approach is being followed by the OGC Coverage Implementation Schema (CIS) standard which establishes a concrete, interoperable data model unifying n-D spatiotemporal regular and irregular grids, point clouds, and meshes. We describe how independence from data formats is achieved, in particular for three practically relevant formats - XML, JSON, and RDF -, thereby fostering integration of hitherto rather separate application domains.

Geo data like satellite imagery and climate simulation output constitute major contributors to today's Big Data deluge. The datacube paradigm is currently being considered not only for getting data analysis ready, but also to offer advanced functionality in an easy-to-use way while allowing for highly effective server-side optimizations. Datacube standards can help substantially in unifying access protocols, thereby easing access for users and also paving the way for cross-data-center datacube fusion. We present OGC datacube model which is based on the coverage data and service model standardized by OGC, ISO, and INSPIRE. This paper aims at contributing to a cross-disciplinary exchange on modeling challenges, successes, and possible overlaps amenable to generalization and harmonization.

Array databases are used to manage and query large N-dimensional arrays, such as sensor data, simulation models and imagery, as well as various time-series. Modern database systems and database applications make extensive use of caching techniques to improve performance. Research on array databases on the other hand has not explored the potential benefits of caching in query processing on big arrays. In this work we propose a design for a content-aware cache for array databases which allows to reuse results of previously evaluated queries. Besides identical query matching, our method also takes into account spatially overlapping queries and queries with common subexpressions. We evaluate performance of the query cache implementation by varying data and query parameters and show that it decreases query execution time by up to 93%, with a potential for even higher savings with increasing query complexity.

With the deluge of scientific big data affecting a large variety of research institutions, support for large multidimensional arrays has gained traction in the database community in the past decade. Array databases aim to cover the gap left by traditional relational database systems in the domains of large scientific data by enabling researchers to efficiently store and process their data through rich declarative query languages. Such large amounts of data need effective systems that are able to distribute the processing at both local level, through exploitation of heterogeneous hardware as well as at network level, enabling both intra-cloud and intra-federation distribution of data and processing. In this demonstration we aim to showcase the capabilities of rasdaman by allowing users to execute queries that combine petabyte datasets stored at two institutions on different continents.

Array Databases close a gap in the database ecosystem by adding modeling, storage, and processing support on multi-dimensional arrays. Declarative queries provide processing of arrays of regularly massive size, such as Tera-to Petabyte datacubes, while allowing internal degrees of freedom in partitioning the large arrays into tractable sub-arrays. Among the important new operations is the array Theta-Join, such as overlaying two images. Evaluation of such joins is complicated by the fact that the participating arrays likely do not align in their partitioning schemes. This can lead to inefficient multiple reads of sub-arrays. We introduce array joins and present an efficient way of pairing corresponding sub-arrays. As a byproduct, this technique delivers information on optimal data placement for parallel join evaluation. The method is implemented in the Array DBMS rasdaman which is in operational use at data centers and mapping agencies.

This contribution introduces the forthcoming extension of the ISO SQL standard for multi-dimensional arrays, SQL/MDA. We present concepts, the language, and highlight how it can be implemented in a scalable manner. Examples used stem from Earth Observation and related domains.

Several scientific domains operate on mesh data structures. Computations on meshes share a set of basic abstractions which relates to topological and geometric structure of meshes. Due to the complexity of modeling and querying meshes, the database community has neglected the need for a dedicated mesh database and in particular a high-level mesh query language. Currently, such computations is done by using general-purpose programming languages. Such implementations are tightly coupled with dataset characteristics and thus are less reusable, their maintenance cost is high, and their performance is limited to the memory size. In this paper, we describe syntax and semantic of a high-level query language for scientific meshes (called MQuery) with a level of abstraction that can express queries based on topological and geometric navigations. The language is build on top of a graph-based conceptual mesh data model. We report our results comparing two implementations of the language on top of relational and graph databases with a mesh library.

Support for large arrays has been increasingly gaining attention by the database community. Array databases are a quickly expanding category of database management systems that treat large, multidimensional array data as first-class database citizens, allowing convenient and efficient storage and retrieval. Large array data on its own, however, is rarely meaningful or machine understandable. Therefore, arrays are almost always linked to additional, small metadata, typically managed in relational databases. This leads to an information integration gap, an "impedance mismatch" that we address with the work presented in this paper.

Flexible, scalable services on massive geo data receive much attention today. In particular, the OGC Web Coverage Service (WCS) standards suite has established a best practice for versatile access and retrieval on spatio-temporal "Big Data". Fewer efforts have been devoted, though, to an easy-to-use, standardized way of maintaining a service's offering. Our experience from supporting a series of heterogeneous, large-scale services reveals that this can become tedious indeed, due to heterogeneity and incompleteness of incoming data, operator-less transformation and ingest of large amounts of files, as well as the need for narrowly focused manual corrections and updates sometimes.

Earth-Science data are composite, multi-dimensional and of significant size, and as such, continue to pose a number of ongoing problems regarding their management. With new and diverse information sources emerging as well as rates of generated data continuously increasing, a persistent challenge becomes more pressing: To make the information existing in multiple heterogeneous resources readily available. The widespread use of the XML data-exchange format has enabled the rapid accumulation of semi-structured metadata for Earth-Science data. In this paper, we exploit this popular use of XML and present the means for querying metadata emanating from multiple sources in a succinct and effective way. Thereby, we release the user from the very tedious and time consuming task of examining individual XML descriptions one by one. Our approach, termed Meta-Array Data Search (MAD Search), brings together diverse data sources while enhancing the user-friendliness of the underlying information sources. We gather metadata using different standards and construct an amalgamated service with the help of tools that discover and harvest such metadata; this service facilitates the end-user by offering easy and timely access to all metadata. The main contribution of our work is a novel query language termed xWCPS, that builds on top of two widely-adopted standards: XQuery and the Web Coverage Processing Service (WCPS). xWCPS furnishes a rich set of features regarding the way scientific data can be queried with. Our proposed unified language allows for requesting metadata while also giving processing directives. Consequently, the xWCPS-enabled MAD Search helps in both retrieval and processing of large data sets hosted in an heterogeneous infrastructure. We demonstrate the effectiveness of our approach through diverse use-cases that provide insights into the syntactic power and overall expressiveness of xWCPS. We evaluate MAD Search in a distributed environment that comprises five high-volume array-databases whose sizes range between 20 and 100 GB and so, we ascertain the applicability and potential of our proposal.

Arrays are among those data types which contribute the most to Big Data -- examples include satellite images and weather simulation output in the Earth sciences, confocal microscopy and CAT scans in the Life sciences, as well as telescope and cosmological observations in Space science, to name but a few. Traditionally, the database community has neglected this, with the effect that ad-hoc implementations prevail. With the advent of NewSQL in recent years, however, the database scope has broadened, and array modelling and query support is seriously considered. Different models have been suggested, some of which are implemented or under implementation, and a consolidation of concepts can be observed. Consequently, integration of array queries into SQL is being addressed.

Although, many applications use unstructured meshes, there is no specialized mesh database which supports storing and querying mesh data. Existing mesh libraries do not support declarative querying and are expensive to maintain. A mesh database can benefit the domains in several ways such as: declarative query language, ease of maintenance, etc. In this paper, we propose the Incidence multi-Graph Complex (ImG-Complex) data model for storing topological aspects of meshes in a database. ImG-Complex extends incidence graph (IG) model with multi-incidence information to represent a new object class which we call ImG-Complexes. We introduce optional and application-specific constraints to limit the ImG model to smaller object classes and validate mesh structures based on the modeled object class properties. We show how Neo4j graph database can be used to query mesh topology based on the (possibly constrained) ImG model. Finally, we experiment Neo4j and PostgreSQL performance on executing topological mesh queries.

In the era of Big Data, being able to work with multidimensional arrays in a robust and consistent manner as the number and variety of dimensions increase, is just as important as being able to handle the large volumes inherent to this type of data. Usually, array analytics is carried out to extract meaningful information for further applications, e.g. slicing and subsetting. While domain-specific dimensions, which are beyond spatio-temporal, underlie rich domain anchor semantics, assigning consistent dimension schema for Points Of Interest (POI) across multidisciplinary data sets is challenging. New compositions of CRSs need to be constructed on the fly by a heterogeneous community with different backgrounds and applications in mind, consequently, linking dimensions via different resolvers to drive away dimension fragments from high-dimensional spaces. We propose to identify dimensions via a linked resolver approach. Such an approach allows CRSs to be referred to and looked up across multidisciplinary applications. Finally, we present a planetary use case, and specification- and scenario-based testing results to validate our approach.

Never before in history mankind has collected data at the rates we face today. Alone in 2002, an estimated 403 Petabyte of data has been acquired, equivalent to all printed information ever created before. Earth orbiting satellites, as well as ground, airborne, and underwater sensors, space observatories scan their environment at unprecedented resolutions, giving rise to "Big Science". The same holds for the life sciences where genomic data, high-resolution scans, and other modalities are collected in steadily increasing streams. Social network analysis, OLAP, and stock exchange trading represent further examples, the latter involving real-time correlation of thousands of ticker time series resulting in Terabytes of data to be analysed per single run. Summarized under Large-Scale Analytics we are witnessing an exploding demand for flexible access to massive volumes of scientific and business data sets. Arguably a large class of these massive data is represented by multi-dimensional arrays. Consequently, large arrays pose new challenges to data modelling, querying, optimization, and maintenance -- in short: we need Large-Scale Array Analytics. This tutorial introduces to the topic from a database perspective. Aspects addressed include modelling, query languages, query optimization and parallelization, and storage management. High emphasis will be devoted to applications in "Big Science", particularly geo, space, and life sciences; real-life use cases will be presented and discussed which stem from our 15 years of experience with the open-source rasdaman array DBMS and our work on geo raster service standardization. We will highlight requirements, achievements, open research issues, and avenues for future research. Discussion will make use of real-life examples, many of which Internet connected participants can replay hands-on.

In the classical triad of vector, raster, and meta data, it is the raster part which is not yet sufficiently supported in SDIs nowadays. Consequently, integration of earth observation imagery, LIDAR, legacy map scans, etc. into Spatial Data Infrastructures (SDIs) remains incomplete. In terms of standards, the OGC Web Coverage Service (WCS) Standard defines open interfaces for accessing and processing of raster data, more generally: coverages. In August 2010, the completely overhauled WCS 2.0 has been adopted by OGC. To make coverages interchangeable across all OGC-based services, WCS 2.0 has been based on Geography Markup Language (GML) 3.2.1, with a small, backwards compatible addition to achieve informational completeness. In parallel to specification writing, its reference implementation and an online demo are being pursued.

SOA is often used today for architectures distributing geo-processing computation tasks. These tasks usually are both data and computationally expensive. A fundamental issue is finding an automated, efficient task distribution and coordination among different services. We propose a language-based approach to this issue. First, the distributed geo-processing problem is written up as a declarative query using the Open Geospatial Consortium (OGC) Web Coverage Processing Service (WCPS) language standard. Second, translate the query is translated into a graph on which a rule-based graph decomposition algorithms generates a task distribution among a subset of the service hosts available. A heuristic cost function, preliminarily based on the data output sizes of each query operator, is used to partition the query and determine the execution hosts of sub-queries. The result is reassembled into a recursively nested query which also performs coordination of its distributed evaluation.

Originally Binary Large Objects (BLOBs) in databases were conceived as a means to capture any large data (whatever large meant at the time of writing) which, for whatever reason, cannot or should not be modeled relationally. Today we find images, movies, XML, formatted documents, and many more data types stored in database BLOBs. A particular challenge obviously is moving such large units of data as fast as possible, hence performance benchmarks are of interest.

Storage management of multidimensional arrays aims at supporting the array model needed by applications and insuring fast execution of access operations. Current approaches to store multidimensional arrays rely on partitioning data into chunks (equally sized subarrays). Regular partitioning however, does not adapt to access patterns, leading to suboptimal access performance. We propose a storage approach for multidimensional discrete data (MDD) based on multidimensional arbitrary tiling. Tiling is arbitrary in that any partitioning into disjoint multidimensional intervals as well as incomplete coverage of n-D space and gradual growth of MDDs are supported. The proposed approach allows the storage structure to be configured according to user access patterns through tunable tiling strategies. We describe four strategies and respective tiling algorithms and present performance measurements which show their effectiveness in reducing disk access and post-processing times for range queries.

Storing multidimensional data in databases is an important topic both in the industrial and scientific database communities. Arrays are offered as a multidimensional data structure by most programming languages. Conventional database systems, however, do not support arrays of arbitrary dimensionality and base type. RasDaMan is a DBMS integrating arrays as a first class data type offering both a declarative query language and a specialised storage structure for arrays. The work presented evaluates the performance of queries on multidimensional array data stored in RasDaMan versus storage in a conventional RDBMS. In the relational system, the data is both mapped to relations and stored directly as binary data in BLOBs. The queries executed were modelled after queries common in scientific applications and decision support.

RasDaMan is a universal — i.e., domain-independent — array DBMS for multidimensional arrays of arbitrary size and structure. A declarative, SQL-based array query language offers flexible retrieval and manipulation. Efficient server-based query evaluation is enabled by an intelligent optimizer and a streamlined storage architecture based on flexible array tiling and compression.

In the RasDaMan project a database system for management of multidimensional arrays is being built. It offers a declarative query language extending SQL-92 with operations on arrays of arbitrary base types and a C++ programming interface. Integrating arrays in the query language enables the system to process complex queries on high-volume multidimensional data in the database server close to physical data storage. Storage of arrays is done in tiles of arbitrary size. Operations on arrays are transformed into operations on tiles during query optimization and execution. These operations are then executed on tiles loaded from mass storage. This paper describes the underlying formal model for tile based operations on multidimensional arrays and its efficient implementation in C++ as part of the RasDaMan system.