Digital twinning has been established as one of the top ten technology trends in the last couple of years. The driver for this is the ongoing fourth industrial revolution in conjunction with the continuous technological developments in the necessary areas such as big data, Internet-of-Things (IoT), cloud computing, and artificial intelligence/machine learning (AI/ML). The goal of digital twinning is the (real-time) optimization of a physical entity based on its digital copy requiring the integration of various concepts of the above mentioned areas. To achieve this integration, various digital twin frameworks have been developed being used in a wide range of application domains. These frameworks are characterized by a high degree of flexibility, which, however, also complicates their usage. To overcome that shortcoming, we propose to design and to develop a comprehensive digital twin system that can be used out-of-the-box. In our view, such a system should be built from a data management perspective and should borrow well-established concepts from other data management systems such as database or data streaming systems.

Abstract Integer compression plays an important role in columnar database systems to reduce the main memory footprint as well as to speedup query processing. To keep the additional computational effort of (de)compression as low as possible, the powerful Single Instruction Multiple Data ( SIMD ) extensions of modern CPUs are heavily applied. While a scalar compression algorithm usually compresses a block of N consecutive integers, the state-of-the-art SIMDified implementation scales the block size to $$k \cdot N$$ k · N with k as the number of elements which could be simultaneously processed in an SIMD register. On the one hand, this scaling SIMD approach improves the performance of (de)compression. But on the other hand, it can lead to a degradation of the memory footprint of the compressed data. Within this article, we analyze this degradation effect for various integer compression algorithms and present a novel SIMD concept to overcome that effect. The core idea of our novel SIMD concept called BOUNCE is to concurrently compress k different blocks of size N within SIMD registers, guaranteeing the same compression ratio as scalar variant. As we are going to show, our proposed SIMD idea works well on various Intel CPUs and may offer a new generalized SIMD concept to optimize further algorithms.

The Single Instruction Multiple Data (SIMD) paradigm became a core principle for optimizing query operators. Until now, only the LOAD/STORE instructions are considered to be efficient enough to achieve the expected speedups, while avoiding GATHER/SCATTER is considered almost imperative. However, the GATHER instruction offers a very flexible way to populate SIMD registers with data elements coming from non-consecutive memory locations. As we will show within the paper, the GATHER instruction can achieve the same performance as the LOAD instruction, if applied properly. To enable the proper usage, we outline a novel access pattern which then allows fine-grained, partition-based SIMD implementations using the GATHER instruction.

For fast and efficient data processing, a common approach in many application domains is to mainly store and process data in form of arrays of integers offering several benefits. For example, with the help of some additional lightweight computations for lossless integer compression, the necessary memory space can be dramatically reduced. Moreover, compressed integer values offer advantages for data processing such as increasing the effective bandwidth to reduce the memory wall effect. Thus, a large corpus of lightweight integer compression formats has been developed. Unfortunately, there is no approach available that allows to define a specific format in a consistent way and to generate executable code for compression and decompression from it. To overcome that, we present the Lightweight Compression Template Library (LCTL) in this paper. As we are going to show, LCTL allows (i) the implementation of a variety of compression formats in an abstract way, (ii) the generation of efficient executable code for compression, and (iii) the automatic derivation of the decompression routines out of a compression format. Thus, LCTL offers a unique approach to comprehensively and systematically implement the large corpus of lightweight integer compression formats.

In this demo, we present MorphStore, an in-memory column store with a novel compression-aware query processing concept. Basically, compression using lightweight integer compression algorithms already plays an important role in existing in-memory column stores, but mainly for base data. The continuous handling of compression from the base data to the intermediate results during query processing has already been discussed, but not investigated in detail since the computational effort for compression as well as decompression is often assumed to exceed the benefits of a reduced transfer cost between CPU and main memory. However, this argument increasingly loses its validity as we are going to show in our demo. Generally, our novel compression-aware query processing concept is characterized by the fact that we are able to speed up the query execution by morphing compressed intermediate results from one scheme to another scheme to dynamically adapt to the changing data characteristics during query processing. Our morphing decisions are made using a cost-based approach.

Lightweight integer compression algorithms are frequently applied in in-memory database systems to tackle the growing gap between processor speed and main memory bandwidth. In recent years, the vectorization of basic techniques such as delta coding and null suppression has considerably enlarged the corpus of available algorithms. As a result, today there is a large number of algorithms to choose from, while different algorithms are tailored to different data characteristics. However, a comparative evaluation of these algorithms with different data and hardware characteristics has never been sufficiently conducted in the literature. To close this gap, we conducted an exhaustive experimental survey by evaluating several state-of-the-art lightweight integer compression algorithms as well as cascades of basic techniques. We systematically investigated the influence of data as well as hardware properties on the performance and the compression rates. The evaluated algorithms are based on publicly available implementations as well as our own vectorized reimplementations. We summarize our experimental findings leading to several new insights and to the conclusion that there is no single-best algorithm. Moreover, in this article, we also introduce and evaluate a novel cost model for the selection of a suitable lightweight integer compression algorithm for a given dataset.

Modern database systems are very often in the position to store and efficiently process their entire data in main memory. Aside from increased main memory capacities, a further driver for in-memory database systems has been the shift to a column-oriented storage format in combination with lightweight data compression techniques. In recent years, a lot of lightweight data compression algorithms have been developed to efficiently support different data characteristics. Therefore, database systems should include a large number of these algorithms. To enable this, we introduce our novel modularization concept including our model kit implementation for lightweight data compression algorithms.