Databases are highly optimized complex systems with a multitude of configuration options. Especially in cloud scenarios with thousands of database deployments, determining optimized database configurations in an automated fashion is of increasing importance for database providers. At the same time, due to increased system complexity, it becomes more challenging to identify well-performing configurations. Therefore, research interest in autonomous or self-driving database systems has increased enormously in recent years. Such systems promise both performance improvements and cost reductions. In the literature, various fully or partially autonomous optimization mechanisms exist that optimize single aspects, e.g., index selection. However, database administrators and developers often distrust autonomous approaches, and there is a lack of practical experimentation opportunities that could create a better understanding. Moreover, the interplay of different autonomous mechanisms under complex workloads remains an open question. The presented cockpit enables an interactive assessment of the impact of autonomous components for database systems by comparing (autonomous) systems with different configurations side by side. Thereby, the cockpit enables users to build trust in autonomous solutions by experimenting with such technologies and observing their effects in practice.

The examination of causal structures is crucial for data scientists in a variety of machine learning application scenarios. In recent years, the corresponding interest in methods of causal structure learning has led to a wide spectrum of independent implementations, each having specific accuracy characteristics and introducing implementation-specific overhead in the runtime. Hence, considering a selection of algorithms or different implementations in different programming languages utilizing different hardware setups becomes a tedious manual task with high setup costs. Consequently, a tool that enables to plug in existing methods from different libraries into a single system to compare and evaluate the results is substantial support for data scientists in their research efforts.

Growing data volumes and velocities in fields such as Industry 4.0 or the Internet of Things have led to the increased popularity of data stream processing systems. Enterprises can leverage these developments by enriching their core business data and analyses with up-to-date streaming data. Comparing streaming architectures for these complex use cases is challenging, as existing benchmarks do not cover them. ESPBench is a new enterprise stream processing benchmark that fills this gap. We present its architecture, the benchmarking process, and the query workload. We employ ESPBench on three state-of-the-art stream processing systems, Apache Spark, Apache Flink, and Hazelcast Jet, using provided query implementations developed with Apache Beam. Our results highlight the need for the provided ESPBench toolkit that supports benchmark execution, as it enables query result validation and objective latency measures.

Many established processing and analysis tools in life sciences are still operating on files, e.g. alignment of genome data. Their integration into optimized workflows requires time-consuming transformation, import and export of data. In the given contribution, we introduce IMDBfs: a shared file system operating on top of an in-memory database system. Our IMDBfs provides transparent access to database resources via traditional file system operations. For the first time, it allows seamless integration of file-based life science tools into processes optimized for latest IMDB technology. As a result, existing tools neither need to be ported nor modified whilst transparent IMDB data access is provided.

Teaching agile software development by pairing lectures with hands-on projects has become the norm. This approach poses the problem of grading and evaluating practical project work as well as process conformance during development. Yet, few best practices exist for measuring the success of students in implementing agile practices. Most university courses rely on observations during the course or final oral exams. In this paper, we propose a set of metrics which give insights into the adherence to agile practices in teams. The metrics identify instances in development data, e.g. commits or user stories, where agile processes were not followed. The identified violations can serve as starting points for further investigation and team discussions. With contextual knowledge of the violation, the executed process or the metric itself can be refined. The metrics reflect our experiences with running a software engineering course over the last five years. They measure aspects which students frequently have issues with and that diminish process adoption and student engagement. We present the proposed metrics, which were tested in the latest course installment, alongside tutoring, lectures, and oral exams.

Emerging non-volatile memory technologies (NVM) offer fast and byte-addressable access, allowing to rethink the durability mechanisms of in-memory databases. Hyrise-NV is a database storage engine that maintains table and index structures on NVM. Our architecture updates the database state and index structures transactionally consistent on NVM using multi-version data structures, allowing to instantly recover data-bases independent of their size. In this paper, we demonstrate the instant restart capabilities of Hyrise-NV, storing all data on non-volatile memory. Recovering a dataset of size 92.2 GB takes about 53 seconds using our log-based approach, whereas Hyrise-NV recovers in under one second.

The ease of setting up collaboration infrastructures for software engineering projects creates a challenge for researchers that aim to analyze the resulting data. As teams can choose from various available software-as-a-service solutions and can configure them with a few clicks, researchers have to create and maintain multiple implementations for collecting and aggregating the collaboration data in order to perform their analyses across different setups. The DataRover system presented in this paper simplifies this task by only requiring custom source code for API authentication and querying. Data transformation and linkage is performed based on mappings, which users can define based on sample responses through a graphical front end. This allows storing the same input data in formats and databases most suitable for the intended analysis without requiring additional coding. Furthermore, API responses are continuously monitored to detect changes and allow users to update their mappings and data collectors accordingly. A screencast of the described use cases is available at https://youtu.be/mt4ztff4SfU

Significant medical knowledge has been generated over past decades, but is stored in databases distributed all over the globe using individual data formats. Detailed diagnostic tests result in steadily growing patient data. Consequently, medical experts are facing challenges outside of their field of expertise, e.g. analyzing, interpreting and linking medical data. In this contribution, we share details about our Medical Knowledge Cockpit, an application designed in an interdisciplinary cooperation with medical experts to improve the identification of relevant knowledge. Built upon latest in-memory database technology, it offers medical experts a unique starting point to link and analyze medical data on their own in real time. Result sets providing links back to primary data sources are assembled using patient specifics.

The introduction of columnar in-memory databases, along with hardware evolution, has made the execution of transactional and analytical workloads on a single system both feasible and viable. Yet, doing analytics directly on the transactional data introduces an increasing amount of resourceintensive aggregate queries which can slow down the overall system performance in a multi-user environment. To increase the scalability of a system in the presence of multiple such queries, we propose an aggregate cache in the general delta-main architecture that provides an ecient means to handle costly aggregate queries by applying incremental materialized view maintenance and query compensation techniques. Handling aggregate queries based on joins of multiple tables however is still a challenge as query compensation can be very expensive in the delta-main architecture of columnar in-memory databases. Our analysis of enterprise applications has revealed several data schema and workload patterns that can be leveraged for addressing performance of query processing using the aggregate cache. We contribute by presenting an approach to transport the application object semantics into the database system, becoming object-aware, and optimize the query processing using the aggregate cache by applying partition pruning and predicate pushdown in such general delta-main architecture. Our experimental validation using customer data and workloads confirms that this type of optimizations enables ecient usage of the aggregate cache for an even higher share of aggregate queries as one mean to scale the system.

Next-generation sequencing enables whole genome sequencing within a few hours at a minimum of cost, entailing advanced medical applications such as personalized treatments. However, this recent technology imposes new challenges to alignment and variant calling as subsequent analysis steps. Compared to former sequencing, both must deal with an increasing amount of data to process at a significantly lower data quality - and are currently not capable of that. In this work, we focus on addressing these challenges for identifying Single Nucleotide Polymorphisms, i.e. SNP calling, in genome data as one subtask of variant calling. We propose the application of a column-store in-memory database for efficient data processing and apply the statistical model that is provided by the Genome Analysis Toolkit's UnifiedGenotyper. Comparisons with the UnifiedGenotyper show that our approach can exploit all computational resources available and accelerates SNP calling up to a factor of 22x.

Latest medical diagnostics generate increasing amounts of big medical data. Specific software tools optimized for the use by healthcare experts and researchers as well as systematic processes for data processing and analysis in clinical and research environments are still missing. Our work focuses on the integration of high-throughput next-generation sequencing data and its systematic processing and its instantaneous analysis to use them in the course of precision medicine. We share our research results on designing a generic research process for drug response analysis including specific software tools built on top of our distributed in-memory computing platform for processing of big medical data. Furthermore, we present our technical foundations as well as process aspects of integrating and combining heterogeneous data sources, such as genome, patient, and experimental data.

Five years ago I proposed a common database approach for transaction processing and analytical systems using a columnar in-memory database, disputing the common belief that column stores are not suitable for transactional workloads. Today, the concept has been widely adopted in academia and industry and it is proven that it is feasible to run analytical queries on large data sets directly on a redundancy-free schema, eliminating the need to maintain pre-built aggregate tables during data entry transactions. The resulting reduction in transaction complexity leads to a dramatic simplification of data models and applications, redefining the way we build enterprise systems. First analyses of productive applications adopting this concept confirm that system architectures enabled by in-memory column stores are conceptually superior for business transaction processing compared to row-based approaches. Additionally, our analyses show a shift of enterprise workloads to even more read-oriented processing due to the elimination of updates of transaction-maintained aggregates.

Database workloads generated by enterprise applications are comprised of short-running transactional as well as long-running analytical queries with resource-intensive aggregations. The expensive aggregate queries can be significantly accelerated by using materialized views. This speed-up, however, comes with the cost of materialized view maintenance which is necessary to guarantee consistency when the underlying data changes. While several view maintenance strategies are applicable in the context of an in-memory column store, their performance depends on various factors, most importantly the ratio between queries accessing the materialized view and queries altering the base data, called insert ratio. As a contribution in this paper, we propose algorithms that determine the best-performing view maintenance strategy based on the currently monitored factors. Using our novel materialized aggregate engine, we are able to switch between view maintenance strategies on demand. We have created cost models for the identified view maintenance strategies that determine at which insert ratio it is advisable to switch to another strategy. Our benchmarks in SanssouciDB reveal that for all identified workloads, switching between maintenance strategies is more beneficial than staying with a single strategy.

Costs and time required for sequencing of DNA and RNA declined through use of next generation sequencing technology, e.g. up to 30-times coverage reads are generated in less than two days. However, its interpretation and analysis is still a time-consuming process potentially taking weeks. In this work, we present a completely new architecture for processing and analyzing genome data. It builds on the in-memory database technology to eliminate time-consuming file-based data operations and to enable real-time data analysis. We found out that the use of in-memory technology as an integral component for genome data processing and its analysis significantly reduces time and costs to obtain relevant results, e.g. in the course of personalized medicine.

A shared-nothing architecture is state-of-the-art for deploying a distributed analytical in-memory database management system: it preserves the in-memory performance advantage by processing data locally on each node but is difficult to scale out. Modern switched fabric communication links such as InfiniBand narrow the performance gap between local and remote DRAM data access to a single order of magnitude. Based on these premises, we introduce a distributed in-memory database architecture that separates the query execution engine and data access: this enables a) the usage of a large-scale DRAM-based storage system such as Stanford's RAMCloud and b) the push-down of bandwidth-intensive database operators into the storage system. We address the resulting challenges such as finding the optimal operator execution strategy and partitioning scheme. We demonstrate that such an architecture delivers both: the elasticity of a shared-storage approach and the performance characteristics of operating on local DRAM.

Enterprise applications such as available-to-promise (ATP), financial accounting, and dunning typically employ a mixed database workload with short-running transactional as well as analytical queries with resource-intensive aggregations. The latter type of queries can be significantly accelerated by using materialized views with pre-calculated aggregates. However, this speed-up comes with the cost of view maintenance which is necessary to guarantee consistency when the underlying data changes. In this paper, we evaluate existing view maintenance strategies in the context of a columnar in-memory database that is designed for mixed workloads. We propose a novel view maintenance strategy that takes the main-delta architecture and resulting merge process of columnar storage into account. A further contribution is a cost model which determines the best maintenance strategy given a specific workload. Our experiments using an ATP application show that our novel strategy outperforms other strategies in mixed workloads with an insert-ratio of more than 40 percent.

In the cloud services industry, a key issue for cloud operators is to minimize operational costs. In this paper, we consider algorithms that elastically contract and expand a cluster of in-memory databases depending on tenants' behavior over time while maintaining response time guarantees.

The introduction of a 64 bit address space in commodity operating systems and the constant drop in hardware prices made large capacities of main memory in the order of terabytes technically feasible and economically viable. Especially column-oriented in-memory databases are a promising platform to improve data management for enterprise applications. As in-memory databases hold the primary persistence in volatile memory, some form of recovery mechanism is required to prevent potential data loss in case of failures. Two desirable characteristics of any recovery mechanism are (1) that it has a minimal impact on the running system, and (2) that the system recovers quickly and without any data loss after a failure. This paper introduces an efficient logging mechanism for dictionary-compressed column structures that addresses these two characteristics by (1) reducing the overall log size by writing dictionary-compressed values and (2) allowing for parallel writing and reading of log files. We demonstrate the efficiency of our logging approach by comparing the resulting log-file size with traditional logical logging on a workload produced by a productive enterprise system.

Precise prediction of query execution performance is the basis for various database optimization strategies. With columnar in-memory databases, cost modeling changes in two dimensions: First, models for disk-based databases are not well-suited as the new bottleneck is main memory access. Second, the possibility to execute mixed workloads creates new challenges. For transactional and analytical queries with aggregation operations, memory access patterns and thus execution times vary significantly. This paper discusses the influences of data characteristics on aggregation operations and elevates not considered factors by existing cost model approaches. Further, we present benchmarks implemented and executed on a columnar in-memory research database to underline our assumptions.

Online transaction processing (OLTP) and online analytical processing (OLAP) are thought of as two separate domains, despite sharing the same business data to operate on. This is the result of performance impairments encountered in the past when running on the same system, the workloads becoming ever more sophisticated, leading to contradictory optimization in database design. Recent developments in hardware and database systems are bringing forth research prototypes supporting mixed OLTP and OLAP workloads, challenging this separation. At the same time new benchmarks are proposed to assess these mixed workload systems. In the demonstration, we show an interactive performance monitor and benchmark driver developed for the Composite Benchmark for Transaction Processing and Reporting. The performance monitor allows us to directly determine the impact of changing shares within the workload and to interactively assess behavioral characteristics of different database systems under changing mixed workload conditions.

Location-based event data is captured in RFID-aided supply chains for tacking individual goods. They are stored in distributed event repositories by involved supply chain parties. Performing anti-counterfeiting checks involves exchange of event data without exposure of sensitive business secrets. Current EPC global standards leave the definition of security strategies open for concrete implementation. We consider data security as the major aspect that needs to be clarified before wide adaption of EPC global standards will be considered by industries. The given work contributes by defining security extensions for EPC global networks and sharing implementation details of our research prototype. We show that incorporating in-memory technology enables history-based access control while keeping response times fast.

In Software-as-a-Service, multiple tenants are typically consolidated into the same database instance to reduce costs. For analytics-as-a-service, in-memory column databases are especially suitable because they offer very short response times. This paper studies the automation of operational tasks in multi-tenant in-memory column database clusters. As a prerequisite, we develop a model for predicting whether the assignment of a particular tenant to a server in the cluster will lead to violations of response time goals. This model is then extended to capture drops in capacity incurred by migrating tenants between servers. We present an algorithm for moving tenants around the cluster to ensure that response time goals are met. In so doing, the number of servers in the cluster may be dynamically increased or decreased. The model is also extended to manage multiple copies of a tenant's data for scalability and availability. We validated the model with an implementation of a multi-tenant clustering framework for SAP's in-memory column database TREX.

Key-value stores which keep the data entirely in main memory can serve applications whose performance criteria cannot be met by disk-based key-value stores. This paper evaluates the performance implications of cache-conscious data placement in an in-memory key-value store by examining how many values have to be stored consecutively in blocks in order to fully exploit memory locality during bandwidth-bound operations. We contribute by introducing a random block traversal main memory access pattern, by describing the corresponding memory access costs as well as by formally and experimentally deriving the correlation between block size and throughput. Our calculations and experiments vary the value and block sizes as well as their placement in the memory and derive their impact on cache-misses throughout the different memory hierarchies, the ability to prefetch data, and the number of needed CPU cycles to perform a certain set of data operations. The paper closes with the insight that a block-wise grouping of relatively few key-value pairs increases the throughput up to a factor six and with a discussion which implications a block-wise grouping of data has on the system design key-value store.

While analyzing a log file of a text-based source code search engine we discovered that developers search for fine-grained syntactical patterns in 36% of queries. Currently, to cope with queries of this kind developers need to use regular expressions, to add redundant terms to the query or to combine searching with other tools provided by the development environment. To improve the expressiveness of the queries, these can be formulated as tree patterns of abstract syntax trees. These search patterns can be expressed by using query languages, such as XPath. However, developers usually do not work with either XPath or with AST. To shield developers from the complexity of query formulation we propose using sample code snippets as queries. The novelty of our approach is the combination of a query language that is very close to the surface programming language and a special database technology to store a large amount of abstract syntax trees. The advantage of this approach over existing source code query languages and search engines is the performance of both query formulation and query execution. This paper describes the technical details of the method and illustrates the value of this approach with performance measures and an industrial controlled experiment. All developers were able to complete the tasks of the experiment faster and more accurately by using our tool (ACS) than by using a text-based search engine. The number of false positives in the result lists was significantly decreased.

Read-optimized columnar databases use differential updates to handle writes by maintaining a separate write-optimized delta partition which is periodically merged with the read-optimized and compressed main partition. This merge process introduces significant overheads and unacceptable downtimes in update intensive systems, aspiring to combine transactional and analytical workloads into one system. In the first part of the paper, we report data analyses of 12 SAP Business Suite customer systems. In the second half, we present an optimized merge process reducing the merge overhead of current systems by a factor of 30. Our linear-time merge algorithm exploits the underlying high compute and bandwidth resources of modern multi-core CPUs with architecture-aware optimizations and efficient parallelization. This enables compressed in-memory column stores to handle the transactional update rate required by enterprise applications, while keeping properties of read-optimized databases for analytic-style queries.

Enterprise applications are presently built on a 20-year old data management infrastructure that was designed to meet a specific set of requirements for OLTP systems. In the meantime, enterprise applications have become more sophisticated, data set sizes have increased, requirements on the freshness of input data have been strengthened, and the time allotted for completing business processes has been reduced. To meet these challenges, enterprise applications have become increasingly complicated to make up for short-comings in the data management infrastructure. To address this issue we investigate recent trends in data management such as main memory databases, column stores and compression techniques with regards to the workload requirements and data characteristics derived from actual customer systems. We show that a main memory column store is better suited for to days enterprise systems, which we validate by using SAP's Net Weaver Business Warehouse Accelerator and a realistic set of data from an inventory management application.

Many relational databases exhibit complex dependencies between data attributes, caused either by the nature of the underlying data or by explicitly denormalized schemas. In data warehouse scenarios, calculated key figures may be materialized or hierarchy levels may be held within a single dimension table. Such column correlations and the resulting data redundancy may result in additional storage requirements. They may also result in bad query performance if inappropriate independence assumptions are made during query compilation. In this paper, we tackle the specific problem of detecting functional dependencies between columns to improve the compression rate for column-based database systems, which both reduces main memory consumption and improves query performance. Although a huge variety of algorithms have been proposed for detecting column dependencies in databases, we maintain that increased data volumes and recent developments in hardware architectures demand novel algorithms with much lower runtime overhead and smaller memory footprint. Our novel approach is based on entropy estimations and exploits a combination of sampling and multiple heuristics to render it applicable for a wide range of use cases. We demonstrate the quality of our approach by means of an implementation within the SAP NetWeaver Business Warehouse Accelerator. Our experiments indicate that our approach scales well with the number of columns and produces reliable dependence structure information. This both reduces memory consumption and improves performance for nontrivial queries.

The emerging trend towards smart grids defines new requirements for designing enterprise applications for the energy market. Current solutions were built to process single billing runs as time-consuming batch jobs. Rather than processing some readings per year and household, a constant stream of meter readings has to be processed in context of a smart power grid. Additionally, consumers demand for convenient ways to monitor power consumption while getting real-time charging information on a daily basis. In this paper, we share our experiences of integrating meter readings in an industry-specific enterprise application for utilities to support real-time billing. Our prototype enables customers to track consumptions and to view billing details online. Besides, we discuss new business scenarios enabled by this real-timeliness.

RFID implementations leverage competitive business advantages in processing, tracking and tracing of fast-moving goods. Most of them suffer from security threats and the resulting privacy risks as RFID technology was not designed for exchange of sensible data. Emerging global RFID-aided supply chains require open interfaces for data exchange of confidential business data between business partners. We present a mutual authentication model based on one-time passwords preventing tag access by unauthorized third parties. Compared to models using complex on-tag encryption methods our implementation focuses on reducing tag-manufacturing costs while increasing customers' acceptance for RFID technology.

In this paper, we describe a main memory hybrid database system called HYRISE, which automatically partitions tables into vertical partitions of varying widths depending on how the columns of the table are accessed. For columns accessed as a part of analytical queries (e.g., via sequential scans), narrow partitions perform better, because, when scanning a single column, cache locality is improved if the values of that column are stored contiguously. In contrast, for columns accessed as a part of OLTP-style queries, wider partitions perform better, because such transactions frequently insert, delete, update, or access many of the fields of a row, and co-locating those fields leads to better cache locality. Using a highly accurate model of cache misses, HYRISE is able to predict the performance of different partitionings, and to automatically select the best partitioning using an automated database design algorithm. We show that, on a realistic workload derived from customer applications, HYRISE can achieve a 20% to 400% performance improvement over pure all-column or all-row designs, and that it is both more scalable and produces better designs than previous vertical partitioning approaches for main memory systems.

When SQL and the relational data model were introduced 25 years ago as a general data management concept, enterprise software migrated quickly to this new technology. It is fair to say that SQL and the various implementations of RDBMSs became the backbone of enterprise systems. In those days. we believed that business planning, transaction processing and analytics should reside in one single system. Despite the incredible improvements in computer hardware, high-speed networks, display devices and the associated software, speed and flexibility remained an issue.

The availability of huge system memory, even on standard servers, generated a lot of interest in main memory database engines. In data warehouse systems, highly compressed column-oriented data structures are quite prominent. In order to scale with the data volume and the system load, many of these systems are highly distributed with a shared-nothing approach. The fundamental principle of all systems is a full table scan over one or multiple compressed columns. Recent research proposed different techniques to speedup table scans like intelligent compression or using an additional hardware such as graphic cards or FPGAs. In this paper, we show that utilizing the embedded Vector Processing Units (VPUs) found in standard superscalar processors can speed up the performance of mainmemory full table scan by factors. This is achieved without changing the hardware architecture and thereby without additional power consumption. Moreover, as on-chip VPUs directly access the system's RAM, no additional costly copy operations are needed for using the new SIMD-scan approach in standard main memory database engines. Therefore, we propose this scan approach to be used as the standard scan operator for compressed column-oriented main memory storage. We then discuss how well our solution scales with the number of processor cores; consequently, to what degree it can be applied in multi-threaded environments. To verify the feasibility of our approach, we implemented the proposed techniques on a modern Intel multi-core processor using Intel® Streaming SIMD Extensions (Intel® SSE). In addition, we integrated the new SIMD-scan approach into SAP® Netweaver® Business Warehouse Accelerator. We conclude with describing the performance benefits of using our approach for processing and scanning compressed data using VPUs in column-oriented main memory database systems.