An in-memory database cluster consists of multiple interconnected nodes with a large capacity of RAM and modern multi-core CPUs. As a conventional query processing strategy, pipelining remains a promising solution for in-memory parallel database systems, as it avoids expensive intermediate result materialization and parallelizes the data processing among nodes. However, to fully unleash the power of pipelining in a cluster with multi-core nodes, it is crucial for the query optimizer to generate good query plans with appropriate intra-node parallelism, in order to maximize CPU and network bandwidth utilization. A suboptimal plan, on the contrary, causes load imbalance in the pipelines and consequently degrades the query performance. Parallelism assignment optimization at compile time is nearly impossible, as the workload in each node is affected by numerous factors and is highly dynamic during query evaluation. To tackle this problem, we propose elastic pipelining, which makes it possible to optimize intra-node parallelism assignments in the pipelines based on the actual workload at runtime. It is achieved with the adoption of new elastic iterator model and a fully optimized dynamic scheduler. The elastic iterator model generally upgrades traditional iterator model with new dynamic multi-core execution adjustment capability. And the dynamic scheduler efficiently provisions CPU cores to query execution segments in the pipelines based on the light-weight measurements on the operators. Extensive experiments on real and synthetic (TPC-H) data show that our proposal achieves almost full CPU utilization on typical decision-making analytical queries, outperforming state-of-the-art open-source systems by a huge margin.

The goal of EII systems is to provide uniform access to multiple data sources without having to first load them into a data warehouse. Since the late 1990's, several EII products have appeared in the marketplace and significant experience has been accumulated from fielding such systems. This collection of articles, by individuals who were involved in this industry in various ways, describes some of these experiences and points to the challenges ahead.

The feasibility of inferring functional dependencies from an example relation is investigated. The problem occurs in the context of automatic database design, when a tool is needed to assist the database designer in the process of specifying logical dependencies. The complexity of the dependency inference problem is inherently exponential. However, algorithms could be developed that perform well when the input relation has certain characteristics. Two such algorithms for dependency inference are implemented and optimized. An extensive set of experiments is presented, in which dependencies were inferred from example relations with different cardinalities, number of attributes, and degree of normalization. It is concluded that for practical example relations, an adequate implementation of a dependence inference function leads to acceptable interactive response times.>

The authors investigate the feasibility and efficiency of a parallel sort-merge algorithm by considering its implementation of the JASMIN prototype, a backend multiprocessor built around a fast packet bus. They describe the design and implementation of a parallel sort utility and present and analyze the results of measurements corresponding to a range of file sizes and processor configurations. The results show that using current, off-the-shelf technology coupled with a streamlined distributed operating system, three- and five-microprocessor configurations, provide a very cost-effective sort of large files. The three-processor configuration sorts a 100-Mb file in 1 hr which compares well to commercial sort packages available on high-performance mainframes. In additional experiments, the authors investigate a model to tune their sort software and scale their results to higher processor and network capabilities.>

Memory residence can buy both functionality and performance for a database management system. In this paper, we present a description and a benchmark of an experimental implementation of a Main Memory Database System (MMDBS) that was designed to support complex interactive queries. We describe and evaluate the main memory database structures and query processing algorithms implemented in this prototype. Our measurements and analysis, focused on aggregates and joins, include both memory requirements and response time, since there is a clear trade-off between space and time in the design of a MMDBS. In contrast to conventional Disk-based Database Systems (DDBS's), we found that an MMDBS can efficiently execute complex relational queries. We identify strategies that exploit memory residence effectively. We also identified a number of performance problems related to query optimization in main memory and memory management for MMDBS's.

This paper presents and analyzes algorithms for parallel processing of relational database operations in a general multiprocessor framework. To analyze alternative algorithms, we introduce an analysis methodology which incorporates I/O, CPU, and message costs and which can be adjusted to fit different multiprocessor architectures. Algorithms are presented and analyzed for sorting, projection, and join operations. While some of these algorithms have been presented and analyzed previously, we have generalized each in order to handle the case where the number of pages is significantly larger than the number of processors. In addition, we present and analyze algorithms for the parallel execution of update and aggregate operations.

The issue of duplicate elimination for large data files in which many occurrences of the same record may appear is addressed. A comprehensive cost analysis of the duplicate elimination operation is presented. This analysis is based on a combinatorial model developed for estimating the size of intermediate runs produced by a modified merge-sort procedure. The performance of this modified merge-sort procedure is demonstrated to be significantly superior to the standard duplicate elimination technique of sorting followed by a sequential pass to locate duplicate records. The results can also be used to provide critical input to a query optimizer in a relational database system.