Group-By-Aggregate (GBA) queries are fundamental in data analysis, allowing users to organize data by specific attributes and apply aggregate functions, such as sum, average, and count, to generate insights from large datasets. Traditionally, DBMSs handle GBA queries through either sort-based or hashbased methods, each with unique strengths and limitations. Sortbased approaches, while effective in certain contexts, can become computationally expensive when few groups are involved due to extensive record comparisons. On the other hand, hash-based methods are efficient with larger group counts but are memoryintensive and may suffer from hash collisions, depending on data distribution and hashing functions.This paper presents an empirical performance study focused on traditional sort- and hash-based GBA operators, introducing a robust hybrid approach that unifies their strengths to deliver consistent, high performance across diverse data conditions. The hybrid GBA method combines hash-based grouping and early aggregation for the local phase with sort-based aggregation for the global phase. It forwards partially aggregated results when memory limits are reached, reducing data volume and minimizing memory usage and disk I/O. By eliminating the need to choose between sort and hash GBA operators and removing dependency on precise statistics or query optimizer intervention, the hybrid method significantly enhances query execution reliability. Extensive testing on Wisconsin, TPC-H, and TPC-DS benchmark datasets demonstrates that the hybrid approach consistently matches the performance of the bestsuited algorithm—whether sort or hash—in each scenario, across varying dataset sizes and complexities. This unified solution replaces separate GBA algorithms, providing a flexible, easily adaptable, and highly efficient method for processing GBA queries in big data management systems.

Efficient multi-join query processing is crucial but remains a complex, ongoing challenge for high-performance data management systems (DBMSs). This paper studies the impact of different memory distribution techniques among join operators on different classes of multi-join query plans under different assumptions regarding memory availability and storage devices such as HDD and SSD on Amazon Web Services (AWS). We re-evaluate the results of one of the early impactful studies from the 1990s that was originally done using a simulator for the Gamma database system.

Over the years, data management systems have undergone significant transformations adapting to advancements in hardware, diverse data types, and varying workloads. Numerous architectures have been proposed to more effectively address the customer needs, taking into account rapidly evolving factors like emerging data types, shifting workload and access patterns, and modern hardware innovations. With emergence of cloud infrastructures and their improved capability in seamless scaling, the shared-nothing architecture, which dominated the field for about three decades, is now being replaced with disaggregated architectures. The considerable cost of cloud resources has significantly increased the interest in further disaggregating resources, enabling them to scale independently of one another as required. Consequently, cost-sharing via multi-tenancy and the benefits of pay-for-use billing policies in serverless computing have become pivotal in shaping the design of new database architectures. This paper provides a concise overview of database architecture evolution and their triggers for change, the contemporary state-of-the-art structures, and highlights open research problems for future exploration.

In database management systems (DBMSs) that handle multiple concurrent queries, adapting to fluctuating workloads is crucial. This flexibility allows the DBMS to revise decisions based on current workload and available resources. As memory availability changes with the arrival or completion of queries, having memory-intensive operators like the Hybrid Hash Join that dynamically adapt is vital. This paper introduces a new memory-adaptive Hash-Based join algorithm design implemented in Apache AsterixDB and evaluates its responsiveness to memory variability.

Hybrid Hash Join (HHJ) has proven to be one of the most efficient and widely-used join algorithms. While HHJ's performance depends largely on accurate statistics and information about the input relations, it may not always be practical or possible for a system to have such information available. HHJ's design depends on many details to perform well. This paper is an experimental and analytical study of the trade-offs in designing a robust and dynamic HHJ operator. We revisit the design and optimization techniques suggested by previous studies through extensive experiments, comparing them with other algorithms designed by us or used in related studies. We explore the impact of the number of partitions on HHJ's performance and propose a new lower bound for the number of partitions. We design and evaluate different partition insertion techniques to maximize memory utilization with the least CPU cost. Additionally, we consider a comprehensive set of algorithms for dynamically selecting a partition to spill and compare the results against previously published studies. We then present and evaluate two alternative growth policies for spilled partitions. These algorithms have been implemented in the context of Apache AsterixDB and evaluated under different scenarios such as variable record sizes, different distributions of join attributes, and different storage types, including HDD, SSD, and Amazon Elastic Block Store (Amazon EBS).

Benchmarks have always been one of the greatest assets in evaluating a Data Management System (DBMS) performance and providing a standard way to compare different DBMSs from various angles. As DBMSs evolve over time, new benchmarks are created, and the older ones need to advance and adapt in order to continue to be a valid and capable comparison tool. Many of the standard benchmarks use synthetic data and as such may not be able to capture the complexity of real data and its relationships correctly. The Wisconsin Benchmark was one of the first and main benchmarking tools made four decades ago by Dewitt et al at the University of Wisconsin. One of the most powerful features of this benchmark is that its relations are designed so their structure and distribution of attributes is easy to understand and control. While the Wisconsin Benchmark was a very powerful and widely-used benchmark years ago, it is given less attention in current studies. We believe that the Wisconsin Benchmark and its carefully designed relations can be utilized and provide capabilities that are unique and useful.. In this paper, we present a flexible, easy-to-use and scalable JSON Data Generator implemented in java based on the Wisconsin Benchmark Data Generator description, with more advanced features to provide relations and attributes closer to real-world data. Attribute skewness and variable length records using different size distributions are some of these newly added features. It is a ready-to-use, parameterized, and scalable data generator tool that since its development has been used in several AsterixDB's [4] performance benchmarking and publications [13,14,17,18], and we believe that can be useful to many others. The source code and more information are provided at [1].

Re-evaluating the Performance Trade-offs for Hash-Based Multi-Join Queries Share on Author: Shiva Jahangiri University of California, Irvine, Irvine, CA, USA University of California, Irvine, Irvine, CA, USASearch about this author Authors Info & Claims SIGMOD '20: Proceedings of the 2020 ACM SIGMOD International Conference on Management of DataJune 2020 Pages 2845–2847https://doi.org/10.1145/3318464.3384406Online:11 June 2020Publication History 1citation131DownloadsMetricsTotal Citations1Total Downloads131Last 12 Months44Last 6 weeks4 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

Summary Traditional relational database systems handle data by dividing their memory into sections such as a buffer cache and working memory, assigning a memory budget to each section to efficiently manage a limited amount of overall memory. They also assign memory budgets to memory‐intensive operators such as sorts and joins and control the allocation of memory to these operators; each memory‐intensive operator attempts to maximize its memory usage to reduce disk I/O cost. Implementing such memory‐intensive operators requires a careful design and application of appropriate algorithms that properly utilize memory. Today's Big Data management systems need the ability to handle large amounts of data similarly, as it is unrealistic to assume that truly big data will fit into memory. In this article, we share our memory management experiences in Apache AsterixDB, an open‐source Big Data management software platform that scales out horizontally on shared‐nothing commodity computing clusters. We describe the implementation of AsterixDB's memory‐intensive operators and their designs related to memory management. We also discuss memory management at the global (cluster) level. We conducted an experimental study using several synthetic and real datasets to explore the impact of this work. We believe that future Big Data management system builders can benefit from these experiences.

We present a new preprocessing algorithm for embedding the nodes of a given edge-weighted undirected graph into a Euclidean space. The Euclidean distance between any two nodes in this space approximates the length of the shortest path between them in the given graph. Later, at runtime, a shortest path between any two nodes can be computed with an A* search using the Euclidean distances as heuristic. Our preprocessing algorithm, called FastMap, is inspired by the data-mining algorithm of the same name and runs in near-linear time. Hence, FastMap is orders of magnitude faster than competing approaches that produce a Euclidean embedding using Semidefinite Programming. FastMap also produces admissible and consistent heuristics and therefore guarantees the generation of shortest paths. Moreover, FastMap applies to general undirected graphs for which many traditional heuristics, such as the Manhattan Distance heuristic, are not well defined. Empirically, we demonstrate that A* search using the FastMap heuristic is competitive with A* search using other state-of-the-art heuristics, such as the Differential heuristic.