Shared-nothing geo-distributed SQL databases, such as CockroachDB, are increasingly vital for enterprise applications requiring data resilience and locality. However, we encountered significant performance degradation at the customer side, especially when their deployments span multiple data centers over a Wide Area Network (WAN). Our investigation identifies the bottleneck in the performance of the Distributed Hash Join (Dist-HJ) algorithm, which is contingent upon a crucial balance between communication overhead and computational load. This balance is severely disrupted when processing skewed data from real-world customer workloads, leading to the observed performance decline. To tackle this challenge, we introduce Bala-Join, an adaptive solution to balance the computation and network load in Dist-HJ execution. Our approach consists of the Balanced Partition and Partial Replication (BPPR) algorithm and a distributed online skewed join key detector. The former achieves balanced redistribution of skewed data through a multicast mechanism to improve computational performance and reduce network overhead. The latter provides real-time skewed join key information tailored to BPPR. Furthermore, an Active-Signaling and Asynchronous-Pulling (ASAP) mechanism is incorporated to enable efficient, real-time synchronization between the detector and the redistribution process with minimal overhead. Empirical study shows that Bala-Join outperforms the popular Dist-HJ solutions, increasing throughput by 25%-61%.

Modern database systems rely on workload forecasting to guide performance tuning, resource allocation, and query optimization. However, evolving query patterns and resource demands make accurate workload forecasting increasingly challenging. Most existing methods separately model query and resource behaviors using simple statistics or single-scale forecasting models, thereby ignoring their inherent correlations and leading to inaccurate modeling of real-world workload behaviors. In practice, real-world workloads show that query and resource behaviors are often interdependent and exhibit temporal dynamics. However, most forecasting methods overlook the inherent correlations between query and resource, as well as the temporal patterns across multi-scale resolutions in real workloads. To address these issues, we propose QuadraFormer, an end-to-end forecasting framework that jointly predicts query and resource usage. First, we introduce a unified workload representation that integrates query and resource information into a structured input, enabling joint modeling of workload across variates. Second, we develop an adaptive multi-scale routing mechanism that selects appropriate temporal resolutions to capture periodic patterns and multi-scale variability. Third, we design a novel Quadra-attention mechanism that models both intra- and inter-patch temporal dependencies as well as intra- and inter-dimensional dependencies. Extensive experiments on real-world workloads show that QuadraFormer consistently outperforms competitive baselines, achieving the highest F1 score of 98.38% and accuracy of 86.17%, with F1 improvements of up to 10.28% over the best-performing baseline, while reducing training time by up to 2.16×.

Learned indexes, emerging as a promising alternative to traditional indexes like B+Tree, utilize machine learning models to enhance query performance and reduce memory usage. However, the widespread adoption of learned indexes is limited by their expensive training cost and the need for high accuracy of internal models. Although some studies attempt to optimize the building process of these learned indexes, existing methods are restrictive in scope and applicability. They are usually tailored to specific index types and heavily rely on pre-trained model knowledge, making deployment a challenging task. In this work, we introduce the Learned Index Optimization Framework (LIOF), a general and easily integrated solution aimed at expediting the training process and improving the accuracy of index model for one-dimensional and multi-dimensional learned indexes. The optimization of LIOF for the learned indexes is intuitive, directly providing optimized parameters for index models based on the distribution of node data. By leveraging the correlation between key distribution and node model parameters, LIOF significantly reduces the training epochs required for each node model. Initially, we introduce an optimization strategy inspired by optimization-based meta-learning to train the LIOF to generate optimized initial parameters for index node models. Subsequently, we present a data-driven encoder and a parameter-centric decoder network, which adaptively translate key distribution into a latent variable representation and decode it into optimized node model initialization. Additionally, to further utilize characteristics of key distribution, we propose a monotonic regularizer and focal loss, guiding LIOF training towards efficiency and precision. Through extensive experimentation on real-world and synthetic datasets, we demonstrate that LIOF provides substantial enhancements in both training efficiency and the predictive accuracy for learned indexes.

Various works have utilized deep learning to address the query optimization problem in database system. They either learn to construct plans from scratch in a bottom-up manner or steer the plan generation behavior of traditional optimizer using hints. While these methods have achieved some success, they face challenges in either low training efficiency or limited plan search space. To address these challenges, we introduce FOSS, a novel framework for query optimization based on deep reinforcement learning. FOSS initiates optimization from the original plan generated by a traditional optimizer and incrementally refines suboptimal nodes of the plan through a sequence of actions. Additionally, we devise an asymmetric advantage model to evaluate the advantage between two plans. We integrate it with a traditional optimizer to form a simulated environment. Leveraging this simulated environment, FOSS can bootstrap itself to rapidly generate a large amount of high-quality simulated experiences. FOSS then learns from these experiences to improve its optimization capability. We evaluate the performance of FOSS on Join Order Benchmark, TPC-DS, and Stack Overflow. The experimental results demonstrate that FOSS outperforms the state-of-the-art methods in terms of latency performance. Compared to PostgreSQL, FOSS achieves speedup ranging from 1.15x to 8.33x in total latency across different benchmarks.

Storage space usage forecasting is critical for the scalability and stability of storage systems. Cloud providers estimate storage usages based on the forecast and allocate resources accordingly. Overestimated space usages require a redundant storage buffer that brings unnecessary cost, and underestimated space usages will cause capacity shortages that may lead to data loss and Service-Level Agreement (SLA) failures. While accurate storage forecasting is important, it is highly challenging due to various storage usage patterns on different workloads and storage systems. Moreover, some operations from users or administrators may cause transient workload burst in historical data, which makes forecasting even harder.In this paper, we propose the Storage Usage Forecasting Service (SUFS) that combines deep neural networks and statistical models adaptively to make predictions for multiple major storage systems in ByteDance. SUFS carries comprehensive analyses of storage usage time series from various storage systems in real business scenarios. To handle workload bursts in historical data, we enhance regular LSTMs using a control signal that is installed on the input gate. When the burst is detected, the control signal reduces the input influences to the cell state. To further improve the prediction accuracy, SUFS integrates the Enhanced-LSTM (ELSTM) with a novel adaptive ensemble method. Different from previous works, our approach learns dynamic ensemble weights for each prediction step on-the-fly, making our model more accurate for multiple-step predictions. SUFS has been deployed to serve more than 150,000 storage instances. We conducted extensive experiments on the storage systems that are widely-used in ByteDance, and the results show that SUFS outperforms the state-of-the-art methods and significantly reduces storage cost.

The accuracy of selectivity estimation is of vital importance to create good query plans. Traditional estimators such as histograms make several assumptions during estimation that can lead to huge errors. Recently the database community started exploring the usage of machine learning in selectivity estimation and won great achievements. However, due to the black box models they used, existing learning-based methods still face several new challenges, including high estimation latency, large training data demanding, and occurrence of illogical results. In this work, we propose a learning-based MOnotonic Selectivity Estimator (MOSE) to address these challenges. We first learn a multi-dimensional cumulative distribution function of the data in a supervised method and then compute selectivity for ad hoc query predicates at rum-time. We propose a novel regularizer and an effective attribute-aware calibration method to improve the estimation accuracy. To further improve the model efficiency, we design a mutual information based model ensemble method. With regard to the heavy cost of training data collection, we design a model-based active learning strategy to generate high-quality training data cost-effectively. We conduct extensive experiments on both real-world and synthetic datasets and the results show that MOSE outperforms the state-of-the-art methods in terms of accuracy and efficiency.

The accuracy of selectivity estimation is of vital importance to create good query plans in database management systems. We propose MOSE, a learning-based MOnotonic Selectivity Estimator, to provide accurate, reliable, and efficient selectivity estimation for query optimization.

Query optimization is crucial for the query performance of database systems. Despite decades of efforts from both research and industrial communities, query optimization remains one of the most challenging problems. Thanks to the advances in artificial intelligence, data-driven and learning-based techniques are seeing traction in database research recently. However, most former learning-based works perform less practical because they are evasive about the interaction between learning components and database systems.