In this paper, we describe veDB-HTAP, a highly integrated, efficient, and adaptive HTAP system recently built in ByteDance. veDB-HTAP adopts a highly integrated system architecture by leveraging the Secondary Engine mechanism provided by MySQL and provides a seamless query processing experience across OLTP and OLAP engines. In addition, we introduce a cost-based and machine-learning-based smart query router that significantly outperforms the rule-based query router used in ByteHTAP, a precursor of veDB-HTAP. A key design principle of veDB-HTAP is the collaboration and adaptability of major system components, including query planning, query execution, and unified storage. Our adaptive query execution can be classified into two categories: 1) adaptive execution that dynamically collects and utilizes runtime statistics for better query performance; 2) utilizing runtime resource information to achieve a high quality of service even under heavy workloads. The experiments show that veDB-HTAP can achieve more than 3× speedup for TPC-H while consuming only one-third of the resources compared to ByteHTAP.

The ubiquity of variable-length integers in data storage and communication necessitates efficient decoding techniques. In this paper, we present SFVInt, a simple and fast approach to decode the prevalent Little Endian Base-128 (LEB128) varints. Our approach effectively utilizes the Bit Manipulation Instruction Set 2 (BMI2) in modern Intel and AMD processors, achieving significant performance improvement while maintaining simplicity and avoiding overengineering. SFVInt, with its generic design, effectively processes both 32-bit and 64-bit unsigned integers using a unified code template, marking a significant leap forward in varint decoding efficiency. We thoroughly evaluate SFVInt's performance across various datasets and scenarios, demonstrating that it achieves up to a 2x increase in decoding speed when compared to varint decoding methods used in established frameworks like Facebook Folly and Google Protobuf.

Shared-nothing, distributed databases scale transactional and analytical processing over a large data volume by spreading data across servers. However, static sharding of data across nodes makes such systems fail to timely adapt to changing workloads and struggle to obey the cloud pay-as-you-go model. Migrating shards between nodes online is a key technique to react to dynamic changes of workloads for cloud elasticity. Existing approaches introduce severely degraded performance and service interruption, resulting in SLA violation on the cloud; or they are tailor-made to deterministic databases. In this paper, we propose Remus, a new live migration approach for shared-nothing, distributed databases with snapshot isolation. Remus migrates shards between nodes with zero service interruption and minimal performance impact. This is achieved by an efficient unidirectional dual execution during migration. We implement Remus on a shared-nothing, distributed version of PolarDB-PG and evaluate it against state-of-the-art approaches using standard OLTP workloads TPC-C and YCSB, and hybrid workloads consisting of long-lived and short transactions. The results demonstrate Remus is the only effective approach to achieve the goal of zero transaction interruption, zero downtime and marginal performance impact, paving the way for applying the shared-nothing architecture to a cloud database which needs to provide elasticity while guaranteeing strict SLAs.

Modern distributed database systems scale horizontally by partitioning their data across a large number of nodes. Most such systems build their transactional layers on a replication layer, employing a consensus protocol to ensure data consistency to achieve fault tolerance. Synchronization among replicated state machines thus becomes a significant overhead of transaction processing. Without careful design, synchronization could amplify transactions' lock duration and impair the system's scalability. Speculative techniques, such as Controlled Lock Violation (CLV) and Early Lock Release (ELR), prove useful in shortening lock's critical path and boosting transaction processing performance. To use these techniques to optimize geo-replicated distributed databases(GDDB) is an intuitive idea. This paper shows that a naive application of speculation is often unhelpful in a distributed environment. Instead, we introduce Distributed Lock Violation (DLV), a specialized speculative technique for geo-replicated distributed databases. DLV can achieve good performance without incurring severe side effects.

Huawei Fusion Insight Libr A (FI-MPPDB) is a petabyte scale enterprise analytics platform developed by the Huawei data-base group. It started as a prototype more than five years ago, and is now being used by many enterprise customers over the globe, including some of the world's largest financial institutions. Our product direction and enhancements have been mainly driven by customer requirements in the fast evolving Chinese market. This paper describes the architecture of FI-MPPDB and some of its major enhancements. In particular, we focus on top four requirements from our customers related to data analytics on the cloud: system availability, auto tuning, query over heterogeneous data models on the cloud, and the ability to utilize powerful modern hardware for good performance. We present our latest advancements in the above areas including online expansion, auto tuning in query optimizer, SQL on HDFS, and intelligent JIT compiled execution. Finally, we present some experimental results to demonstrate the effectiveness of these technologies.

This paper presents an energy-conservation method for multiple disks and their cache memory. Our method periodically resizes the cache memory and controls the rotation speeds under performance constraints. The cache memory stores the data from the disks for reuse. Enlarging the cache memory reduces disk accesses and disk utilization. This allows the disks to reduce their speeds and conserve energy because the disks' power consumption is quadratic to their speeds. However, the cache memory itself consumes power to retain data. Shrinking cache memory can save memory power while increasing disk accesses and degrading performance. Choosing proper cache sizes and rotation speeds can reduce the energy consumption of both memory and disks with satisfactory performance. We model cache resizing and speed setting as an optimization problem with minimizing the power consumption as objective and limiting disk utilization as constraints. We compare our method with the methods resizing cache based on request rates. The simulation results show that our method achieves better energy savings while limiting disk access latency.

This paper presents a method that uses data buffers to create long periods of idleness to exploit power management. This method considers the power consumed by the buffers and assigns an energy penalty for buffer underflow. Our approach provides analytic formulas for calculating the optimal buffer sizes without subjective or heuristic decisions. We simulate four different hardware configurations with MPEG-1, MPEG-2, and MPEG-4 formats as a case study. Our results indicate that the optimal buffer size varies significantly for different data formats on different hardware. Simulation results indicate that 16 MB buffers are sufficient for MPEG-1, MPEG-2, and MPEG-4 video streams from a microdrive or a network card, but transfers from an IDE disk require buffer sizes ranging from 16 MB to 176 MB, depending on each video's statistical properties.

This paper presents a method to combine memory resizing and disk shutdown to achieve better energy savings than can be achieved individually. The method periodically adjusts the size of physical memory and the timeout value to shut down a hard disk to reduce the average energy consumption. Pareto distributions are used to model the disk idle time. The parameters of the distributions are estimated at runtime and used to calculate the appropriate timeout value. The memory size is changed based on the predicted number of disk accesses at different memory sizes. The method also considers the delay caused by power management and limits the performance degradation. The method is simulated and compared with other power management methods. Simulation results show that the method consistently achieves better energy savings and less performance degradation across different workloads

The paper presents a scheme to combine memory and power management for achieving better energy reduction. Our method periodically adjusts the size of physical memory and the timeout value to shut down a hard disk for reducing the average power consumption. We use Pareto distributions to model the distributions of idle time. The parameters of the distributions are adjusted at run-time for calculating the corresponding timeout value of the disk power management. The memory size is changed based on the inclusion property to predict the number of disk accesses at different memory sizes. Experimental results show more than 50% energy savings compared to a 2-competitive fixed-timeout method.

This work presents a new approach for energy management by inserting data buffers. The buffers are managed using the concept of inventory control by treating energy as cost and computation as production. The data stored in the buffers are considered as merchandise. Our method provides a mathematical framework to develop general strategies managing the energy consumption in computers. The approach can solve a wide range of problems. For example, it can calculate the needed buffer sizes to achieve optimal energy savings. It can derive the conditions when a standby state saves energy. The method also shows the effect of load balancing on energy conservation and can handle the variations of data rates. We use sensor networks as case studies and demonstrate more than 20% energy savings.

This paper presents a method to reduce energy consumption by inserting data buffers. The method determines whether power can be reduced by inserting a buffer between two components and periodically turning off one of them. This method calculates the length of the period and the required buffer size to achieve the optimal energy savings. Our approach can be applied to any applications whose data arrival and departure rates are different and known in advance.

This paper presents a method that uses data buffers to smoothen request variations and to create long idleness for power management. This method considers the power consumed by the buffers and assigns an energy penalty for buffer underflow. Our approach provides analytic formulas for calculating the optimal buffer sizes and the amount of data to store in the buffers. We use video prefetching as a case study and obtain power savings of more than 74% for MPEG-1 and 34% for MPEG-2 videos.