Many cloud databases provide fine-grained regular snapshots and sparsely deleted snapshots based on importance, and dynamically maintain large-scale snapshots to ensure data security and mine the value of cold data. However, in existing snapshot technologies, the write amplification feature of Copy-on-Write (CoW) introduces additional expensive I/O operations in a cloud environment. In Redirect-on-Write (RoW), the modified data blocks are scattered among the snapshots, resulting in a dependency between the snapshots, which seriously affects the recovery performance. In this paper, we observed that access to snapshots has the characteristics of locality and continuity. We therefore propose an efficient Batch-Based Snapshot index, called BBS, which batches snapshot indexes according to database workload and access behavior of snapshots. Specifically, we use two key techniques: Shared-Subtrees Indexing and Batch-Based Dividing, to perform split dependency of the snapshot index. The snapshot index dependency chain is divided into batches, and there is no dependency on snapshot indexes between batches. In-batch snapshot indexes reduce memory overhead by sharing subtrees. The index can directly locate data blocks instead of iterative traversal. At the same time, the design of the snapshot index deletion method is adapted to the snapshot sparse deletion model. We have implemented a working system in Ceph. Evaluation results on datasets demonstrate that, compared with existing techniques, BBS can effectively balance the overhead between index memory capacity and recovery time.

Emerging persistent memory (PM) technologies offer data persistence at close-to-DRAM latency, causing the legacy multi-layer storage stack to become a bottleneck. On the other hand, PM improves I/O efficiency while also introducing unique hardware features that require software modifications to exploit them. In this paper, we propose HYTE, a native table storage engine for hybrid memory, which abstracts the data service for the database’s table. HYTE performs a cross-layer design across the file system and storage engine to avoid the pitfalls of layered abstraction and provides SQL-compatible APIs. Meanwhile, it co-designs with PM’s properties to unlock the hardware’s performance potential. Furthermore, HYTE is hotness-aware and equipped with a suite of lightweight, customized fetching and eviction strategies. Our evaluation shows that HYTE can perform up to 1.4-7.2× better than existing state-of-the-art PM-based storage systems in industry and academia.

With the rapid development of storage and network technology, emerging high-performance hardware is being widely applied to the distributed storage cluster. However, existing distributed storage systems employing multi-layer abstractions to provide table data services result in leaving high-speed hardware under-exploited. In this paper, we propose TEngine, a native distributed table storage engine designed for NVMe SSD and RDMA. The key is that TEngine removes the file abstraction to construct table structures on the device directly. For metadata service, TEngine designs a decoupled single metadata server, reducing distributed coordination, easing the burden on the metadata node, and enabling localized data node access. For data service, TEngine optimizes the parallel processing capability of NVMe devices by integrating upper-level multi-thread parallel operations with lower-level NVMe devices' parallel I/O processing. Moreover, TEngine introduces a periodic pull-based data synchronization approach to transform data pushing into periodic data pulling, which offloads the synchronization burden from the leader to the followers. The experimental results show that TEngine outperforms state-of-the-art distributed storage systems using the same hardware environment.

With the development of AI generated content and LLM (Large Language Model), demands of vector management have brought prosperity to vector databases. However, the status that vectors cannot be retrieved before being indexed, harms timeliness of vector databases. Updating indexes immediately when adding new vectors, reduces throughput of storage. Due to this contradiction, when facing streaming data, using vector database solely in vector services cannot have it both ways: real-time searches and high-throughput storage. This paper proposes a vector buffer engine, Neos. It is designed for real-time unindexed-vector searches on streaming input and buffering vectors with high throughput before loading them into vector databases. On one hand, we build a lightweight storage on raw NVMe device and liberate throughput from indexes, to maximize storage performance. On the other hand, we realize direct NVMe-GPUs 110 stack and a CPU-GPU heterogeneous task architecture for low-latency unindexed-vector searches on streaming data. Experiments show that our approach performs with 1.5x to 3.4x bandwidth, as low as 20% latency compared to existing 110 stacks, and up to orders-of-magnitude higher vector storage throughput under concurrent RIW workloads. Further, N eos can handle real-time unindexed - vector searches with millisecond-level latency on streaming input, a capability that current vector systems lack.

Embedded systems have evolved tremendously in recent years. We perform a study on SQLite and find that the multiple layers of abstraction drastically reduce bandwidth utilization. To minimize the bandwidth loss in the I/O path, we propose Lunar, a novel native table storage engine. Lunar performs a cross-layer design across the database and file system to avoid the pitfalls of multi-layer abstraction while providing SQL-compatible APIs. It employs a type-aware storage layout that considers the access patterns of different data types. Then, Lunar designs a variable-size allocator to reduce fragmentation and optimize RAM and I/O bandwidth usage. Further, considering the limited resources on embedded devices, Lunar employs a modular architecture that enables selecting modules on demand. It also offers optional consistency modes to make a trade-off between resource consumption and consistency. Experiments show that Lunar achieves higher bandwidth utilization, outperforming state-of-the-art approaches while consuming fewer resources.