Modern data-centric applications frequently need to store and read data with low latency. These requirements are difficult to achieve, even on high performance processors paired with fast solid state drives (SSDs). To this end, LSM tree is widely used in many systems such as in RocksDB and considered as an ideal index structure that fits SSDs. However, in spite of many improvements to LSM tree over the years, fundamental problems of limited read performance and expensive compaction operations remain. Microsoft Research proposed Bw-tree, a variant of B+ tree layered on top of log structured storage. Bw-tree achieves fast ingestion of data, similar to LSM tree, meanwhile it has less drawback on read performance and compaction. However, except for Microsoft, the industrial strength implementation of Bw-tree is rare. The open source OpenBw-Tree from Carnegie Mellon University was designed only for main memory. This paper describes Bwe-tree, an implementation and a significant evolution of Bw-tree on fast storage. It makes two contributions. First, Bwe-tree addresses reliability and performance issues revealed during running Bw-tree on fast storage in production, by revising structural modification operations, introducing page concurrency control, and storing large-size values off-tree. Performance improvements over Bw-tree are verified by experiments. Second, it demonstrates that Bw-tree is an effective alternative tree structure on SSDs. Compared to RocksDB (LSM tree) and BerkeleyDB (B+ tree), Bwe-vtree performs dramatically better (up to 3X or more) for the YCSB workloads. Our Bwe-vtree implementation has been integrated into production systems in Alibaba, including a flagshin cloud-native database service.

Logs are ubiquitous digital footprints, playing an indispensable role in system diagnostics, security analysis, and performance optimization. The extraction of actionable insights from logs is critically dependent on the log parsing process, which converts raw logs into structured formats for downstream analysis. Yet, the complexities of contemporary systems and the dynamic nature of logs pose significant challenges to existing automatic parsing techniques. The emergence of Large Language Models (LLM) offers new horizons. With their expansive knowledge and contextual prowess, LLMs have been transformative across diverse applications. Building on this, we introduce LogParser-LLM, a novel log parser integrated with LLM capabilities. This union seamlessly blends semantic insights with statistical nuances, obviating the need for hyper-parameter tuning and labeled training data, while ensuring rapid adaptability through online parsing. Further deepening our exploration, we address the intricate challenge of parsing granularity, proposing a new metric and integrating human interactions to allow users to calibrate granularity to their specific needs. Our method's efficacy is empirically demonstrated through evaluations on the Loghub-2k and the large-scale LogPub benchmark. In evaluations on the LogPub benchmark, involving an average of 3.6 million logs per dataset across 14 datasets, our LogParser-LLM requires only 272.5 LLM invocations on average, achieving a 90.6% F1 score for grouping accuracy and an 81.1% for parsing accuracy. These results demonstrate the method's high efficiency and accuracy, outperforming current state-of-the-art log parsers, including pattern-based, neural network-based, and existing LLM-enhanced approaches.

Persistent key-value stores play a crucial role in enabling internet-scale services. At Alibaba Cloud, scale-out cloud storage services including Object Storage Service, File Storage Service and Tablestore are built on distributed key-value stores. Key challenges in the design of the underlying key-value engine for these services lie in utilization of disaggregated storage, supporting write and range query-heavy workloads, and balancing of scalability, availability and resource usage. This paper presents ArkDB, a key-value engine designed to address these challenges by combining advantages of both LSM tree and Bw-tree, and leveraging advances in hardware technologies. Built on top of Pangu, an append-only distributed file system, ArkDB's innovations include shrinkable page mapping table, clear separation of system and user states for fast recovery, write amplification reduction, efficient garbage collection and lightweight partition split and merge. Experimental results demonstrate ArkDB's improvements over existing designs. Compared with Bw-tree, ArkDB efficiently stabilizes the mapping table size despite continuous write working set growth. Compared with RocksDB, an LSM tree-based key-value engine, ArkDB increases ingestion throughput by 2.16x, while reducing write amplification by 3.1x. It outperforms RocksDB by 52% and 37% respectively on a write-heavy workload and a range query-intensive workload of the Yahoo! Cloud Serving Benchmark. Experiments running in Tablestore in a cluster environment further demonstrate ArkDB's performance on Pangu and its efficient partition split/merge support.