Demo Data from SIGIR 2026 / reviewers in the wild / expert
Anqun Pan
dblp:224/6370
· DBLP profile ↗
35ranked-venue papers
1as first author
31since 2021 · last 2026
0000-0002-6756-149XORCID · verified
Domains — the database's venue-derived domains; a paper can count in several
Databases, data management, data science and information retrieval · 28 · 1 first-author · 25 since 2021Systems, architecture and hardware · 3 · 3 since 2021Applied, interdisciplinary, general and emerging computing · 3 · 2 since 2021Security and privacy · 1 · 1 since 2021
Expertise — from the expertise taxonomy: the topics of the expert's papers under the CCF categories. A weight counts papers with recency: 1 for a paper about the topic, 0.3 when the topic is its context, halved every five years.
| Databases, data mining, and information retrieval
24 papers |
Transaction processing and concurrency control · 56% Indexing and storage engines · 15% Data stream processing · 6% | |
| Computer architecture, parallel and distributed computing, and storage systems
8 papers |
Storage systems · 46% Memory systems · 28% Distributed systems · 11% | |
| Theoretical computer science
2 papers |
Distributed computing theory · 40% Algorithms and data structures · 30% Graph algorithms and graph theory · 30% |
Topics — the 30 heaviest of 69, each with the papers that count most for it
| Topic | Weight | Papers | Last | Evidence papers |
|---|---|---|---|---|
Transaction processing and concurrency control
distributed transaction processing |
2.4 | 3 | 2025 | GeoTP: Latency-Aware Geo-Distributed Transaction Processing in Database Middlewares · ICDE 2025 TDSQL: Tencent Distributed Database System · Proc. VLDB Endow. 2024 Lion: Minimizing Distributed Transactions Through Adaptive Replica Provision · ICDE 2024 |
Transaction processing and concurrency control
isolation level verification |
1.7 | 2 | 2025 | Boosting End-to-End Database Isolation Checking via Mini-Transactions · ICDE 2025 Online Timestamp-Based Transactional Isolation Checking of Database Systems · ICDE 2025 |
Transaction processing and concurrency control › concurrency control
multiversion concurrency control |
1.7 | 2 | 2026 | AKV: Agile Read-Efficiently Key-Value OLTP Engine for Non-Volatile Memory · IEEE Trans. Knowl. Data Eng. 2026 T-SQL: A Lightweight Implementation to Enable Built-in Temporal Support in MVCC-Based RDBMSs · IEEE Trans. Knowl. Data Eng. 2023 |
Transaction processing and concurrency control
serializability |
1.5 | 2 | 2025 | Online Timestamp-Based Transactional Isolation Checking of Database Systems · ICDE 2025 Efficiently Supporting Multi-Level Serializability in Decentralized Database Systems · IEEE Trans. Knowl. Data Eng. 2023 |
Transaction processing and concurrency control › isolation levels
snapshot isolation |
1.5 | 2 | 2025 | Online Timestamp-Based Transactional Isolation Checking of Database Systems · ICDE 2025 Efficient Black-box Checking of Snapshot Isolation in Databases · Proc. VLDB Endow. 2023 |
Indexing and storage engines
learned index |
1.4 | 2 | 2024 | G-Learned Index: Enabling Efficient Learned Index on GPU · IEEE Trans. Parallel Distributed Syst. 2024 SALI: A Scalable Adaptive Learned Index Framework based on Probability Models · Proc. ACM Manag. Data 2023 |
Transaction processing and concurrency control
isolation levels |
1.1 | 2 | 2025 | Boosting End-to-End Database Isolation Checking via Mini-Transactions · ICDE 2025 Efficient Black-box Checking of Snapshot Isolation in Databases · Proc. VLDB Endow. 2023 |
Memory systems
non-volatile memory |
1.1 | 2 | 2026 | FluidKV: Seamlessly Bridging the Gap between Indexing Performance and Memory-Footprint on Ultra-Fast Storage · Proc. VLDB Endow. 2024 AKV: Agile Read-Efficiently Key-Value OLTP Engine for Non-Volatile Memory · IEEE Trans. Knowl. Data Eng. 2026 |
Debugging and program repair
fault localization |
1.0 | 1 | 2026 | Pisco: An Isolation Bug Case Reduction and Deduplication Framework · Proc. VLDB Endow. 2026 |
Storage systems
key-value storage |
1.0 | 1 | 2026 | AKV: Agile Read-Efficiently Key-Value OLTP Engine for Non-Volatile Memory · IEEE Trans. Knowl. Data Eng. 2026 |
Indexing and storage engines
key-value store |
1.0 | 2 | 2024 | FluidKV: Seamlessly Bridging the Gap between Indexing Performance and Memory-Footprint on Ultra-Fast Storage · Proc. VLDB Endow. 2024 T-SQL: A Lightweight Implementation to Enable Built-in Temporal Support in MVCC-Based RDBMSs · IEEE Trans. Knowl. Data Eng. 2023 |
Transaction processing and concurrency control › isolation level verification
black-box isolation checking |
0.9 | 1 | 2025 | Boosting End-to-End Database Isolation Checking via Mini-Transactions · ICDE 2025 |
Transaction processing and concurrency control
concurrency control |
0.9 | 1 | 2025 | A Hybrid Approach to Integrating Deterministic and Non-deterministic Concurrency Control in Database Systems · Proc. VLDB Endow. 2025 |
Indexing and storage engines
hash index |
0.9 | 1 | 2025 | Shard: A Scalable and Resize-optimized Hash Index on Disaggregated Memory · Proc. VLDB Endow. 2025 |
Transaction processing and concurrency control › contention management
lock contention |
0.9 | 1 | 2025 | GeoTP: Latency-Aware Geo-Distributed Transaction Processing in Database Middlewares · ICDE 2025 |
Transaction processing and concurrency control › concurrency control
optimistic concurrency control |
0.9 | 1 | 2025 | A Hybrid Approach to Integrating Deterministic and Non-deterministic Concurrency Control in Database Systems · Proc. VLDB Endow. 2025 |
Transaction processing and concurrency control › serializability
serializable isolation |
0.9 | 1 | 2025 | TxnSails: Achieving Serializable Transaction Scheduling with Self-Adaptive Isolation Level Selection · Proc. VLDB Endow. 2025 |
Transaction processing and concurrency control › serializability
strict serializability |
0.9 | 1 | 2025 | Boosting End-to-End Database Isolation Checking via Mini-Transactions · ICDE 2025 |
Transaction processing and concurrency control
transaction isolation |
0.9 | 1 | 2025 | Fast Verification of Strong Database Isolation · Proc. VLDB Endow. 2025 |
Memory systems
memory disaggregation |
0.9 | 1 | 2025 | Shard: A Scalable and Resize-optimized Hash Index on Disaggregated Memory · Proc. VLDB Endow. 2025 |
Distributed systems › replication
replicated data types |
0.9 | 1 | 2025 | A Generic Specification Framework for Weakly Consistent Replicated Data Types · IEEE Trans. Parallel Distributed Syst. 2025 |
Distributed computing theory
consistency model |
0.9 | 1 | 2025 | A Generic Specification Framework for Weakly Consistent Replicated Data Types · IEEE Trans. Parallel Distributed Syst. 2025 |
Data stream processing
adaptive compression |
0.8 | 1 | 2024 | Data-Aware Adaptive Compression for Stream Processing · IEEE Trans. Knowl. Data Eng. 2024 |
Data stream processing
continuous query processing |
0.8 | 1 | 2024 | Data-Aware Adaptive Compression for Stream Processing · IEEE Trans. Knowl. Data Eng. 2024 |
Data integration and cleaning
database integration |
0.8 | 1 | 2024 | Compressed Data Direct Computing for Databases · IEEE Trans. Knowl. Data Eng. 2024 |
Transaction processing and concurrency control › distributed transaction processing
distributed transaction reduction |
0.8 | 1 | 2024 | Lion: Minimizing Distributed Transactions Through Adaptive Replica Provision · ICDE 2024 |
Information retrieval › indexing
index design |
0.8 | 1 | 2024 | IndeXY: A Framework for Constructing Indexes Larger than Memory · ICDE 2024 |
Query processing and optimization › query optimization › transformation-based optimization
query pushdown |
0.8 | 1 | 2024 | Compressed Data Direct Computing for Databases · IEEE Trans. Knowl. Data Eng. 2024 |
Distributed and cloud data management › data replication
replica placement |
0.8 | 1 | 2024 | Lion: Minimizing Distributed Transactions Through Adaptive Replica Provision · ICDE 2024 |
Data stream processing
stream processing systems |
0.8 | 1 | 2024 | Data-Aware Adaptive Compression for Stream Processing · IEEE Trans. Knowl. Data Eng. 2024 |
Methods — techniques the papers use, named apart from their topics
read abort optimization · 2.0circular dual-version storage · 2.0formal specification · 1.7delta debugging · 1.0unified concurrency control · 0.9timestamp-based checking · 0.9sharding · 0.9resize optimization · 0.9randomized testing · 0.9mini-transactions · 0.9lock-sharing · 0.9incremental dependency graph construction · 0.9heuristic optimization · 0.9global validation · 0.9deep learning · 0.9decentralized prepare · 0.9cross-isolation validation · 0.9two-stage stripe write · 0.8
| Year | Publication | Venue | Position |
|---|---|---|---|
| 2026 | Doux: Decoupling Values from Keys for Real-Time Analytics
Shiming Yang, Shuang Liu 0007, Wei Lu 0015, Kuien Liu, Yuxing Chen 0003, Anqun Pan, Lixiong Zheng, Xiaoyong Du 0001 |
ICDE | 7 |
| 2026 | Efficient Query Rewrite Rule Discovery via Standardized Enumeration and Learning-to-RankabstractQuery rewriting is essential for database performance optimization, but existing automated rule enumeration methods suffer from exponential search spaces, severe redundancy, and poor scalability, especially when handling complex query plans with five or more nodes, where a node represents an operator in the plan tree. We present SLER, a scalable system that enables efficient and effective rewrite rule discovery by combining standardized template enumeration with a learning to rank approach. SLER uses standardized templates, abstractions of query plans with operator structures preserved but data specific details removed, to eliminate structural redundancies and drastically reduce the search space. A learn to rank model guides enumeration by pre filtering the most promising template pairs, enabling scalable rule generation for large node templates. Evaluated on over 11000 real world SQL queries from both open source and commercial workloads, SLER has automatically constructed a rewrite rule repository exceeding 1 million rules - the largest empirically validated rewrite rule library to date. Notably, at the scale of one million rules, SLER supports query plan templates with complexity up to channel level depth. This unprecedented scale opens the door to discovering highly intricate transformations across diverse query patterns. Critically, SLER's template driven design and learned ranking mechanism are inherently extensible, allowing seamless integration of new and complex operators, paving the way for next generation optimizers powered by comprehensive, adaptive rule spaces. Yuxing Chen 0003, Yuekun Yu, Jinbin Huang, Rui Mao 0001, Anqun Pan, Lixiong Zheng, Jianbin Qin |
ICDE | 6 |
| 2026 | Telescope: A Learned What-If Call for Column Store Selection in HTAP Databases
Zhengkun Wu, Ju Fan, Anqun Pan |
ICDE | 8 |
| 2026 | Pisco: An Isolation Bug Case Reduction and Deduplication Framework
Siyang Weng, Zirui Hu, Rong Zhang 0002, Chengcheng Yang, Xuan Zhou 0001, Yuxing Chen 0003, Anqun Pan |
Proc. VLDB Endow. | 10 |
| 2026 | AKV: Agile Read-Efficiently Key-Value OLTP Engine for Non-Volatile MemoryabstractNon-volatile memory (NVM), as an emergingstor age technology, offers several advantageous features for OLTP engines, including byte-addressability, high capacity, low energy consumption, and data persistence across power failures. Despite these benefits, the current mainstream OLTP engines still commonly adopt a hybrid architecture that deeply couples DRAM with NVM, which results in a complex system architecture and high recovery costs. In this paper, we aim to construct a highly available, stable, and recoverable OLTP engine that guarantees ACID properties through anagile system architecture. We introduce AKV (Agile Key-Value), an NVM-only OLTP storage engine designed to provide effective space utilization, high throughput, and fast failure recovery. AKV addresses the challenges of NVM space management, write redundancy, and concurrency control with two novel techniques: dual-version concurrency control and circular dual-version storage. Experimental results demonstrate that AKV achieves higher throughput (up to 69.7%) and faster recovery (up to 54×) compared to existing storage engines in most scenarios of the TPC-C benchmarks. Additionally, the codebase of AKV (4k+ lines) is more concise than that of SOTA OLTP engines like Zen (8k+ lines) and Falcon (11k+ lines). In addition, this study innovatively proposes a read abort optimization strategy based on dynamic version changes. The experimental results show that this strategy can significantly reduce the transaction abort rate of AKV in specific workload scenarios while maintaining stable system throughput, achieving a maximum reduction of up to 73% in the abort count. Jianbin Qin, Tianyu Wang 0009, Yuxing Chen 0003, Anqun Pan, Rui Mao 0001, Yu-Xuan Qiu, Makoto Onizuka, Chuan Xiao 0001 |
IEEE Trans. Knowl. Data Eng. | 5 |
| 2025 | Online Timestamp-Based Transactional Isolation Checking of Database SystemsabstractSerializability (SER) and snapshot isolation (SI) are widely used transactional isolation levels in database systems. The isolation checking problem asks whether a given execution history of a database system satisfies a specified isolation level. However, existing SER and SI checkers, whether traditional black-box checkers or recent timestamp-based white-box ones, operate offline and require the entire history to be available to construct a dependency graph, making them unsuitable for continuous and ever-growing histories. This paper addresses online isolation checking by extending the timestamp-based isolation checking approach to online settings. Specifically, we design Chronos, an efficient timestamp-based offline SI checker. Chronos is incremental and avoids constructing a start-ordered serialization graph for the entire history, making it well-suited for online scenarios. We further extend Chronos into an online SI checker, Aion, addressing several key challenges unique to online settings. Additionally, we develop Aion-SER for online SER checking. Experiments highlight that Chronos processes offline histories with up to one million transactions in seconds, greatly outperforming existing SI checkers. Furthermore, Aion and Aion-ser sustain a throughput of approximately 12K transactions per second, demonstrating their practicality for online isolation checking. Hexu Li, Hengfeng Wei, Hongrong Ouyang, Yuxing Chen 0003, Ruohao Zhang, Anqun Pan |
ICDE | 7 |
| 2025 | Boosting End-to-End Database Isolation Checking via Mini-TransactionsabstractTransactional isolation guarantees are crucial for database correctness. However, recent studies have uncovered numerous isolation bugs in production databases. The common black-box approach to isolation checking stresses databases with large, concurrent, randomized transaction workloads and verifies whether the resulting execution histories satisfy specified isolation levels. For strong isolation levels such as strict serializability, serializability, and snapshot isolation, this approach often incurs significant end-to-end checking overhead during both history generation and verification. We address these inefficiencies through the novel design of Mini-Transactions (MTs). MTs are compact, short transactions that execute much faster than general workloads, reducing overhead during history generation by minimizing database blocking and transaction retries. By leveraging MTs' read-modify-write pattern, we develop highly efficient algorithms to verify strong isolation levels in linear or quadratic time. Despite their simplicity, MTs are semantically rich and effectively capture common isolation anomalies described in the literature. We implement our verification algorithms and an MT workload generator in a tool called MTC. Experimental results show that MTC outperforms state-of-the-art tools in both history generation and verification. Moreover, MTC can detect bugs across various isolation levels in production databases while maintaining the effectiveness of randomized testing with general workloads, making it a cost-effective solution for black-box isolation checking. Hengfeng Wei, Si Liu 0003, Zijing Yin, Yuxing Chen 0003, Anqun Pan |
ICDE | 7 |
| 2025 | GeoTP: Latency-Aware Geo-Distributed Transaction Processing in Database MiddlewaresabstractThe widespread adoption of database middleware for supporting distributed transaction processing is prevalent in numerous applications, with heterogeneous data sources deployed across national and international boundaries. However, transaction processing performance significantly drops due to the high network latency between the middleware and data sources and the long lock contention span, where transactions may be blocked while waiting for the locks held by concurrent transactions. In this paper, we propose GeoTP, a latency-aware geo-distributed transaction processing approach in database middleware. GeoTP incorporates three key techniques to enhance performance in geo-distributed scenarios. First, we propose a decentralized prepare mechanism to reduce network round-trips for distributed transactions. Second, we design a latency-aware scheduler to minimize the lock contention span by strategically delaying the lock acquisition. Third, heuristic optimizations are proposed for the scheduler to reduce the lock contention span further. We implemented GeoTP on Apache Shardingsphere, a state-of-the-art middleware, and extended it into Apache ScalarDB. Experimental results on YCSB and TPC-C demonstrate that GeoTP achieves up to 17.7x performance improvement. Qiyu Zhuang, Shuang Liu 0007, Wei Lu 0015, Zhanhao Zhao, Yuxing Chen 0003, Tong Li 0014, Anqun Pan, Xiaoyong Du 0001 |
ICDE | 8 |
| 2025 | Fast Verification of Strong Database Isolation
Zhiheng Cai, Si Liu 0003, Hengfeng Wei, Yuxing Chen 0003, Anqun Pan |
Proc. VLDB Endow. | 5 |
| 2025 | A Hybrid Approach to Integrating Deterministic and Non-deterministic Concurrency Control in Database SystemsabstractDeterministic and non-deterministic concurrency control algorithms have shown respective advantages under diverse workloads. Thus, a natural idea is to blend them together. However, because deterministic algorithms work with stringent assumptions, e.g., batched execution and non-interactive transactions, they hardly work together with non-deterministic algorithms. To address this issue, we propose HDCC, a hybrid approach that adaptively employs Calvin and OCC, which have distinct concurrency control and logging schemes, in the same database system. To ensure serializability and recovery correctness, we introduce lock-sharing, global validation, and two-log-interleaving mechanisms. Additionally, we introduce a rule-based assignment mechanism to dynamically select Calvin or OCC based on workload characteristics. Experimental results using TPC-C and YCSB benchmarks demonstrate that HDCC surpasses existing hybrid approaches by up to 3.1×. Yinhao Hong, Hongyao Zhao, Wei Lu 0015, Xiaoyong Du 0001, Yuxing Chen 0003, Anqun Pan, Lixiong Zheng |
Proc. VLDB Endow. | 6 |
| 2025 | Shard: A Scalable and Resize-optimized Hash Index on Disaggregated Memory
Hantian Zha, Teng Ma 0006, Baotong Lu, Yuansen Wang, Dongbiao He, Yuanhui Luo, Yunpeng Chai, Yuxing Chen 0003, Anqun Pan |
Proc. VLDB Endow. | 10 |
| 2025 | TxnSails: Achieving Serializable Transaction Scheduling with Self-Adaptive Isolation Level SelectionabstractAchieving the serializable isolation level is costly. Recent studies have revealed that adjusting specific query patterns within the workload can still achieve serializability, even at lower isolation levels. Nevertheless, these studies typically overlook the trade-off between the performance advantages of lower isolation levels and the overhead required to maintain serializability, potentially leading to suboptimal isolation level choices that fail to maximize performance. In this paper, we present TxnSails, a middle-tier solution designed to achieve serializable scheduling with self-adaptive isolation level selection. First, TxnSails incorporates a unified concurrency control algorithm that achieves serializability at lower isolation levels with minimal overhead. Second, TxnSails employs a deep learning method to characterize the trade-off between the performance benefits and overhead associated with lower isolation levels, thus predicting the optimal isolation level. Finally, TxnSails implements a cross-isolation validation mechanism to ensure serializability during real-time isolation level transitions. Extensive experiments demonstrate that TxnSails outperforms existing solutions by up to 26.7× and PostgreSQL 's serializable isolation level by up to 4.8×. Qiyu Zhuang, Wei Lu 0015, Shuang Liu 0007, Yuxing Chen 0003, Zhanhao Zhao, Yipeng Sun, Anqun Pan, Xiaoyong Du 0001 |
Proc. VLDB Endow. | 8 |
| 2025 | A Generic Specification Framework for Weakly Consistent Replicated Data TypesabstractBurckhardt et al. proposed a formal specification framework for eventually consistent replicated data types, denoted$(vis, ar)$, based on the notions of visibility and arbitration relations. However, being specific to eventually consistent systems, this framework has two limitations. First, it does not cover non-convergent consistency models since arbitration$ar$is a total order over events. Second, it does not cover the consistency models in which each event is required to be aware of the return values of some events that are visible to it when justifying its return value. These limitations make the$(vis, ar)$framework not generic enough to specify and reason about important weak consistency models such as Causal Memory and PRAM. In this article, we extend this framework to a more generic one called$(vis, ar, V)$for weakly consistent replicated data types. To specify non-convergent consistency models as well, we relax the arbitration relation$ar$to be a partial order. To overcome the second limitation, we allow to specify for each event$e$, a subset$V(e)$of its visible set whose return values cannot be ignored when justifying the return value of$e$. To make it practically feasible, we provide candidates for the visibility and arbitration relations and the$V$function. By combining candidates for these three components, we are able to specify not only existing consistency models but also new ones that are reasonable and promising for practical usefulness. We then show how to specify consistency models in our framework, and provide three case studies. Hengfeng Wei, Yu Huang 0002, Yuxing Chen 0003, Anqun Pan |
IEEE Trans. Parallel Distributed Syst. | 5 |
| 2024 | Lion: Minimizing Distributed Transactions Through Adaptive Replica ProvisionabstractDistributed transaction processing often involves multiple rounds of cross-node communications, and therefore, tends to be slow. To improve performance, existing approaches convert distributed transactions into single-node transactions by either migrating co-accessed partitions onto the same nodes or establishing a super node housing replicas of the entire database. However, migration-based methods might cause transactions to be blocked due to waiting for data migration, while the super node can become a bottleneck. In this paper, we present Lion, a novel transaction processing protocol that utilizes partition-based replication to reduce the occurrence of distributed transactions. Inspired by the fact that modern distributed databases horizontally partition data, with each partition having multiple replicas, Lion aims to assign a node with one replica from each partition involved in a given transaction's read or write operations. To ensure such a node is available, we propose an adaptive replica provision mechanism, enhanced with an LSTM-based workload prediction algorithm, to determine the appropriate node for locating replicas of co-accessed partitions. The adaptation of replica placement is conducted preemptively and asynchronously, thereby minimizing its impact on performance. By employing this adaptive replica placement strategy, we ensure that the majority of transactions can be efficiently processed on a single node without additional overhead. Only a small fraction of transactions will need to be treated as regular distributed transactions when such a node is unavailable. Consequently, Lion effectively minimizes distributed transactions, while avoiding any disruption caused by data migration or the creation of a super node. We conduct extensive experiments to compare Lion against various transaction processing protocols. The results show that Lion achieves up to 2.7x higher throughput and 76.4% better scalability against these state-of-the-art approaches. Qiushi Zheng, Zhanhao Zhao, Wei Lu 0015, Chang Yao 0001, Yuxing Chen 0003, Anqun Pan, Xiaoyong Du 0001 |
ICDE | 6 |
| 2024 | IndeXY: A Framework for Constructing Indexes Larger than MemoryabstractIndexes in a database system can consume a large amount of memory. When they grow too large to be entirely held in the memory, selected portions of the indexes have to be unloaded to the secondary storage. There are a number of challenges in the design of an extensible index spanning memory and disk. First, the designs of in-memory portion and on-disk portion of the index must be decoupled so that the best choice for each device can be independently made. Second, selective unloading of in-memory portion to the disk must be carefully designed to maximize chance of memory access and to produce the most disk-friendly I/O access. Third, the strategy for index reloading from the disk and retaining in the memory must be optimized for the highest memory efficiency. In this paper, we proposed a memory-disk-spanning index design, named IndeXY, to effectively address the challenges. IndeXY distinguishes itself by being a framework that allows separate adoption of an in-memory index design and an on-disk data organization and access scheme that are deemed most efficient to its workloads. Instead of being just another one-size-fit-all index across memory and disk, the framework provides well-designed mechanisms and policies to integrate a selected in-memory index (Index X) and an on-disk index (Index Y) into one extensible index (IndeXY). We have implemented IndeXY with alternative in-memory indexes (ART tree or B+ tree) and alternative disk indexes (LSM tree or B+ tree). As an anecdotal example, experiments show that integrating the ART tree and an LSM tree in the framework can lead to a throughput improvement by as high as an 8.6X on a TPC-C workload over LeanStore that uses B+-tree indexes in the memory and disk, and can improve performance for almost all YCSB workloads. Chen Zhong 0002, Yuxing Chen 0003, Xingsheng Zhao, Kuang He, Anqun Pan, Song Jiang 0001 |
ICDE | 6 |
| 2024 | TDSQL: Tencent Distributed Database SystemabstractDistributed databases have become indispensable in contemporary computing and data processing, owing to their pivotal role in ensuring high availability and scalability. They effectively cater to the requirements of data management and high-concurrency access. However, developing a distributed database system that is well-suited for diverse application scenarios, particularly for large-scale applications, presents several challenges. These challenges include ensuring data consistency and achieving high levels of performance. This paper presents TDSQL, a distributed database system that prioritizes core design principles of distributed systems, including high availability, strong consistency, and scalability. In particular, TDSQL has achieved high performance through over a decade of practical experience and optimization in various modules, such as the kernel, synchronous replication, and transaction processing, in large-scale application scenarios. By conducting the TPC-C benchmark test, TDSQL demonstrated outstanding performance, achieving a throughput of 814 million tpmC across 1650 database nodes, with a jitter rate of less than 0.2%. This jitter rate is an order of magnitude lower than the standard required, showcasing the system's stability and reliability. During the 8-hour TPC-C standard stress test, TDSQL successfully completed over 860 billion transactions and processed 40 trillion order details, with zero forced rollbacks and zero data inconsistency. Yuxing Chen 0003, Anqun Pan, Hailin Lei, Anda Ye, Wei Lu 0015, Yunpeng Chai, Feng Zhang 0007, Xiaoyong Du 0001 |
Proc. VLDB Endow. | 2 |
| 2024 | FluidKV: Seamlessly Bridging the Gap between Indexing Performance and Memory-Footprint on Ultra-Fast StorageabstractOur extensive experiments reveal that existing key-value stores (KVSs) achieve high performance at the expense of a huge memory footprint that is often impractical or unacceptable. Even with the emerging ultra-fast byte-addressable persistent memory (PM), KVSs fall far short of delivering the high performance promised by PM's superior I/O bandwidth. To find the root causes and bridge the huge performance/memory-footprint gap, we revisit the architectural features of two representative indexing mechanisms (single-stage and multi-stage) and propose a three-stage KVS called FluidKV. FluidKV effectively consolidates these indexes by fast and seamlessly running incoming key-value request stream from the write-concurrent frontend stage to the memory-efficient backend stage across an intermediate stage. FluidKV also designs important enabling techniques, such as thread-exclusive logging, PM-friendly KV-block structures, and dual-grained indexes, to fully utilize both parallel-processing and high-bandwidth capabilities of ultra-fast storage hardware while reducing the overhead. We implemented a FluidKV prototype and evaluated it under a variety of workloads. The results show that FluidKV outperforms the state-of-the-art PM-aware KVSs, including ListDB and FlatStore with different indexes, by up to 9× and 3.9× in write and read throughput respectively, while cutting up to 90% of the DRAM footprint. Ziyi Lu, Qiang Cao 0001, Hong Jiang 0001, Yuxing Chen 0003, Jie Yao 0001, Anqun Pan |
Proc. VLDB Endow. | 6 |
| 2024 | Compressed Data Direct Computing for DatabasesabstractDirectly performing operations on compressed data has been proven to be a big success facing Big Data problems in modern data management systems. These systems have demonstrated significant compression benefits and performance improvement for data analytics applications. However, current systems only focus on data queries, while a complete Big Data system must support both data query and data manipulation. To solve this problem, we develop CompressDB, which is a new storage engine that can support data processing for databases without decompression. CompressDB has the following advantages. First, CompressDB utilizes context-free grammar to compress data, and supports both data query and data manipulation. Second, for adaptability, we integrate CompressDB to file systems so that a wide range of databases can directly use CompressDB without any change. Third, we enable operation pushdown to storage so that we can perform data query and manipulation in storage systems without bringing large data to memory for high efficiency. We validate the efficacy of CompressDB supporting various kinds of database systems, including SQLite, MySQL, LevelDB, MongoDB, ClickHouse, and Neo4j. We evaluate our method using seven real-world datasets with various lengths, structures, and content in both single node and cluster environments. Experiments show that CompressDB achieves 40% throughput improvement and 44% latency reduction, along with 1.75 compression ratio on average. Weitao Wan, Feng Zhang 0007, Chenyang Zhang 0005, Mingde Zhang, Jidong Zhai, Yunpeng Chai, Huanchen Zhang, Wei Lu 0015, Yuxing Chen 0003, Haixiang Li, Anqun Pan, Xiaoyong Du 0001 |
IEEE Trans. Knowl. Data Eng. | 11 |
| 2024 | Data-Aware Adaptive Compression for Stream ProcessingabstractStream processing has been in widespread use, and one of the most common application scenarios is SQL query on streams. By 2021, the global deployment of IoT endpoints reached 12.3 billion, indicating a surge in data generation. However, the escalating demands for high throughput and low latency in stream processing systems have posed significant challenges due to the increasing data volume and evolving user requirements. We present a compression-based stream processing engine, called CompressStreamDB, which enables adaptive fine-grained stream processing directly on compressed streams, to significantly enhance the performance of existing stream processing solutions. CompressStreamDB utilizes nine diverse compression methods tailored for different stream data types and integrates a cost model to automatically select the most efficient compression schemes. CompressStreamDB provides high throughput with low latency in stream SQL processing by identifying and eliminating redundant data among streams. Our evaluation demonstrates that CompressStreamDB improves average performance by 3.84× and reduces average delay by 68.0% compared to the state-of-the-art stream processing solution for uncompressed streams, along with 68.7% space savings. Besides, our edge trials show an average throughput/price ratio of 9.95× and a throughput/power ratio of 7.32× compared to the cloud design. Yu Zhang 0027, Feng Zhang 0007, Hourun Li, Shuhao Zhang 0001, Xiaoguang Guo, Yuxing Chen 0003, Anqun Pan, Xiaoyong Du 0001 |
IEEE Trans. Knowl. Data Eng. | 7 |
| 2024 | Explorations and Exploitation for Parity-based RAIDs with Ultra-fast SSDsabstractFollowing a conventional design principle that pays more fast-CPU-cycles for fewer slow-I/Os, popular software storage architecture Linux Multiple-Disk (MD) for parity-based RAID (e.g., RAID5 and RAID6) assigns one or more centralized worker threads to efficiently process all user requests based on multi-stage asynchronous control and global data structures, successfully exploiting characteristics of slow devices, e.g., Hard Disk Drives (HDDs). However, we observe that, with high-performance NVMe-based Solid State Drives (SSDs), even the recently added multi-worker processing mode in MD achieves only limited performance gain because of the severe lock contentions under intensive write workloads. In this paper, we propose a novel stripe-threaded RAID architecture, StRAID, assigning a dedicated worker thread for each stripe-write (one-for-one model) to sufficiently exploit high parallelism inherent among RAID stripes, multi-core processors, and SSDs. For the notoriously performance-punishing partial-stripe writes that induce extra read and write I/Os, StRAID presents a two-stage stripe write mechanism and a two-dimensional multi-log SSD buffer. All writes first are opportunistically batched in memory, and then are written into the primary RAID for aggregated full-stripe writes or conditionally redirected to the buffer for partial-stripe writes. These buffered data are strategically reclaimed to the primary RAID. We evaluate a StRAID prototype with a variety of benchmarks and real-world traces. StRAID is demonstrated to outperform MD by up to 5.8 times in write throughput. Shucheng Wang, Qiang Cao 0001, Hong Jiang 0001, Ziyi Lu, Jie Yao 0001, Yuxing Chen 0003, Anqun Pan |
ACM Trans. Storage | 7 |
| 2024 | G-Learned Index: Enabling Efficient Learned Index on GPUabstractAI and GPU technologies have been widely applied to solve big data problems. The total data volume worldwide reaches 200 zettabytes in 2022. How to efficiently index the required content among massive data becomes serious. Recently, a promising learned index has been proposed to address this challenge: It has extremely high efficiency while retaining marginal space overhead. However, we notice that previous learned indexes have mainly focused on CPU architecture, while ignoring the advantages of GPU. Because traditional indexes like B-Tree, LSM, and bitmap have greatly benefited from GPU acceleration, a combination of a learned index and GPU has great potentials to reach tremendous speedups. In this paper, we propose a GPU-based learned index, called G-Learned Index, to significantly improve the performance of learned index structures. The primary challenges in developing G-Learned Index lie in the use of thousands of GPU cores including minimization of synchronization and branch divergence, data structure design for parallel operations, and usage of memory bandwidth including limited memory transactions and multi-memory hierarchy. To overcome these challenges, a series of novel technologies are developed, including efficient thread organization, succinct data structures, and heterogeneous memory hierarchy utilization. Compared to the state-of-the-art learned index, the proposed G-Learned Index achieves an average of 174× speedup (and 107× of its parallel version). Meanwhile, we attain 2× less query time over the state-of-the-art GPU B-Tree. Our further exploration of range queries shows that G-Learned Index is 17× faster than CPU multi-dimensional learned index. We have made G-Learned Index available athttps://anonymous.4open.science/r/G-Learned-Index-8D89. Jiesong Liu, Feng Zhang 0007, Lv Lu, Chang Qi, Xiaoguang Guo, Dong Deng 0001, Guoliang Li 0001, Huanchen Zhang, Jidong Zhai, Hechen Zhang, Yuxing Chen 0003, Anqun Pan, Xiaoyong Du 0001 |
IEEE Trans. Parallel Distributed Syst. | 12 |
| 2024 | RCBench: an RDMA-enabled transaction framework for analyzing concurrency control algorithms
Hongyao Zhao, Wei Lu 0015, Wanqing Yang, Jiajia Zhong, Meihui Zhang 0001, Haixiang Li, Xiaoyong Du 0001, Anqun Pan |
VLDB J. | 10 |
| 2023 | Context-Aware Semantic Type Identification for Relational Attributes
Yuhe Guo, Wei Lu 0015, Haixiang Li, Meihui Zhang 0001, Hui Li 0046, Anqun Pan, Xiaoyong Du 0001 |
J. Comput. Sci. Technol. | 7 |
| 2023 | SALI: A Scalable Adaptive Learned Index Framework based on Probability ModelsabstractThe growth in data storage capacity and the increasing demands for high performance have created several challenges for concurrent indexing structures. One promising solution is the learned index, which uses a learning-based approach to fit the distribution of stored data and predictively locate target keys, significantly improving lookup performance. Despite their advantages, prevailing learned indexes exhibit constraints and encounter issues of scalability on multi-core data storage. This paper introduces SALI, the Scalable Adaptive Learned Index framework, which incorporates two strategies aimed at achieving high scalability, improving efficiency, and enhancing the robustness of the learned index. Firstly, a set of node-evolving strategies is defined to enable the learned index to adapt to various workload skews and enhance its concurrency performance in such scenarios. Secondly, a lightweight strategy is proposed to maintain statistical information within the learned index, with the goal of further improving the scalability of the index. Furthermore, to validate their effectiveness, SALI applied the two strategies mentioned above to the learned index structure that utilizes fine-grained write locks, known as LIPP. The experimental results have demonstrated that SALI significantly enhances the insertion throughput with 64 threads by an average of 2.04x compared to the second-best learned index. Furthermore, SALI accomplishes a lookup throughput similar to that of LIPP+. Jiake Ge, Huanchen Zhang, Boyu Shi, Yuanhui Luo, Yunda Guo, Yunpeng Chai, Yuxing Chen 0003, Anqun Pan |
Proc. ACM Manag. Data | 8 |
| 2023 | Homomorphic Compression: Making Text Processing on Compression UnlimitedabstractLossless data compression is an effective way to handle the huge transmission and storage overhead of massive text data. Its utility is even more significant today when data volumes are skyrocketing. The concept of operating on compressed data infuses new blood into efficient text management by enabling mainly access-oriented text processing tasks to be done directly on compressed data without decompression. Facing limitations of the existing compressed text processing schemes such as limited types of operations supported, low efficiency, and high space occupation, we address these problems by proposing a homomorphic compression theory. It enables the generalization and characterization of algorithms with compression processing capabilities. On this basis, we develop HOCO, an efficient text data management engine that supports a variety of processing tasks on compressed text. We select three representative compression schemes and implement them combined with homomorphism in HOCO. HOCO supports the extension of homomorphic compression schemes through a modular and object-oriented design and has convenient interfaces for text processing tasks. We evaluate HOCO on six real-world datasets. The three schemes implemented in HOCO show trade-offs in terms of compression ratio, supported operation types, and efficiency. Experiments also show that HOCO can achieve higher throughput in random access and modification operations (averagely 9.18× than the state-of-the-art) and lower latency in text analytic tasks (averagely 7.16× than processing on uncompressed text) without compromising compression efficacy. Jiawei Guan, Feng Zhang 0007, Siqi Ma 0001, Kuangyu Chen, Yihua Hu 0003, Yuxing Chen 0003, Anqun Pan, Xiaoyong Du 0001 |
Proc. ACM Manag. Data | 7 |
| 2023 | Efficient and Accurate SimRank-based Similarity Joins: Experiments, Analysis, and ImprovementabstractSimRank-based similarity joins, which mainly include threshold-based and top- k similarity joins, are important types of all-pair SimRank queries. Although a line of related algorithms have been proposed recently, they still fall short of providing approximation guarantee and suffer from scalability issues on medium and large graphs. Meanwhile, we also lack an extensive analysis of existing techniques in terms of accuracy and efficiency. Motivated by these challenges, we first conduct detailed analysis of state-of-the-art algorithms and provide additional theoretical results. Second, to address the limitations of existing techniques, we propose simple yet effective algorithm frameworks for both queries to theoretically guarantee the approximation bound, and present a more efficient all-pair algorithm inspired by randomized local push of Personalized PageRank. Next, we analyze the algorithmic complexity of threshold-based and top- k similarity joins by leveraging a reasonable assumption of SimRank distribution. Through extensive experiments, we find that our proposed methods far exceed existing ones with respect to query efficiency, approximation guarantee and practical accuracy, while our theoretical analysis nicely matches the empirical study. Yu Liu 0070, Yuetian Sun, Lei Zou 0001, Yuxing Chen 0003, Anqun Pan |
Proc. VLDB Endow. | 7 |
| 2023 | Efficient Black-box Checking of Snapshot Isolation in DatabasesabstractSnapshot isolation (SI) is a prevalent weak isolation level that avoids the performance penalty imposed by serializability and simultaneously prevents various undesired data anomalies. Nevertheless, SI anomalies have recently been found in production cloud databases that claim to provide the SI guarantee. Given the complex and often unavailable internals of such databases, a black-box SI checker is highly desirable. In this paper we present PolySI, a black-box checker that efficiently checks SI and provides understandable counterexamples upon detecting violations. PolySI builds on a characterization of SI using generalized polygraphs (GPs), for which we establish its soundness and completeness. PolySI employs an SMT solver and also accelerates SMT solving by utilizing a compact constraint encoding of GPs and domain-specific optimizations for pruning constraints. As our extensive assessment demonstrates, PolySI successfully reproduces all of 2477 known SI anomalies, detects novel SI violations in three production cloud databases, identifies their causes, outperforms the state-of-the-art black-box checkers under a wide range of workloads, and can scale up to large workloads. Kaile Huang, Si Liu 0003, Zhenge Chen, Hengfeng Wei, David A. Basin, Haixiang Li, Anqun Pan |
Proc. VLDB Endow. | 7 |
| 2023 | T-SQL: A Lightweight Implementation to Enable Built-in Temporal Support in MVCC-Based RDBMSsabstractThe adoption of temporal expressions into SQL:2011 has continuously driven the extensions of temporal support in relational database systems (a.b.a. RDBMSs). In this paper, we present T-SQL, a lightweight yet efficient built-in temporal implementation in RDBMSs. T-SQL completely relies on multi-version concurrency control (MVCC) which is widely adopted in RDMBSs to manage temporal data. For temporal data, current records are maintained in legacy databases, and historical records, i.e., previoius versions of current records (if any), which used to be periodically reclaimed are separately maintained in KV stores. To enable temporal query processing under SQL:2011, we extend the query engine in legacy RDBMSs to support query processing over either historical records or current records or both. Further, regarding temporal data are ever-increasing, we propose various optimizations to reduce the storage overhead of KV stores while keeping efficient query performance. We elaborate on a publicly available implementation, on how to integrate T-SQL into both centralized and distributed RDBMSs. We conduct extensive experiments on both YCSB and TPC-series benchmarks by comparing T-SQL with other temporal database systems. The results show that T-SQL is both lightweight and efficient. Zhanhao Zhao, Wei Lu 0015, Hongyao Zhao, Zongyan He, Haixiang Li, Anqun Pan, Xiaoyong Du 0001 |
IEEE Trans. Knowl. Data Eng. | 6 |
| 2023 | Efficiently Supporting Multi-Level Serializability in Decentralized Database SystemsabstractIn decentralized database systems, it is reported that serializability could still produce unexpected transaction orderings, leading to the stale read anomaly. To eliminate this anomaly, strict serializability imposes an additional ordering constraint, called the real-time order, which is required to be preserved among serializable transactions. Yet, preserving the real-time order in strict serializability often causes the performance to drop significantly. Because a weaker data consistency often yields better performance, in this paper, we model serializability from different consistency perspectives to properly leverage the performance and consistency. To do this, we first define a group of orderings, based on which we formulate multi-level serializability by preserving a certain set of ordering constraints among transactions. We then propose a bidirectional timestamp adjustment algorithm (abbreviated as BDTA) to support multi-level serializability with various optimizations. Our special design makes ordering constraints among transactions be preserved simply by adjusting timestamp intervals. Finally, we conduct extensive experiments to show the necessity of introducing multi-level serializability and confirm that BDTA achieves up to 1.19 × better performance than the state-of-the-art concurrency control algorithms. Zhanhao Zhao, Hongyao Zhao, Qiyu Zhuang, Wei Lu 0015, Haixiang Li, Meihui Zhang 0001, Anqun Pan, Xiaoyong Du 0001 |
IEEE Trans. Knowl. Data Eng. | 7 |
| 2022 | Checking Causal Consistency of MongoDB
Hongrong Ouyang, Heng-Feng Wei, Haixiang Li, Anqun Pan, Yu Huang 0002 |
J. Comput. Sci. Technol. | 4 |
| 2021 | Byz-GentleRain: An Efficient Byzantine-Tolerant Causal Consistency Protocol
Kaile Huang, Hengfeng Wei, Yu Huang 0002, Haixiang Li, Anqun Pan |
SSS | 5 |
| 2019 | A Lightweight and Efficient Temporal Database Management System in TDSQLabstractDriven by the recent adoption of temporal expressions into SQL:2011, extensions of temporal support in conventional database management systems (a.b.a. DBMSs) have re-emerged as a research hotspot. In this paper, we present a lightweight yet efficient built-in temporal implementation in Tencent's distributed database management system, namely TDSQL. The novelty of TDSQL's temporal implementation includes: (1) a new temporal data model with the extension of SQL:2011, (2) a built-in temporal implementation with various optimizations, which are also applicable to other DBMSs, and (3) a low-storage-consumption in which only data changes are maintained. For the repeatability purpose, we elaborate the integration of our proposed techniques into MySQL. We conduct extensive experiments on both real-life dataset and synthetic TPC benchmarks by comparing TD-SQL with other temporal databases. The results show that TDSQL is lightweight and efficient. Wei Lu 0015, Zhanhao Zhao, Haixiang Li, Zhenmiao Zhang, Zhiyu Shui, Anqun Pan, Xiaoyong Du 0001 |
Proc. VLDB Endow. | 8 |
| 2019 | Efficient time-interval data extraction in MVCC-based RDBMS
Haixiang Li, Zhanhao Zhao, Yijian Cheng, Wei Lu 0015, Xiaoyong Du 0001, Anqun Pan |
World Wide Web | 6 |
| 2018 | Conceptual Modeling on Tencent's Distributed Database Systems
Anqun Pan, Haixiang Li |
ER | 1 |
| 2018 | MSQL+: a Plugin Toolkit for Similarity Search under Metric Spaces in Distributed Relational Database SystemsabstractSimilarity search is a primitive operation in various database applications. Thus far, a large number of access methods have been proposed to accelerate the similarity query processing. Nonetheless, these methods mostly focus on developing standalone systems by proposing new indices. Given the fact that existing RDBMS merely support traditional indices, it is of great necessity and practical importance to develop a standard RDBMS built-in index based approach to speeding up the query processing. In this demonstration, we introduce MSQL+, a plugin toolkit that enable users to answer similarity queries in metric spaces simply using standard SQL statements. This toolkit can help existing RDBMS to effectively and efficiently handle with big data due to the following three advantages. First, MSQL+ enables users to find similar objects by submitting SELECT-FROM-WHERE statements so that it can be easily integrated into existing RDBMS. Second, MSQL+ works in a more general data space. Objects of any type can be indexed by B + -trees and the query processing can be boosted by using index seeks, as long as the similarity function is metric. Third, MSQL+ supports the parallelization of both pre-processing and query processing in distributed RDBMS. Wei Lu 0015, Xinyi Zhang 0002, Zhiyu Shui, Zhe Peng, Xiao Zhang 0001, Xiaoyong Du 0001, Hao Huang 0001, Anqun Pan, Haixiang Li |
Proc. VLDB Endow. | 9 |