The cloud-native version of SAP IQ aims to reduce the storage and compute costs by storing data directly on object stores while benefiting from the greater elasticity and scale-out properties that are offered. In the cloud-native version of SAP IQ, the buffer manager has gone through a significant re-design for two major reasons. First, RAM is more expensive on the cloud; therefore, a buffer manager that relies exclusively on RAM would not have been a viable solution. Second, on object stores, read and write operations are generally associated with higher latencies. To counteract the impact of higher latencies without utilizing more RAM, we have extended SAP IQ's buffer manager with a second layer of cache, namely, the Extended Cache Manager (ECM). The ECM relies on fast solid state drives (SSDs). In this paper, we describe our experience with the design and implementation of the ECM, in particular, the design choices we had to make to overcome the fact that, when compared to object stores, SSDs have a much more limited I/O bandwidth.

In this paper, we describe our journey of transforming SAP IQ into a relational database management system (RDBMS) that utilizes cheap, elastically scalable object stores on the cloud. SAP IQ is a three-decade old, disk-based, columnar RDBMS that is optimized for complex online analytical processing (OLAP) workloads. Traditionally, SAP IQ has been designed to operate on shared storage devices with strong consistency guarantees (e.g., high-caliber storage area network devices). Therefore, deploying SAP IQ on the cloud, as is, would have meant utilizing storage solutions such as NetApp or AWS EFS that provide a POSIX compliant file interface and strong consistency guarantees, but at a much higher monetary cost. These costs can accumulate easily to diminish the economies of scale that one would expect on the cloud, which can be undesirable. Instead, we have enhanced the design of SAP IQ to operate on cloud object stores such as AWS S3 and Azure Blob Storage. Object stores rely on a weaker consistency model, and potentially have higher latency; however, because of these design trade-offs, they are able to offer (i) better pricing, (ii) enhanced durability, (iii) improved elasticity, and (iv) higher throughput. By enhancing SAP IQ to operate under these design trade-offs, we have unlocked many of the opportunities offered by object stores. More specifically, we have extended SAP IQ's buffer manager and transaction manager, and have introduced a new caching layer that utilizes instance storage on AWS EC2. Experiments using the TPC-H benchmark demonstrate that we can gain an order of magnitude reduction in data-at rest storage costs while improving query and load performance.