The Kafka streaming platform has at its heart a distributed commit log. This resembles the change log that exists in every relational database system. It has been suggested that Kafka be viewed not just as a messaging system, but as the core of a database. The database is in effect “turned inside out” as the normally hidden change log becomes the first class entity of the system, while what is normally considered primary, i.e. the table, view, indexes etc. are just derived from this log. This is appealing as a vision, but raises challenges when applied within an actual enterprise system. The challenges arise from the conflicting interests and requirements of analytics and transactional systems. Running everything on a single system leads to tradeoffs; our intentions here is to identify some of the practical problems with using Kafka as a single data store within an enterprise and to describe our initial approach to resolving them. In particular we present preliminary approaches to ensure consistency and coherence of data from multiple database tables when distributed over Kafka and how to address compliance by encrypting/decrypting data at the Kafka producers and consumers.

Over the last few years, NAND flash manufacturers have steadily increased the number of bits stored per cell to achieve significant cost reductions. However, the increased density does not come without drawbacks. All key flash performance metrics, including latency and endurance, significantly degrade as bit density increases. Particularly, sustained write throughput is the worst affected as writes are roughly one order of magnitude slower than reads and further require precursory block erases in the background. As a result, many recent flash controllers operate flash blocks both in single-bit (high endurance and performance) and in multi-bit (high density) mode. In theory, such hybrid controllers are a great way of hiding flash technology limitations. A controller can use a small percentage of the flash blocks in single-bit mode as a cache which allows orders of magnitude higher write bandwidth and endurance in environments where the access patterns of the workload are skewed and bursty. In practice, however, many devices fall short of expectations when write performance varies significantly and utilization increases. We argue that a principled approach is required to understand the design trade-offs of hybrid NAND flash controllers. To this end, we develop a modeling framework for estimating the performance and endurance of hybrid controllers. The modeling framework computes the internal data movement generated by a hybrid controller by relying on advanced analytical models that offer both accurate and fast predictions. The data flow is then translated into higher-level metrics that quantify upper bounds for the overall performance of an SSD such as write throughput, latency, and device endurance. Using our modeling framework, we compare different controller architectures, identify their strong and weak points, and show that there is room to improve the efficiency of the hybrid controllers used today.

Despite its widespread use in consumer devices and enterprise storage systems, NAND flash faces a growing number of challenges. While technology advances have helped to increase the storage density and reduce costs, they have also led to reduced endurance and larger block variations, which cannot be compensated solely by stronger ECC or read-retry schemes but have to be addressed holistically. Our goal is to enable low-cost NAND flash in enterprise storage for cost efficiency. We present novel flash-management approaches that reduce write amplification, achieve better wear leveling, and enhance endurance without sacrificing performance. We introduce block calibration, a technique to determine optimal read-threshold voltage levels that minimize error rates, and novel garbage-collection as well as data-placement schemes that alleviate the effects of block health variability and show how these techniques complement one another and thereby achieve enterprise storage requirements. By combining the proposed schemes, we improve endurance by up to 15× compared to the baseline endurance of NAND flash without using a stronger ECC scheme. The flash-management algorithms presented herein were designed and implemented in simulators, hardware test platforms, and eventually in the flash controllers of production enterprise all-flash arrays. Their effectiveness has been validated across thousands of customer deployments since 2015.

In recent years, the adoption of NAND flash in enterprise storage systems has been progressing rapidly. Todays all-flash storage arrays exhibit excellent I/O throughput, latency, storage density, and energy efficiency. However, the advancements in NAND technology are driven mostly by the consumer market, which makes NAND flash manufacturers focus primarily on reducing cost ($/GiB) and increasing the storage density by technology node scaling, by increasing the number of bits stored per cell, and by stacking cells vertically (3D-NAND). This comes at the cost of reduced endurance of the raw NAND flash, larger variations across blocks, and longer latencies, especially with extremely high error rates (due to the use of read-retry operations).

Hardware-assisted watchpoint systems enhance the execution of numerous dynamic software techniques, such as memory protection, module isolation, deterministic execution, and data race detection. In this paper, we show that previous hardware proposals may introduce significant energy overheads, and propose WatchPoint Filtering (WPF), a novel filtering mechanism that eliminates unnecessary watchpoint checks. We evaluate WPF on two state-of-the-art proposals for hardware-assisted watchpoints using two common memory checkers. WPF eliminates 83% of the watchpoint checks (up to 99.7%) and reduces 57% of the dynamic energy overhead (up to 78%) on average, without introducing additional performance execution overhead.

In this paper we introduce TagTM, a Software Transactional Memory (STM) system augmented with a new hardware mechanism that we call GTags. GTags are new hardware cache coherent tags that are used for fast meta-data access. TagTM uses GTags to reduce the cost associated with accesses to the transactional data and corresponding metadata. For the evaluation of TagTM, we use the STAMP TM benchmark suite. In the average case TagTM provides a speedup of 7–15% (across all STAMP applications), and in the best case shows up to 52% speedup of committed transaction execution time (for SSCA2 application).

Architectural simulator platforms are particularly complex and error-prone programs that aim to simulate all hardware details of a given target architecture. The development of a stable cycle-accurate architectural simulator can easily take several man-years. Discovering and fixing all visible errors in the simulator often requires significant effort, much higher than for writing the simulator in the first place. In addition, there are no guarantees that all programming errors will be eliminated, no matter how much effort is put into it. This paper presents dynamic runtime testing, a methodology for rapid development and accurate error detection in architectural cycle-accurate simulators. In dynamic runtime testing, the simulator execution is dynamically compared with a simple and functionally equivalent emulator. A possible error is detected if any instruction produces different results in the simulator and the emulator. Dynamic testing can help the developers of architectural simulators to get a reliable and accurate verification of functional correctness. Based on our experience, dynamic testing reduced the simulator modification time from 12-18 person-months to 3-4 person-months, and it only modestly reduced the simulator performance (in our case under 20%).

Transactional Memory aims to provide a programming model that makes parallel programming easier. Hardware implementations of transactional memory (HTM) suffer from fewer overheads than implementations in software, and refinements in conflict management strategies for HTM allow for even larger improvements. In particular, lazy conflict management has been shown to deliver better performance, but it has hitherto required complex protocols and implementations.