Over the last decade Kafka has become a key component of Big Data processing pipelines. Kafka is increasingly used as a store of data, not just as a means of transferring it from one location to another. For cloud providers and enterprises this means that Kafka must conform to the same security and compliance requirements as conventional data storage systems such as relational databases. One important requirement is encryption-at-rest which is not currently supported by Kafka. We present an analysis of different ways to implement encryption in Kafka and then describe the first c omplete s ystem f or implementing encryption-at-rest at the granularity of a Kafka topic at scale. We demonstrate the challenges in implementing encryption policy, key distribution, key rotation and data re-encryption in Kafka, using our working implementation for illustration.

Pathfinder is a novel way of discovering, curating and disseminating heterogeneous metadata from multiple distributed systems at the scale of an entire enterprise. The management of metadata at scale has aspects that are similar to big data processing systems in regard to the need for handling high volumes of fast changing data derived from very varied sources. Pathfinder uses well known techniques from the data processing world such as schema-on-read, immutable logs, eventing, loose coupling etc. that are currently not widely used in metadata systems. At the core of Pathfinder is a Write-Ahead Log (WAL) for the entire set of metadata changes in an enterprise. This log contains a graph of an enterprise's data ecosystem that we call the Enterprise Data Map (EDM). The EDM is used by multiple independent processing systems that extract actionable information from the gathered metadata.

The Kafka streaming platform has at its heart a distributed commit log. This resembles the change log that exists in every relational database system. It has been suggested that Kafka be viewed not just as a messaging system, but as the core of a database. The database is in effect “turned inside out” as the normally hidden change log becomes the first class entity of the system, while what is normally considered primary, i.e. the table, view, indexes etc. are just derived from this log. This is appealing as a vision, but raises challenges when applied within an actual enterprise system. The challenges arise from the conflicting interests and requirements of analytics and transactional systems. Running everything on a single system leads to tradeoffs; our intentions here is to identify some of the practical problems with using Kafka as a single data store within an enterprise and to describe our initial approach to resolving them. In particular we present preliminary approaches to ensure consistency and coherence of data from multiple database tables when distributed over Kafka and how to address compliance by encrypting/decrypting data at the Kafka producers and consumers.

Moving data from an on-premise system to the cloud is often seen as a one-off exercise performed as part of a single migration. In a hybrid cloud system it is more typically a continual process in which data is refreshed from systems of record onto the public cloud at some interval. Of particular interest to industry is the data held within commercial relational databases. Data obtained from one or more relational databases may be combined within a system of insight in order to derive useful actionable information. We outline the issues involved in frequently moving such data and describe a system which performs these tasks.