Privacy regulations are multiplying at the same time the importance of data accelerates, highlighting a fundamental tension between data protection and utility. Data are often distributed and transferred across geographic and administrative domains, subjecting them to multiple jurisdictions. In this complex environment, compliance requires privacy by design and automation in enforcing policies and detecting violations. We propose an architecture implementing these principles and discuss how it supports new connected-vehicle services enabled by processing data in a GDPR-compliant way compliant.

Over the last decade Kafka has become a key component of Big Data processing pipelines. Kafka is increasingly used as a store of data, not just as a means of transferring it from one location to another. For cloud providers and enterprises this means that Kafka must conform to the same security and compliance requirements as conventional data storage systems such as relational databases. One important requirement is encryption-at-rest which is not currently supported by Kafka. We present an analysis of different ways to implement encryption in Kafka and then describe the first c omplete s ystem f or implementing encryption-at-rest at the granularity of a Kafka topic at scale. We demonstrate the challenges in implementing encryption policy, key distribution, key rotation and data re-encryption in Kafka, using our working implementation for illustration.

The Kafka streaming platform has at its heart a distributed commit log. This resembles the change log that exists in every relational database system. It has been suggested that Kafka be viewed not just as a messaging system, but as the core of a database. The database is in effect “turned inside out” as the normally hidden change log becomes the first class entity of the system, while what is normally considered primary, i.e. the table, view, indexes etc. are just derived from this log. This is appealing as a vision, but raises challenges when applied within an actual enterprise system. The challenges arise from the conflicting interests and requirements of analytics and transactional systems. Running everything on a single system leads to tradeoffs; our intentions here is to identify some of the practical problems with using Kafka as a single data store within an enterprise and to describe our initial approach to resolving them. In particular we present preliminary approaches to ensure consistency and coherence of data from multiple database tables when distributed over Kafka and how to address compliance by encrypting/decrypting data at the Kafka producers and consumers.