Bringing a new AI system into a production environment involves multiple different stakeholders such as business owners, risk officer, ethics officers approving the AI System for a specific usage. Governance frameworks typically include multiple manual steps, including curating information needed to assess risks and reviewing outcomes to identify appropriate actions and governance strategies. We demo a human-in-the-loop automation system that takes a natural language description of an intended use case for an AI system in order to create semi-structured governance information, recommend the most appropriate model for that use case, prioritise risks to be evaluated, automatically running those evaluations and finally storing these results for auditing, reporting and future recommendations. As a result we increase transparency to stakeholders and provide valuable information to aid in decision making when assessing risks associated with an AI solution.

Over the last decade Kafka has become a key component of Big Data processing pipelines. Kafka is increasingly used as a store of data, not just as a means of transferring it from one location to another. For cloud providers and enterprises this means that Kafka must conform to the same security and compliance requirements as conventional data storage systems such as relational databases. One important requirement is encryption-at-rest which is not currently supported by Kafka. We present an analysis of different ways to implement encryption in Kafka and then describe the first c omplete s ystem f or implementing encryption-at-rest at the granularity of a Kafka topic at scale. We demonstrate the challenges in implementing encryption policy, key distribution, key rotation and data re-encryption in Kafka, using our working implementation for illustration.

Pathfinder is a novel way of discovering, curating and disseminating heterogeneous metadata from multiple distributed systems at the scale of an entire enterprise. The management of metadata at scale has aspects that are similar to big data processing systems in regard to the need for handling high volumes of fast changing data derived from very varied sources. Pathfinder uses well known techniques from the data processing world such as schema-on-read, immutable logs, eventing, loose coupling etc. that are currently not widely used in metadata systems. At the core of Pathfinder is a Write-Ahead Log (WAL) for the entire set of metadata changes in an enterprise. This log contains a graph of an enterprise's data ecosystem that we call the Enterprise Data Map (EDM). The EDM is used by multiple independent processing systems that extract actionable information from the gathered metadata.

Over the last three years we have been running a large-scale data processing platform for applying analytics to corporate data on a private cloud instance. We control every level in the stack from the processing engines down to the hardware. One very common pattern of usage is for data scientists to use SQL/Hadoop to explore and analysis data sets. Data scientists are free to run whatever queries they want on this shared environment. Here we report on the patterns of usage of data scientists and the measured performance of the queries they create. We motivate why it is difficult to estimate the resource usage of a SQL query on such a system ahead of time and explain the consequences for the design of enterprise datalakes.

The Kafka streaming platform has at its heart a distributed commit log. This resembles the change log that exists in every relational database system. It has been suggested that Kafka be viewed not just as a messaging system, but as the core of a database. The database is in effect “turned inside out” as the normally hidden change log becomes the first class entity of the system, while what is normally considered primary, i.e. the table, view, indexes etc. are just derived from this log. This is appealing as a vision, but raises challenges when applied within an actual enterprise system. The challenges arise from the conflicting interests and requirements of analytics and transactional systems. Running everything on a single system leads to tradeoffs; our intentions here is to identify some of the practical problems with using Kafka as a single data store within an enterprise and to describe our initial approach to resolving them. In particular we present preliminary approaches to ensure consistency and coherence of data from multiple database tables when distributed over Kafka and how to address compliance by encrypting/decrypting data at the Kafka producers and consumers.

Moving data from an on-premise system to the cloud is often seen as a one-off exercise performed as part of a single migration. In a hybrid cloud system it is more typically a continual process in which data is refreshed from systems of record onto the public cloud at some interval. Of particular interest to industry is the data held within commercial relational databases. Data obtained from one or more relational databases may be combined within a system of insight in order to derive useful actionable information. We outline the issues involved in frequently moving such data and describe a system which performs these tasks.

We describe a distributed protocol that guarantees the consistency of multiple replicas of a given disk image represented by a large file. The image replicas are used as read/write caches for the primary disk image located at a server. The protocol allows stale caches to be incrementally invalidated minimizing the amount of data that must be copied and recopied from the server. We prove that the protocol ensures the consistency of the disk from the point of view of any client using it, and we describe an implementation within an existing streaming framework.

A network deployment of generally available operating systems (OS) usually takes in the order of tens of minutes. This is prohibitive in an environment in which OSs must be dynamically and frequently provisioned in response to external requests. By exploiting the fact that in general only a small part of an OS image is actually required to be present to perform useful tasks, we demonstrate how an OS can perform work shortly after a deployment has begun. This requires the insertion of a streaming device between the operating system and the disk. We have implemented such a device for Windows* and Linux*. We show that such an OS streaming deployment reduces significantly (i.e., to a few seconds) the time between the start of the deployment and the moment at which the OS is available. Furthermore, we demonstrate that the performance overhead of using the OS during streaming is negligible as the penalty introduced by the streaming device is minor and the I/O performance is completely dominated by the multiple caches between the application and the disk.

Real-time performance analysis of processor behaviour requires the efficient gathering of micro-architectural information from processor cores. Such information can be expected to be highly structured allowing it to be compressed, but the computational burden of conventional compression techniques exclude their use in this environment. We consider the use of new mathematical techniques that allow a signal to be compressed and recovered from a relatively small number of samples. These techniques, collectively termed compressive sampling, are asymmetric in that compression is simple, but recovery is complex. This makes them appropriate for applications in which the simplicity of the sensor can be offset against complexity at the ultimate recipient of the sensed information. We evaluate the practicality of using such techniques in the transfer of signals representing one or more micro-architectural counters from a processor core. We show that compressive sampling is usable to recover such performance signals, evaluating the trade-off between efficiency, accuracy and practicability within its various variants.

We contend that event systems will become the kernel of distributed middleware for multicore systems due to their asynchronous nature. In a concurrent system, the fewer points of synchronization between independent activities, the better the achievable parallelization. As physical devices are integrated into distributed middleware the timeliness of the execution of functions becomes increasingly important. Precise guarantees require complete control of all the elements which influence the execution. However, the purpose of middleware is to abstract from the exact nature of the underlying platform. Here we outline an approach that allocates processor shares to event queues, thus controlling the rate of events handled at any given queue. We show how this in turn allows us to predict the likely through put through a queue. This prediction is achieved through model fitting and calibration.

Cheap programmable sensor devices are becoming commercially available. They offer the possibility of transforming existing enterprise applications and enabling entirely new ones. The merging of sensor networks into the enterprise network poses some distinct problems. In particular, information from theses devices must be obtained in a way which minimizes their energy use and must be aggregated and filtered before being sent to the application server to prevent it from being overwhelmed. We describe a range of complementary techniques for integrating sensors into an enterprise network. These comprise new architectural entities within the enterprise network — edge server — new means of sharing information within the enterprise network — messaging binning — and new protocols for extracting information from the sensor network — Messo.

Edge servers are used to connect sensor networks to the enterprise network. Using asynchronous messaging for communication between components is attractive as it lessens the degree of coupling within the system, thereby increasing the system robustness. In order to be scalable, the messaging system itself must be distributed. Existing work in the literature describes how to build distributed messaging systems which scale to networks the size of the Internet using message routing. Here we explain why we find these solutions inappropriate for communication within the edge server architecture. We describe an alternative means of achieving scalable distributed messaging that we call message binning. We motivate the feasibility of our architecture through a description of its implementation and use in our edge server system.

Floating car data (FCD), i.e. traffic and in-car data collected and transmitted by moving cars, is an emerging telematics application providing a range of networked services to road users. FCD is an example of a large sensor network, whose complexity increases with the number of session participants. Our work examines how deploying application-specific functions in the network can help in scaling such systems. We have chosen to configure these functions on a dedicated device that is commonly called a middlebox. These middleboxes use programmable network processors in order to attain the required processing and forwarding speeds, while communicating with each other and a back-end server using standard middleware components. We present our experience in using such a programmable middlebox in the scaling of a large telematics service, explaining how the middlebox fits into the end-to-end application.

Research in ATM management has delegated control functions from the ATM switch into higher level open control architectures. For example, functions such as call admission control are implemented within such a control architecture rather than as an integral part of the switch. The association between the switch fabric and the switch control functions is made less strict, allowing, for example, a single entity to perform routing for a group of switches. This greater flexibility in the provision of control is accompanied by an increase in complexity resulting from the possibility of making control decisions remotely from the controlled network element. The location of the controller in relation to the fabric is not just an implementation issue; it has a large impact on the way we handle traditional operation and maintenance (OAM) functions. For example, recovering from partial failure of the network is problematic when the controlling entity is communicating with the switch it controls across that same network. Most research into such control architectures has concentrated on the advantages of increasing flexibility; little work has looked at how the traditional techniques for creating robust networks can be adapted for use in this environment. Mobile software agents can move to where their functions are carried out. When an agent has moved across the network, it executes its function there. The function will not fail due to network problems. They can also reduce the amount of management state that needs to be carried across the network for control purposes. This is important when recovering from failure since the network is stressed and minimal expectations should be placed on its behavior. We show how mobile agents can be used within an open control architecture in order to implement robust operation and maintenance functions. We explain in some detail our work in implementing one such function.