Bringing a new AI system into a production environment involves multiple different stakeholders such as business owners, risk officer, ethics officers approving the AI System for a specific usage. Governance frameworks typically include multiple manual steps, including curating information needed to assess risks and reviewing outcomes to identify appropriate actions and governance strategies. We demo a human-in-the-loop automation system that takes a natural language description of an intended use case for an AI system in order to create semi-structured governance information, recommend the most appropriate model for that use case, prioritise risks to be evaluated, automatically running those evaluations and finally storing these results for auditing, reporting and future recommendations. As a result we increase transparency to stakeholders and provide valuable information to aid in decision making when assessing risks associated with an AI solution.

Pathfinder is a novel way of discovering, curating and disseminating heterogeneous metadata from multiple distributed systems at the scale of an entire enterprise. The management of metadata at scale has aspects that are similar to big data processing systems in regard to the need for handling high volumes of fast changing data derived from very varied sources. Pathfinder uses well known techniques from the data processing world such as schema-on-read, immutable logs, eventing, loose coupling etc. that are currently not widely used in metadata systems. At the core of Pathfinder is a Write-Ahead Log (WAL) for the entire set of metadata changes in an enterprise. This log contains a graph of an enterprise's data ecosystem that we call the Enterprise Data Map (EDM). The EDM is used by multiple independent processing systems that extract actionable information from the gathered metadata.

Over the last three years we have been running a large-scale data processing platform for applying analytics to corporate data on a private cloud instance. We control every level in the stack from the processing engines down to the hardware. One very common pattern of usage is for data scientists to use SQL/Hadoop to explore and analysis data sets. Data scientists are free to run whatever queries they want on this shared environment. Here we report on the patterns of usage of data scientists and the measured performance of the queries they create. We motivate why it is difficult to estimate the resource usage of a SQL query on such a system ahead of time and explain the consequences for the design of enterprise datalakes.

The Kafka streaming platform has at its heart a distributed commit log. This resembles the change log that exists in every relational database system. It has been suggested that Kafka be viewed not just as a messaging system, but as the core of a database. The database is in effect “turned inside out” as the normally hidden change log becomes the first class entity of the system, while what is normally considered primary, i.e. the table, view, indexes etc. are just derived from this log. This is appealing as a vision, but raises challenges when applied within an actual enterprise system. The challenges arise from the conflicting interests and requirements of analytics and transactional systems. Running everything on a single system leads to tradeoffs; our intentions here is to identify some of the practical problems with using Kafka as a single data store within an enterprise and to describe our initial approach to resolving them. In particular we present preliminary approaches to ensure consistency and coherence of data from multiple database tables when distributed over Kafka and how to address compliance by encrypting/decrypting data at the Kafka producers and consumers.

Moving data from an on-premise system to the cloud is often seen as a one-off exercise performed as part of a single migration. In a hybrid cloud system it is more typically a continual process in which data is refreshed from systems of record onto the public cloud at some interval. Of particular interest to industry is the data held within commercial relational databases. Data obtained from one or more relational databases may be combined within a system of insight in order to derive useful actionable information. We outline the issues involved in frequently moving such data and describe a system which performs these tasks.

We describe a distributed protocol that guarantees the consistency of multiple replicas of a given disk image represented by a large file. The image replicas are used as read/write caches for the primary disk image located at a server. The protocol allows stale caches to be incrementally invalidated minimizing the amount of data that must be copied and recopied from the server. We prove that the protocol ensures the consistency of the disk from the point of view of any client using it, and we describe an implementation within an existing streaming framework.

The service-oriented architecture (SOA) has been successfully applied in enterprise environments. Due to decentralized set-ups, mergers and acquisitions and organizational boundaries, many enterprises today operate multiple, fragmented and heterogeneous service infrastructures that are administered by different organizational units. This fragmented infrastructure causes service duplication and unnecessary redundancy. This paper proposes an approach of cross-domain service integration through an automated federation of Enterprise Service Buses (ESBs). ESBs are the mediation centers within a service domain that enable service interaction across technological boundaries by using service proxies. We present DISCE, a configuration engine prototype that enables an operator to configure service connectivity in such an environment in a declarative form, by specifying simple rules. The engine produces a configuration consisting of a set of proxies interconnecting clients and services.

A network deployment of generally available operating systems (OS) usually takes in the order of tens of minutes. This is prohibitive in an environment in which OSs must be dynamically and frequently provisioned in response to external requests. By exploiting the fact that in general only a small part of an OS image is actually required to be present to perform useful tasks, we demonstrate how an OS can perform work shortly after a deployment has begun. This requires the insertion of a streaming device between the operating system and the disk. We have implemented such a device for Windows* and Linux*. We show that such an OS streaming deployment reduces significantly (i.e., to a few seconds) the time between the start of the deployment and the moment at which the OS is available. Furthermore, we demonstrate that the performance overhead of using the OS during streaming is negligible as the penalty introduced by the streaming device is minor and the I/O performance is completely dominated by the multiple caches between the application and the disk.

Service-oriented architecture (SOA) has become the preferred methodology to build complex enterprise applications. Inside an enterprise, an enterprise service bus (ESB) has emerged as a powerful and flexible solution to have services and applications interact with each other, independently of their location and connectivity requirements. ESBs are implemented with some form of middleware, and messaging middleware has been playing a major role in integrating applications and services in the J2EE space thanks to standards such as the Java messaging service (JMS). We observe that in an enterprise the need to use real-world current information and the shift of complexity toward higher layers of the SOA-based applications (e.g., the business process layer) favor the introduction of lightweight messaging middleware, benefiting real-time applications. This paper describes an architecture for a messaging middleware that aims at fulfilling the requirements of a real-time ESB, while maintaining compatibility with the applications already deployed.We present performance results of a first prototype that show low, regular latency and high throughput. We contend that this middleware is an essential building block for real-time SOA in the enterprise.

The software in modern systems has become too complex to make accurate predictions about their performance under different configurations. Real-time or even responsiveness requirements cannot be met because it is not possible to perform admission control for new or changing tasks if we cannot tell how their execution affects the other tasks already running. Previously, we proposed a resource allocation middleware that manages the execution of tasks in a complex distributed system with real-time requirements. The middleware behavior can be modeled depending on the configuration of the tasks running, so that the performance of any given configuration can be calculated. This makes it possible to have admission control in such a system, but the model requires knowledge of run-time parameters. We propose the utilization of machine learning algorithms to obtain the model parameters, and be able to predict the system performance under any configuration, so that we can provide a full admission control mechanism for complex software systems.

We contend that event systems will become the kernel of distributed middleware for multicore systems due to their asynchronous nature. In a concurrent system, the fewer points of synchronization between independent activities, the better the achievable parallelization. As physical devices are integrated into distributed middleware the timeliness of the execution of functions becomes increasingly important. Precise guarantees require complete control of all the elements which influence the execution. However, the purpose of middleware is to abstract from the exact nature of the underlying platform. Here we outline an approach that allocates processor shares to event queues, thus controlling the rate of events handled at any given queue. We show how this in turn allows us to predict the likely through put through a queue. This prediction is achieved through model fitting and calibration.

Peer-to-peer distributed hash table (DHT) systems make it simple to discover specific data when their complete identifiers - or keys - are known in advance. In practice, however, users looking up resources stored in peer-to-peer systems often have only partial information for identifying these resources. We describe techniques for indexing data stored in peer-to-peer DHT networks, and discovering the resources that match a given user query. Our system creates multiple indexes, organized hierarchically, which permit users to locate data even using scarce information, although at the price of a higher lookup cost. The data itself is stored on only one (or few) of the nodes. Experimental evaluation demonstrates the effectiveness of our indexing techniques on a distributed peer-to-peer bibliographic database with realistic user query workloads.