Pathfinder is a novel way of discovering, curating and disseminating heterogeneous metadata from multiple distributed systems at the scale of an entire enterprise. The management of metadata at scale has aspects that are similar to big data processing systems in regard to the need for handling high volumes of fast changing data derived from very varied sources. Pathfinder uses well known techniques from the data processing world such as schema-on-read, immutable logs, eventing, loose coupling etc. that are currently not widely used in metadata systems. At the core of Pathfinder is a Write-Ahead Log (WAL) for the entire set of metadata changes in an enterprise. This log contains a graph of an enterprise's data ecosystem that we call the Enterprise Data Map (EDM). The EDM is used by multiple independent processing systems that extract actionable information from the gathered metadata.

The Kafka streaming platform has at its heart a distributed commit log. This resembles the change log that exists in every relational database system. It has been suggested that Kafka be viewed not just as a messaging system, but as the core of a database. The database is in effect “turned inside out” as the normally hidden change log becomes the first class entity of the system, while what is normally considered primary, i.e. the table, view, indexes etc. are just derived from this log. This is appealing as a vision, but raises challenges when applied within an actual enterprise system. The challenges arise from the conflicting interests and requirements of analytics and transactional systems. Running everything on a single system leads to tradeoffs; our intentions here is to identify some of the practical problems with using Kafka as a single data store within an enterprise and to describe our initial approach to resolving them. In particular we present preliminary approaches to ensure consistency and coherence of data from multiple database tables when distributed over Kafka and how to address compliance by encrypting/decrypting data at the Kafka producers and consumers.

Moving data from an on-premise system to the cloud is often seen as a one-off exercise performed as part of a single migration. In a hybrid cloud system it is more typically a continual process in which data is refreshed from systems of record onto the public cloud at some interval. Of particular interest to industry is the data held within commercial relational databases. Data obtained from one or more relational databases may be combined within a system of insight in order to derive useful actionable information. We outline the issues involved in frequently moving such data and describe a system which performs these tasks.

Distributed infrastructures providing cloud services as well as compute- and storage clusters are notoriously difficult to administer and optimize. Management applications benefit from up-to-date data on system performance. This paper describes the Performance Monitoring and Management System (PMMS), a light-weight and versatile performance monitoring tool that collects hundreds of thousands of metrics per second and delivers this information as time-series in near real-time. At its core lies an in-memory database that scales through federation to large clusters of several hundreds of nodes. Data is collected using sensors which collect and periodically send their metric data to the PMMS collectors. Designed as a component that is embeddable into a larger system, PMMS is light-weight, with little impact on the system resources and it is easy to install and to configure.

The service-oriented architecture (SOA) has been successfully applied in enterprise environments. Due to decentralized set-ups, mergers and acquisitions and organizational boundaries, many enterprises today operate multiple, fragmented and heterogeneous service infrastructures that are administered by different organizational units. This fragmented infrastructure causes service duplication and unnecessary redundancy. This paper proposes an approach of cross-domain service integration through an automated federation of Enterprise Service Buses (ESBs). ESBs are the mediation centers within a service domain that enable service interaction across technological boundaries by using service proxies. We present DISCE, a configuration engine prototype that enables an operator to configure service connectivity in such an environment in a declarative form, by specifying simple rules. The engine produces a configuration consisting of a set of proxies interconnecting clients and services.

Much research has been done in the area of software transactional memory (STM) as a new programming paradigm to help ease the implementation of parallel applications. While most research has been invested for answering the question of how STM should be implemented, there is less work about how to use STM efficiently. This paper is focused on the challenge of how to use STM for efficient and scalable implementations of non-trivial applications. We present a fine-grained STM-based concurrent binary heap, an application of STM for a data structure that is notoriously difficult to parallelize. We describe extensions to the basic STM approach and also the benefits of our proposal. Our results show that the fine-grained STM-based binary heap provides very good scalability compared to the naive approach. Nevertheless, we reach a point where the complexity of some fine-grained techniques do not justify its use for the increase in performance that can be obtained.

Algorithms for concurrent data structures have gained attention in recent years as multi-core processors have become unbiquitous. Using the example of a concurrent priority queue, this paper investigates different synchonization methods and concurrent algorithms. It covers traditional lock-based approaches, non-blocking algorithms as well as a method based on software transactional memory. Besides discussing correctness criteria for the various approaches, we also present performance results for all algorithms as well as a method based on software transactional memory. Besides discussing correctness criteria for the various approaches, we also present performance results for all algorithms for various scenarios. Somewhat surprisingly, we find that a simple lock-based approach performs reasonable well, even though it does not scale with the number of threads. Better scalability is achieved by non-blocking approaches.

Interest in distributed storage is fueled by demand for reliability and resilience combined with decreasing hardware costs. Peer-to-peer storage networks based on distributed hash tables are attractive for their efficient use of resources and resulting performance. The placement and subsequent efficient location of replicas in such systems remain open problems, especially (1) the requirement to update replicated content, (2) working in the absence of global information, and (3) determination of the locations in a dynamic system without introducing single points of failure. We present and evaluate a novel and versatile technique, replica enumeration, which allows for controlled replication and replica access. The possibility of enumerating and addressing individual replicas allows dynamic updates as well as superior performance without burdening the network with state information, yet taking advantage of locality information when available. We simulate, analyze, and prove properties of the system, and discuss some applications.

Cheap programmable sensor devices are becoming commercially available. They offer the possibility of transforming existing enterprise applications and enabling entirely new ones. The merging of sensor networks into the enterprise network poses some distinct problems. In particular, information from theses devices must be obtained in a way which minimizes their energy use and must be aggregated and filtered before being sent to the application server to prevent it from being overwhelmed. We describe a range of complementary techniques for integrating sensors into an enterprise network. These comprise new architectural entities within the enterprise network — edge server — new means of sharing information within the enterprise network — messaging binning — and new protocols for extracting information from the sensor network — Messo.

Edge servers are used to connect sensor networks to the enterprise network. Using asynchronous messaging for communication between components is attractive as it lessens the degree of coupling within the system, thereby increasing the system robustness. In order to be scalable, the messaging system itself must be distributed. Existing work in the literature describes how to build distributed messaging systems which scale to networks the size of the Internet using message routing. Here we explain why we find these solutions inappropriate for communication within the edge server architecture. We describe an alternative means of achieving scalable distributed messaging that we call message binning. We motivate the feasibility of our architecture through a description of its implementation and use in our edge server system.

Interest in distributed storage is fueled by demand for reliability and resilience combined with ubiquitous availability. Peer-to-peer (P2P) storage networks are known for their decentralized control, self-organization, and adaptation. Advanced searching for documents and resources remains an open problem. The flooding approach favored by some P2P networks is inefficient in resource usage, but more scalable and resource-efficient solutions based on distributed hash tables (DHT) lack in query expressiveness and flexibility. In this paper, we address this issue and introduce new efficient, scalable, and completely distributed methods that strive to keep resource consumption by queries and index information as low as possible. We describe how to improve the handling of multiple subqueries combined through Boolean set operators. The need for these operators is intensified by applications to go beyond simple exact keyword matches. We discuss, optimize, and analyze appropriate extensions to support range and prefix matching in DHT.

In a hierarchical network, groups of nodes are represented by logical nodes for the purposes of simplifying routing. Each group has a set of ingress-egress nodes, and routing information is conveyed to the outside world in the form of a transition matrix that gives the cost of traversing the network between each ingress-egress node pair. In this paper, we present a minimal logical node representation that has sufficient descriptive power to enable path selection in support of connection admission control for services that have both path (restrictive) and link (additive) constraints. For example, the representation can be used to find a path that maximizes bandwidth subject to a delay constraint or minimizes delay subject to a bandwidth constraint. We present our solution in the form of a matrix whose elements are vectors, each of which specifies the efficient frontier of the solution space, and we specify an efficient procedure for constructing the efficient frontier. We present the least upper bound on the number of elements that must be present in the efficient frontier. We provide numerical examples that illustrate construction of the efficient frontier.

Distributed multimedia applications require a variety of communication services. These services and different application requirements have to be provided and supported within: (1) end-systems in an efficient and integrated manner, combining the precise specification of quality-of-service (QoS) requirements, application interfaces, multicast support, and security features and (2) the network. The Da CaPo++ system presented in this paper provides an efficient end-system middleware for multimedia applications, capable of handling various types of applications in a modular fashion. Application needs and communication demands are specified by values in terms of QoS attributes and functional properties, such as encryption requirements or multicast support. Da CaPo++ automatically configures suitable communication protocols, provides for an efficient runtime support, and offers an easy-to-use, object-oriented application programming interface. While its applicability to real-life applications was shown by prototype implementations, performance evaluations have been carried out yielding practical experiences and numerical results.

In contrast to traditional one-to-one applications, distributed multimedia applications are based on communication scenarios that involve huge but often limited groups of participants. In general, tele-teaching and virtual reality scenarios require multicasting capabilities that have to deal with reliable data, e.g., simulation data, in addition to isochronous data, e.g., audio or video, that allow for losses depending an the compression technique or color tables. In particular, reliable control messages are needed for multicast signalling purposes. Existing networks have to be enhanced by integrating reliable multicast data transmission protocols to overcome this deficiency. The presented approach provides a network independent error-control scheme for a multicast protocol, offering a high stability in the local area, and an adaptable solution for metropolitan and wide area networks.

Shared window systems allow collaboration transparent, single-user applications interacted with on multiple users' workstations, enabling the members of a cooperative ensemble to simultaneously share and revise information. The paper presents a system capable of sharing applications running under the X Window System. In contrast to previously implemented systems, the present shared window system addresses issues that are crucial for general-purpose use. The shared window system is policy-free, i.e., there are no preferred policies for handling issues such as admission and floor control. Instead, it offers a set of essential mechanisms on top of which various policies and user paradigms may be realized. Further, the system distributes the sharing functionality among all sites involved in a cooperative activity. Measurements have shown a positive impact of this on the overall performance of the system and thus justified the viability of the design decisions taken.>