A distributed system’s functionality must continuously evolve, especially when environmental context changes. Such required evolution imposes unbearable complexity on system development. An alternative is to make systems able to self-adapt by opportunistically composing at runtime to generate systems of systems (SoSs) that offer value-added functionality. The success of such an approach calls for abstracting the heterogeneity of systems and enabling the programmatic construction of SoSs with minimal developer intervention. We propose a general ontology-based approach to describe distributed systems, seeking to achieve abstraction and enable runtime reasoning between systems. We also propose an architecture for systems that utilizes such ontologies to enable systems to discover and ‘understand’ each other, and potentially compose, all at runtime. We detail features of the ontology and the architecture through three contrasting case studies: one on controlling multiple systems in smart home environment, another on the management of dynamic computing clusters, and a third on autonomic connection of rescue teams. We also quantitatively evaluate the scalability and validity of our approach through experiments and simulations. Our approach enables system developers to focus on high-level SoS composition without being constrained by deployment-specific implementation details. We demonstrate the feasibility of our approach to raise the level of abstraction of SoS construction through reasoned composition at runtime. Our architecture presents a strong foundation for further work due to its generality and extensibility.

Users of Infrastructure as a Service (IaaS) are increasingly overwhelmed with the wide range of providers and services offered by each provider. As such, many users select services based on description alone. An emerging alternative is to use a decision support system (DSS), which typically relies on gaining insights from observational data in order to assist a customer in making decisions regarding optimal deployment of cloud applications. The primary activity of such systems is the generation of a prediction model (e.g. using machine learning), which requires a significantly large amount of training data. However, considering the varying architectures of applications, cloud providers, and cloud offerings, this activity is not sustainable as it incurs additional time and cost to collect data to train the models. We overcome this through developing a Transfer Learning (TL) approach where knowledge (in the form of a prediction model and associated data set) gained from running an application on a particular IaaS is transferred in order to substantially reduce the overhead of building new models for the performance of new applications and/or cloud infrastructures. In this article, we present our approach and evaluate it through extensive experimentation involving three real world applications over two major public cloud providers, namely Amazon and Google. Our evaluation shows that our novel two-mode TL scheme increases overall efficiency with a factor of 60 percent reduction in the time and cost of generating a new prediction model. We test this under a number of cross-application and cross-cloud scenarios.

The diversity of cloud offerings motivated the proposition of cloud modelling languages (CMLs) to abstract complexities related to selection of cloud services. However, current CMLs lack the support for modelling service level objectives (SLOs) that are required for the customer applications. Consequently, we propose an application- and provider-independent SLO modelling language (SLO-ML) to enable customers to specify the required SLOs. We also sketch the architecture to realise SLO-ML.

Users of cloud computing are overwhelmed with choice, even within the services offered by one provider. As such, many users select cloud services based on description alone. In this quantitative study, we investigate the services of 2 of major IaaS providers. We use 2 representative applications to obtain longitudinal observations over 7 days of the week and over different times of the day, totalling over 14,000 executions. We give evidence of significant variations of performance offered within IaaS services, calling for data-driven brokers that are able to offer automated and adaptive decision making processes with means for incorporating expressive user constraints.

Environmental data are being generated and collected at unprecedented rates. However, the diversity in form and format of these environmental assets poses challenges for collaborative and reproducible science. Moreover, access constraints that surround environmental data lead to difficulty in use and interpretation of results. Cloud computing offers high potential to break down such barriers and engender collaboration, attribution, reuse, and reproducibility. In this article we review the design of the Environmental Virtual Observatory pilot (EVOp) that was conceived as a cloud-enabled virtual research space for different users interested in environmental science, ranging from domain specialists to the general public. We discuss the key technologies and processes used: a hybrid cloud infrastructure; standard service interfaces; a unified service delivery platform; and a test-driven development cycle. We also discuss the methodology by showcasing one of the exemplars developed in EVOp, stressing the importance of weaving stakeholder engagement from the beginning and throughout the process. We also briefly highlight some of the lessons learnt of working in an interdisciplinary team.

Distributed systems are at a watershed due to their increasing complexity. The heart of the problem is the extreme level of heterogeneity exhibited by contemporary distributed systems coupled with the need to be dynamic and responsive to change. In effect, we have moved from distributed systems to systems of systems. Following on from this, middleware is also at a watershed. The traditional view of middleware is no longer valid (i.e. as a layer of abstraction, masking the complexity of the underlying distributed system and providing a high-level programming model). In practice, middleware is often by-passed with complex systems constructed in a rather ad hoc manner as a mash-up of a variety of technologies. The end result is that middleware is no longer sure of its form or purpose and this lack of a viable approach is a huge barrier to the emergence of areas such as smart cities and emergency response systems. This paper argues that there is a need to fundamentally rethink the middleware landscape related to complex distributed systems. The core contribution of the paper is a set of fresh perspectives, which lead us in turn to novel principles and patterns for middleware and subsequently to new styles of platform. These perspectives include a move to emergent middleware, seeking flexible meta-structures for distributed systems, and a step away from generic to domain-specific technologies. A number of case studies are also presented to demonstrate what this might mean for future distributed systems.

Decision making in cloud environments is quite challenging due to the diversity in service offerings and pricing models, especially considering that the cloud market is an incredibly fast moving one. In addition, there are no hard and fast rules; each customer has a specific set of constraints (e.g. budget) and application requirements (e.g. minimum computational resources). Machine learning can help address some of the complicated decisions by carrying out customer-specific analytics to determine the most suitable instance type(s) and the most opportune time for starting or migrating instances. We employ machine learning techniques to develop an adaptive deployment policy, providing an optimal match between the customer demands and the available cloud service offerings. We provide an experimental study based on extensive set of job executions over a major public cloud infrastructure.

The world's computing infrastructure is increasingly differentiating into self-contained sub-systems (e.g. Internet of Things installations, clouds, VANETs, ...), which are post-hoc composed to generate value-added functionality (“systems of systems”). Today, however, such system-of-systems composition is typically carried out in an ad-hoc and infrastructure-dependent manner, with obvious associated disadvantages. In this paper, we propose a generalised system-of-systems-oriented programming approach that enables programmers to manage the composition of systems without a need for intimate knowledge of their internals, and also facilitates dynamic and spontaneous system composition, as systems discover each other opportunistically in their environment.

Although the principles of gossip protocols are relatively easy to grasp, their variety can make their design and evaluation highly time consuming. This problem is compounded by the lack of a unified programming framework for gossip, which means developers cannot easily reuse, compose, or adapt existing solutions to fit their needs, and have limited opportunities to share knowledge and ideas. In this paper, we consider how component frameworks, which have been widely applied to implement middleware solutions, can facilitate the development of gossip-based systems in a way that is both generic and simple. We show how such an approach can maximize code reuse, simplify the implementation of gossip protocols, and facilitate dynamic evolution and redeployment.Also known as “epidemic” protocols.

Service Discovery Protocols (SDPs) provide mechanisms that allow networked devices and applications to advertise and locate services with minimum or no human intervention. For fixed networked devices, SDPs such as SLP, Bonjour, and UPnP have been proposed. For more dynamic networked environments such as ad hoc, sensor, and wireless networks, a set of discovery technologies have been designed to operate optimally in such conditions; these include protocols like Ariadne, Allia, GSD, and UDDI. Hence, it is possible to advertise and discover services in diverse network types using one of a suite of technologies. Importantly, there are differences between individual protocols in terms of: (i) service description language, (ii) message format, (iii) directory architecture, (iv) discovery protocol behavior, (v) network communication, and (vi) nonfunctional features. These differences mean it is not possible to discover services with one protocol that is advertised by another and vice versa; we characterize this as service discovery protocol heterogeneity . In this article, we propose SeDiM, a dynamic middleware solution to allow heterogeneous discovery protocols within and across different domains to interoperate with one another. SeDiM is evaluated within a case study that demonstrates transparent interoperation of protocols including SLP and Bonjour in highly heterogeneous environments.

The CONNECT European project that started in February 2009 aims at dropping the interoperability barrier faced by todaypsilas distributed systems. It does so by adopting a revolutionary approach to the seamless networking of digital systems, that is, synthesizing on the fly the connectors via which networked systems communicate. CONNECT then investigates formal foundations for connectors together with associated automated support for learning, reasoning about and adapting the interaction behavior of networked systems.

In order to provide an increasing number of functionalities and benefit from sophisticated and application-tailored services from the network, distributed applications are led to integrate an ever-widening range of networking technologies. As these applications become more complex, this requirement for 'network heterogeneity' is becoming a crucial issue in their development. Although progress has been made in the networking community in addressing such needs through the development of network overlays, we claim in this paper that the middleware community has been slow to integrate these advances into middleware architectures, and, hence, to provide the foundational bedrock for heterogeneous distributed applications. In response, we propose our 'open overlays' framework. This framework, which is part of a wider middleware architecture, accommodates 'overlay plug-ins', allows physical nodes to support multiple overlays, supports the stacking of overlays to create composite protocols, and adopts a declarative approach to configurable deployment and dynamic reconfigurability. The framework has been in development for a number of years and supports an extensive range of overlay plug-ins including popular protocols such as Chord and Pastry. We report on our experiences with the open overlays framework, evaluate it in detail, and illustrate its application in a detailed case study of network heterogeneity.

Engineering adaptive software is an increasingly complex task. Here, we demonstrate Genie, a tool that supports the modelling, generation, and operation of highly reconfigurable, component-based systems. We showcase how Genie is used in two case-studies: i) the development and operation of an adaptive flood warning system, and ii) a service discovery application. In this context, adaptation is enabled by the Gridkit reflective middleware platform.

Abstract Flooding is a growing problem, which affects more than 10% of the U.K. population. The cost of damage caused by flooding correlates closely with the warning time given before a flood event, making flood monitoring and prediction critical to minimizing the cost of flood damage. This paper describes a wireless sensor network (WSN) for flood warning, which is capable of not only integrating with remote fixed‐network grids for computationally intensive flood modelling purposes but also performing on‐site grid computation. This functionality is supported by the reflective and component‐based GridKit middleware, which provides support for both WSN and grid application domains. Copyright © 2007 John Wiley & Sons, Ltd.

Component-based software structuring principles are now commonplace at the application level; but componentization is far less established when it comes to building low-level systems software. Although there have been pioneering efforts in applying componentization to systems-building, these efforts have tended to target specific application domains (e.g., embedded systems, operating systems, communications systems, programmable networking environments, or middleware platforms). They also tend to be targeted at specific deployment environments (e.g., standard personal computer (PC) environments, network processors, or microcontrollers). The disadvantage of this narrow targeting is that it fails to maximize the genericity and abstraction potential of the component approach. In this article, we argue for the benefits and feasibility of a generic yet tailorable approach to component-based systems-building that offers a uniform programming model that is applicable in a wide range of systems-oriented target domains and deployment environments. The component model, called OpenCom , is supported by a reflective runtime architecture that is itself built from components. After describing OpenCom and evaluating its performance and overhead characteristics, we present and evaluate two case studies of systems we have built using OpenCom technology, thus illustrating its benefits and its general applicability.

Transaction execution in mobile environments needs to be flexible, not only to support typical mobile computing characteristics, like movement, disconnections and limited resources, but also to support the variety of transactional properties that might be required by different applications. This paper presents an approach for flexible transaction processing in mobile applications, and describes an adaptive middleware for mobile transaction services (MobileTSe), that makes transaction services with different properties available on the mobile unit. We suggest a solution with transaction service discovery and control using UPnP.

Current approaches to flood monitoring (e.g. in river valleys) involve statically deploying depth and ultrasoundbased flow sensors across flood-prone areas, and feeding the collected data off-site (e.g. using GSM) to grid-based

The cost of damage caused by flooding is directly related to the warning-time given before a flood occurs. Therefore, improving the coverage, accuracy and reliability of flood prediction systems is of great importance. This paper proposes a novel Grid-based approach to supporting flood prediction through the use of embedded sensor nodes equipped with wireless networking technology. These nodes implement a light-weight Grid capable of collecting and transmitting data gathered by flood sensors for off-site analysis. Additionally, the nodes are capable of performing basic on-site analysis that is used to inform system behavior. The proposed system utilizes Lancaster's next-generation component-based Grid middleware platform 'GridKit', which offers inherent support for reconfiguration, embedded devices and heterogeneous network technologies, making it ideally suited to this application.

Abstract Significant progress has been made in the design and development of Grid middleware which, in its present form, is founded on Web services technologies. However, we argue that present‐day Grid middleware is severely limited in supporting projected next‐generation applications which will involve pervasive and heterogeneous networked infrastructures, and advanced services such as collaborative distributed visualization. In this paper we discuss a new Grid middleware framework that features (i) support for advanced network services based on the novel concept of pluggable overlay networks, (ii) an architectural framework for constructing bespoke Grid middleware platforms in terms of ‘middleware domains’ such as extensible interaction types and resource discovery. We believe that such features will become increasingly essential with the emergence of next‐generation e‐Science applications. Copyright © 2005 John Wiley & Sons, Ltd.

Reflection is widely acknowledged as a useful mechanism for facilitating the run-time adaptation/ reconfiguration of software. Often, however, it is also thought to impose intolerably high overheads in performance-critical systems software environments like middleware platforms, operating systems, or programmable routers. In this paper we argue that there are many varieties of reflection (i.e. many types of 'meta-models') and that some of these in fact impose zero or negligible overhead. We further argue that in many cases, particularly cases involving highly dynamic software environments, reflection can actually enhance performance by laying open choices of alternative mechanisms that are best suited to current environmental conditions. Finally, we discuss the case of interception meta-models which, in many implementations, do lead to significant performance problems, and present our experience to date in attempting to minimise these problems.

Abstract Significant progress has been made in the design and development of object‐based distributed systems and, more recently, in component‐based reflective middleware platforms—i.e. middleware platforms that, through reflection, can be flexibly configured, and runtime adapted/reconfigured—especially in terms of non‐functional properties like quality of service (QoS). In this paper we consider how ideas and principles from such platforms can usefully be applied in a Grid context, specifically, an Open Grid Services Architecture (OGSA) context. To this end, we identify weaknesses in OGSA in the areas of generic service provision, scalability/performance engineering, and, especially, lack of support for QoS specification and realization, and suggest how these can be alleviated. We believe that such areas will become increasingly important as sophisticated e‐Science applications start to exploit the potential of OGSA's service‐based architecture. We also report on our current research on developing a binding‐type framework, a resource management framework, and a set of performance optimizations for an ongoing OGSA implementation we are working on. Copyright © 2004 John Wiley & Sons, Ltd.

One of the most significant limitations of current middleware platforms, both commercial and research, is that they typically support only a small, pre-defined, set of fundamental binding types (e.g., remote method invocation). This restriction limits the scope of platforms in that they cannot easily accommodate, or easily be extended to accommodate, richer or more specialized forms of interaction (e.g. events, media streaming, multicast, and many others discussed in the paper). This paper describes a highly extensible, component-oriented framework for the definition and implementation of such binding types. We motivate and specify the framework in detail and evaluate it by providing examples of its use.

The emergence of collaborative virtual world applications that run over the Internet has presented Virtual Reality (VR) application designers with new challenges. In an environment where the public internet streams multimedia data and is constantly under pressure to deliver over widely heterogeneous user-platforms, there has been a growing need that distributed virtual world applications be aware of and adapt to frequent variations in their context of execution. In this paper, we argue that in contrast to research efforts targeted at improvement of scalability, persistence and responsiveness capabilities, much less attempts have been aimed at addressing the flexibility, maintainability and extensibility requirements in contemporary Distributed VR applications. We propose the use of structural reflection as an approach that not only addresses these requirements but also offers added value in the form of providing a framework for scalability, persistence and responsiveness that is itself flexible, maintainable and extensible.

The authors introduce a reflective resource management framework that offers facilities for resource awareness and dynamic reallocation of resources for an adaptive middleware architecture. The main emphasis of the paper is the design and implementation of a resource management framework for such an architecture. We are not carrying out research in specific resource management algorithms; rather, we focus on providing a general framework which can accommodate best practice in the area. Such facilities ease the management of resources by providing a complete and consistent model of resources in the system at varying levels of abstractions. In addition, activities in the system are represented in a task model which provides a higher level of resource management.

The traditional approach to developing middleware platforms is to adopt a "black box" philosophy whereby the platform offers a fixed programming model to applications together with fixed per-platform implementations. In this paper, we describe research which is exploring an open approach to the implementation of middleware platforms. Our motivation is to accommodate the demanding requirements for quality-of-service (QoS) adaptation as imposed by mobile multimedia applications. We use an extended CORBA computational model featuring the concept of explicit open bindings. This provides an architectural framework for openness and QoS adaptation. The paper concludes by considering the more general application of an open systems philosophy; in particular, we introduce the concept of reflective middleware.

One of the major requirements of distributed multimedia applications is the need to maintain often complex, real-time synchronization constraints. More specifically, it is necessary to be able to manage arbitrary intra- and inter-media synchronization across activities in the distributed environment. Furthermore, it is important that such developments are integrated into emerging object-oriented standards for distributed computing. This paper presents an object-oriented programming model and associated implementation to meet these requirements. The main concepts behind the proposed approach are, firstly, the use of reactive objects for real-time control and synchronization and, secondly, quality of service controlled bindings for predictable communication between objects. The flexibility of the approach is demonstrated by three contrasting examples of real-time synchronization. The implementation extends the real-time capabilities of the Chorus micro-kernel by introducing the concepts of rtports, rthandlers and quality of service controlled connections. The paper demonstrates how reactive objects and bindings are realized on this infrastructure.

We describe the design of an application platform able to run distributed real-time and multimedia applications alongside conventional UNIX programs. The platform is embedded in a microkernel/PC environment and supported by an ATM-based, QoS-driven communications stack. In particular, we focus on resource-management aspects of the design and deal with CPU scheduling, network resource-management and memory-management issues. An architecture is presented that guarantees QoS levels of both communications and processing with varying degrees of commitment as specified by user-level QoS parameters. The architecture uses admission tests to determine whether or not new activities can be accepted and includes modules to translate user-level QoS parameters into representations usable by the scheduling, network, and memory-management subsystems.>

This article considers the problems of ensuring reliability in distributed cooperative systems through the transaction methodology. From the analysis of several transaction models, it is argued that the diversity of requirements in the distributed environment demands a novel approach. The transaction model proposed aims at providing the required flexibility. In this model, applications have more control over transaction management through the specification of policies for services such as concurrency control and recovery. The approach also features a number of mechanisms specifically tailored for distributed cooperative systems including notifications and reservations. The model feasibility is demonstrated by layering the transaction facilities on top of a distributed platform based on ANSA.>

There is currently considerable interest in developing multimedia applications in open distributed systems. However, it is now becoming clear that existing architectures for open distributed systems do not support the particular requirements of continuous media types such as digital audio and video. This is particularly the case in the important areas of quality of service support and real-time synchronization. This paper presents results from the Sumo project which aims at supporting continuous media types within the framework defined by the draft Open Distributed Processing standard. The paper advocates the use of synchronous languages within this framework for specifying and implementing real-time synchronization and QoS monitoring. A computational model and the realization of an infrastructure supporting this view are presented.

The combination of high-speed multiservice networks and multimedia workstations offers considerable potential for the development of distributed multimedia applications. A key problem is how to integrate continuous-media types such as audio and video into a distributed workstation environment. An experimental system architecture based on a specialized multimedia network interface that attempts to provide this integration is described. The design and implementation of this system are discussed in depth in terms of workstation enhancement and distributed system support. A new approach to the problem of media synchronization is introduced, and the importance of quality of service in the architecture is highlighted. Experiences encountered during this work are described, comparison with other approaches is made, and likely future developments in multimedia network interfacing are discussed.>

The authors argue that immutability is a suitable base on which to build distributed software engineering environments. They discuss the various approaches to maintaining consistency in immutable object systems and compare D.P. Reed's model of time domain addressing (ACM Trans. Comput. Syst., vol.1, no.1, p.3-23, Feb. 1983), with their own model of domain relative addressing. They demonstrate the suitability of domain relative addressing for use in distributed-software engineering environments.>

The recent developments in ‘fifth generation’ computing, for example in the areas of programming environments and man-machine interfaces, have shown up inadequacies in the current generation of operating systems. In addition, most operating systems are designed to run on a single computer (uniprocessor or multiprocessor) and to provide a multi-programming environment. Consequently, they cannot realise the full potential offered by a new class of computer systems consisting of computers interconnected by a high-speed local area network. In this paper we present a proposal for an operating system kernel to meet these new demands. We concentrate on one particular aspect of the design – its knowledge base. It is suggested that a knowledge base can provide support for more intelligent systems. It is further suggested that this knowledge base should be at the heart of the operating system design. Some examples are presented to illustrate the use of a knowledge base to support more intelligent applications and to provide a better environment for the user.

Abstract The UNIX operating system enjoys an ever increasing popularity throughout the computing community; there will be 1.4 million UNIX licences distributed by 1985, rising at a rate of about 400,000 per annum. With universal acceptance of a system comes a dangerously high degree of inertia. Consider the analogous area of programming languages, where there has been great resistance to change, despite major advancements. This paper presents a critique of UNIX, based on three areas which we consider to be of vital importance to future operating systems. These areas are operating system structures and design, programming support environments and distributed computing. The criticisms presented are in no way intended to discredit UNIX. UNIX compares favourably with most of the present generation of operating systems. The intention is to highlight deficiencies in the state of the art in operating system design.