Stream Processing (SP) is now a major paradigm to timely handle large volumes of data generated at the edge of the Internet. Stream processing engines (SPE) are tools easing the specification, deployment and monitoring of SP applications. Such applications are typically programmed as a directed acyclic graph (DAG) of operators to be applied on each data item. Yet, SPEs are mostly equipped to deploy one application at a time without seeking synergies between those applications. Yet, in many domains, the set of operators composing applications overlap for a non-negligible amount. We envision a platform on which applications are submitted dynamically, each new graph of operators potentially sharing some of them with the currently running operators. We assume a homogeneous platform, a graph being deployed over multiple nodes. We need to minimize the inter-node traffic while guaranteeing that the capacity of a node is not exceeded. This paper presents the Merge, Split and Cluster approach: each time a new DAG of operators is submitted, i) its operators are first merged with the already running operators, ii) if an operator's load thus created exceeds the nodes' capacity, the operators gets split into several instances, and iii) the operators of the resulting graph are clustered, each cluster being hosted by a single node so as to maximize intra-node traffic. Two heuristics are proposed for this last phase. Simulation results show that i) merging allows to drastically reduce the needs in computing resources, and ii) that the heuristic provides an efficient clustering minimizing the intra-node traffic.

The constant growth of maritime traffic leads to the need of automatic anomaly detection, which has been attracting great research attention. Information provided by AIS (Automatic Identification System) data, together with recent outstanding progresses of deep learning, make vessel monitoring using neural networks (NNs) a very promising approach. This paper analyse a novel neural network we have recently introduced -GeoTrackNet- regarding operational contexts. Especially, we aim to evaluate (i) the relevance of the abnormal behaviours detected by GeoTrackNet with respect to expert interpretations, (ii) the extent to which GeoTrackNet may process AIS data streams in real time. We report experiments showing the high potential to meet operational level of the model.

Over the last decade, stream processing (SP) gained momentum as an efficient paradigm dealing with the real-time processing of large streams of data. As mature as they are, mainstream stream processing engines are not ready to be used over more distributed platforms that are today emerging such as Fog platforms. Streams being produced at the edge of the platform, and edges being composed of less powerful compute nodes, a decentralized approach of Stream Processing is needed. This paper moves a step towards decentralized SP by proposing a decentralized autoscaling mechanism for SP applications. Autoscaling refers to the ability for a system to automatically scale in and out as the load varies. As a stream processing application is generally composed of a set of operators in a pipeline, our protocol lets each operator's replica take scaling decisions and enforce them independently. Assuming a fully decentralized setting, we consider each replica has only a partial view of the graph of operators. Then, without care, concurrent distributed scaling could lead to inconsistencies in replicas' views and to data loss. To address this issue, we devise a maintenance protocol ensuring the consistency of replicas' views, no data loss and no downtime during reconfigurations. Finally, the paper describes the early experimentations conducted with a software prototype of a decentralized SP engine over a computing cluster, showing the usability of the mechanism.

Microservices architectures combine the use of fine-grained and independently-scalable services with lightweight communication protocols, such as REST calls over HTTP. Microservices bring flexibility to the development and deployment of application back-ends in the cloud. Applications such as collaborative editing tools require frequent interactions between the front-end running on users' machines and a back-end formed of multiple microservices. User-perceived latencies depend on their connection to microservices, but also on the interaction patterns between these services and their databases. Placing services at the edge of the network, closer to the users, is necessary to reduce user-perceived latencies. It is however difficult to decide on the placement of complete stateful microservices at one specific core or edge location without trading between a latency reduction for some users and a latency increase for the others. We present how to dynamically deploy microservices on a combination of core and edge resources to systematically reduce user-perceived latencies. Our approach enables the split of stateful microservices, and the placement of the resulting splits on appropriate core and edge sites. Koala, a decentralized and resource-driven service discovery middleware, enables REST calls to reach and use the appropriate split, with only minimal changes to a legacy microservices application. Locality awareness using network coordinates further enables to automatically migrate services split and follow the location of the users. We confirm the effectiveness of our approach with a full prototype and an application to ShareLatex, a microservices-based collaborative editing application.

Current cloud computing infrastructures and their management are highly centralized, and therefore they suffer from limitations in terms of network latency, energy consumption, and possible legal restrictions. Decentralizing the Cloud has been recently proposed as an alternative. However, the efficient management of a geographically dispersed platform brings new challenges related to service localization, network utilization and locality-awareness. We here consider a cloud topology composed of many small datacenters geographically dispersed within the backbone network. In this paper, we present the design, development and experimental validation of Koala, a novel overlay network that specifically targets such a geographically distributed cloud platform. The three key characteristics of Koala are laziness, latency-awareness and topology-awareness. By using application traffic, Koala maintains the overlay lazily while it takes locality into account in each routing decision. Although Koala's performance depends on application traffic, through simulation experiments we show that for a uniformly distributed traffic, Koala delivers similar routing complexity and reduced latency compared to a traditional proactive protocol, such as Chord. Additionally, we show that despite its passive maintenance, Koala can appropriately deal with churn by keeping the number of routing failures low, without significantly degrading the routing performance. Finally, we show how such an overlay adapts to a decentralized cloud composed of multiple small datacenters.

Recent increase in demand for next-to-source data processing and low-latency applications has shifted attention from the traditional centralized cloud to more distributed models such as edge computing. In order to fully leverage these models it is necessary to decentralize not only the computing resources but also their management. While a decentralized cloud has various inherent advantages, it also introduces different challenges with respect to coordination and collaboration between resources. A large-scale system with multiple administrative entities requires an overlay network which enables data and service localization based only on a partial view of the network. Numerous existing overlay networks target different properties but they are built in a generic context, without taking into account the specific requirements of a decentralized cloud. In this paper we identify some of these requirements and introduce Koala, a novel overlay network designed specifically to meet them.

Workflow-based computing has become a dominant paradigm to design and execute scientific applications. After the initial breakthrough of now standard workflow management systems, several approaches have recently proposed to decentralise the coordination of the execution. In particular, shared space-based coordination has been shown to provide appropriate building blocks for such a decentralised execution. Uncertainty is also still a major concern in scientific workflows. The ability to adapt the workflow, change its shape and switch for alternate scenarios on-the-fly is still missing in workflow management systems. In this paper, based on a shared space approach, we firstly devise a programmatic way to specify such adaptive workflows. We use a reactive, rule-based programming model to modify the workflow description by changing its associated direct acyclic graph on-the-fly without needing to stop and restart the execution from the beginning. Secondly, we present the GinFlow middleware, a resilient decentralised workflow execution manager implementing these concepts. Through a set of deployments of adaptive workflows of different characteristics, we discuss the GinFlow performance and resilience and show the limited overhead of the adaptiveness mechanism, making it a promising decentralised adaptive workflow execution manager.

With the recent widespread adoption of service-oriented architecture, the dynamic composition of services is now a crucial issue in the area of distributed computing. The coordination and execution of composite Web services are today typically conducted by heavyweight centralized workflow engines, leading to an increasing probability of processing and communication bottlenecks and failures. In addition, centralization induces higher deployment costs, such as the computing infrastructure to support the workflow engine, which is not affordable for a large number of small businesses and end-users. In a world where platforms are more and more dynamic and elastic as promised by cloud computing, decentralized and dynamic interaction schemes are required. Addressing the characteristics of such platforms, nature-inspired analogies recently regained attention to provide autonomous service coordination on top of dynamic large scale platforms. In this paper, we propose an approach for the decentralized execution of composite Web services based on an unconventional programming paradigm that relies on the chemical metaphor. It provides a high-level execution model that allows executing composite services in a decentralized manner. Composed of services communicating through a persistent shared space containing control and data flows between services, our architecture allows to distribute the composition coordination among nodes. A proof of concept is given, through the deployment of a software prototype implementing these concepts, showing the viability of an autonomic vision of service composition.

With the proliferation of Web Services, scientific applications are more and more designed as temporal compositions of services, commonly referred to as, workflows. To address this paradigm shift, different workflow management systems have been proposed. If their efficiency has been established over centralized reliable systems, it is questionable over highly decentralized failure-prone platforms. Scientific applications recently started to be deployed over clouds, leading to new issues, like elasticity, i.e., the possibility to dynamically refine, at runtime, the amount of resources dedicated to an application. This raised a new demand for programming models, able to express autonomic self-coordination of services in a dynamic, elastic platform. Chemistry-inspired computing recently regained momentum in this context, naturally expressing parallelism, distribution, and autonomic behaviors. While its high expressiveness and adequacy for this context has been established, the chemical model severely suffers from a lack of proof of concepts. In this paper, we concretely show how to leverage such models in this context. We focus on the design, the implementation and the experimental validation of a chemistry-inspired scientific workflow management system.

Nowadays, executions of composite Web services are typically conducted by heavyweight centralized engines. A centralized is a potential processing and communication bottleneck as well as a central point of failure, and its cost of deployment is usually too high for a large number of small businesses or end-users. In addition, it presents some weaknesses dealing with privacy issues. These drawbacks, and requirements such as loose coupling, are pushing service infrastructures toward more decentralized and dynamic interaction schemes. In this paper, we propose a decentralized alternative system for the execution of composite Web services based on the chemical analogy. The chemical paradigm provides a high-level execution model that allows executing composite services in a fully decentralized manner. Our architecture is composed by nodes communicating through a shared space containing both control and data flows, allowing to distribute the composition among nodes without the need for any centralized coordination.

Chord is a protocol to distribute and retrieve information at large scale. It builds a large but rigid overlay network without taking into account the social nature and the underlying topology of large platforms, made of the interconnection of many independent smaller networks. Thus, new approaches are required to build overlay networks. In this paper, we propose Babelchord, a more flexible and social overlay interconnecting different Chord networks, which are floors of a social tower. Peers can belong to several floors, allowing this interconnection. By connecting smaller structured overlay networks in an unstructured way, it provides a costeffective alternative to hierarchical structured P2P systems requiring costly merging. Routing of lookup messages is performed as in Chord within one floor, but a peer belonging to several floors forwards the request to the different floors it belongs to. These co-located peers act as a sort of neural synapse. Results from simulations show that Babelchord scales up logarithmically with the number of Babelchord peers. Moreover a small number of synapses is enough to ensure a high exhaustiveness level.

The efficiency of service discovery is a crucial point in the development of fully decentralized middlewares intended to manage large scale computational grids. The work conducted on this issue led to the design of many peer- to-peer fashioned approaches. More specifically, the need for flexibility and complexity in the service discovery has seen the emergence of a new kind of overlays, based on tries, also known as lexicographic trees. Although these overlays are efficient and well designed, they require a costly maintenance and do not accurately take into account the heterogeneity of nodes and the changing popularity of the services requested by users. In this paper, we focus on reducing the cost of the maintenance of a particular architecture, based on a dynamic prefix tree, while enhancing it with some load balancing techniques that dynamically adapt the load of the nodes in order to maximize the throughput of the system. The algorithms developed couple a self-organizing prefix tree overlay with load balancing techniques inspired by similar previous works undertaken for distributed hash tables. After some simulation results showing how our load balancing heuristics perform in such an overlay and compare to other heuristics, we provide a fair comparison of this architecture and similar overlays recently proposed.

The efficiency of service discovery is critical in the development of fully decentralized middleware intended to manage large scale computational grids. This demand influenced the design of many peer-to-peer based approaches. The ability to cope with the expressiveness of the service discovery was behind the design of a new kind of overlay structures that is based on tries, or prefix trees. Although these overlays are well designed, one of their weaknesses is the lack of any concrete fault tolerant mechanism, especially in dynamic platforms; the faults are handled by using preventive and costly mechanisms, \eg using a high degree of replication. Moreover, those systems cannot handle any arbitrary transient failure. Self-stabilization, which is an efficient approach to designreliable solutions for dynamic systems, was recently suggested to be a good alternative to inject fault-tolerance in peer-to-peer systems. However, most of the previous research on self-stabilization in tree and/or P2P networks was designed in theoretical models, making these approaches hard to implement in practice. In this paper, we provide a self-stabilizing message passing protocol to maintain prefix trees over practical peer-to-peer networks. A complete correctness proof is provided, as well as simulation results to estimate the practical impact of our protocol.

Within computational grids, some services (software components, linear algebra libraries, etc.) are made available by some servers to some clients. In spite of the growing popularity of such grids, the service discovery, although efficient in many cases, does not reach several requirements. Among them, the flexibility of the discovery and its efficiency on wide-area dynamic platforms are two major issues. Therefore, it becomes crucial to propose new tools coping with such platforms. Emerging peer-to-peer technologies provide algorithms allowing the distribution and the retrieval of data items while addressing the dynamicity of the underlying network. We study in this paper the service discovery in a pure peer-to-peer environment. We describe a new trie-based approach for the service discovery that supports range queries and automatic completion of partial search strings, while providing fault-tolerance, and partially taking into account the topology of the underlying network. We validate this approach both by analysis and simulation. Traditional metrics considered in peer-to-peer systems exhibits interesting complexities within our architecture. The analysis' results are confirmed by some simulation experiments run using several grid's data sets

DIET (Distributed Interactive Engineering Toolbox) is a set of hierarchical components to design network enabled server (NES) systems. In these systems, clients ask to agents (discovery and scheduling components) to find servers able to solve their problem using some performance metrics and information about the location of data already on the network. Today's NES middleware, in which agents are statically connected and potentially bottlenecks, don't cope with the dynamic and heterogeneous nature of future grid environments. In this paper, we present the design, the implementation and the experimentation of the first architecture extending traditional NES systems with an unstructured peer-to-peer network dynamically connecting distributed agents, to provide to clients an entry point to servers geographically distributed. Our implementation is based on DIET and the JXTA toolbox. Different algorithms have been implemented and experimented over the VTHD network which connects several supercomputers in different research institutes through a high-speed network, showing the scalability of the system