Managing complex distributed applications in the Cloud-Edge continuum, including deployment on diverse resources and runtime operations, presents significant challenges. Orchestrators play a key role by automating resource discovery, optimisation, deployment, and life-cycle management while ensuring system performance. This paper introduces Swarmchestrate, a decentralised, application-centric orchestration framework inspired by self-organising Swarms. Our initial findings, based on the implementation in a Cloud-Edge simulator, demonstrate Swarmchestrate's potential, offering insights into resource coordination and optimised allocation for scalable systems.

With the emergence of Internet of Things (IoT) devices collecting large amounts of data at the edges of the network, a new generation of hyper-distributed applications is emerging, spanning cloud, fog, and edge computing resources.The automated deployment and management of such applications requires orchestration tools that take a deployment descriptor (e.g.Kubernetes manifest, Helm chart or TOSCA) as input, and deploy and manage the execution of applications at run-time.While most deployment descriptors are prepared by a single person or organisation at one specific time, there are notable scenarios where such descriptors need to be created collaboratively by different roles or organisations, and at different times of the application's life cycle.An example of this scenario is the modular development of digital twins, composed of the basic building blocks of data, model and algorithm.Each of these building blocks can be created independently from each other, by different individuals or companies, at different times.The challenge here is to compose and build a deployment descriptor from these individual components automatically.This paper presents a novel solution to automate the collaborative composition and generation of deployment descriptors for distributed applications within the cloud-to-edge continuum.The implemented solution has been prototyped in over 25 industrial use cases within the DIGITbrain project, one of which is described in the paper as a representative example.

In June 2023, the Imperial College London Graduate School's Research Computing and Data Science group invited thirty PhD students for a one-day Hackathon on AI-assisted programming. This poster abstract presents our experiences in planning and running the event, and shares student and organiser reflections on the event and where it might lead.

Abstract Science gateways have been widely utilized by a large number of user communities to simplify access to complex distributed computing infrastructures. While science gateways are still becoming increasingly popular and the number of user communities is growing, the fast and efficient creation of new science gateways and the flexibility to deploy these gateways on‐demand on heterogeneous computational resources, remain a challenge. Additionally, the increase in the number of users, especially with very different backgrounds, requires intuitive embedded e‐learning tools that support all stakeholders to find related learning material and to guide the learning process. This paper introduces a novel science gateway framework that addresses these challenges. The framework supports the creation, publication, selection, and deployment of cloud‐based reference architectures that can be automatically instantiated and executed even by nontechnical users. The framework also incorporates a knowledge repository exchange and learning module that provides embedded e‐learning support. To demonstrate the feasibility of the proposed solution, two scientific case studies are presented based on the requirements of the plasmasphere, ionosphere, and thermosphere research communities.

Summary As cloud adoption increases, so do the number of available cloud service providers. Moving complex applications between clouds can be beneficial—or other times necessary—but achieving this so‐called cloud portability is rarely straightforward. This article presents the adoption of OASIS TOSCA, a standard in the declarative description of cloud applications, to encourage and facilitate cloud portability in MiCADO, an application‐level multi‐cloud orchestration and auto‐scaling framework. The interface to MiCADO is an Application Description Template, which draws from the TOSCA specification to describe an application in MiCADO. The generic design of these templates is presented and their applicability for achieving portability between different container and cloud environments is analysed and evaluated. A proof‐of‐concept where MiCADO serves as the deployment and execution engine for a Science Gateway in Sleep Healthcare is then described. In this proof‐of‐concept, MiCADO facilitates the deployment of a complex healthcare application, which is then moved from one cloud service provider to another with only minimal changes to the template which originally described it. This TOSCA‐based approach to templates in MiCADO encourages movement between clouds by making cloud portability more approachable.

Abstract Automated deployment and run-time management of microservices-based applications in cloud computing environments is relatively well studied with several mature solutions. However, managing such applications and tasks in the cloud-to-edge continuum is far from trivial, with no robust, production-level solutions currently available. This paper presents our first attempt to extend an application-level cloud orchestration framework called MiCADO to utilise edge and fog worker nodes. The paper illustrates how MiCADO-Edge can automatically deploy complex sets of interconnected microservices in such multi-layered cloud-to-edge environments. Additionally, it shows how monitoring information can be collected from such services and how complex, user- defined run-time management policies can be enforced on application components running at any layer of the architecture. The implemented solution is demonstrated and evaluated using two realistic case studies from the areas of video processing and secure healthcare data analysis.

In the big data era, creating self-managing scalable platforms for running big data applications is a fundamental task. Such self-managing and self-healing platforms involve a proper reaction to hardware (e.g., cluster nodes) and software (e.g., big data tools) failures, besides a dynamic resizing of the allocated resources based on overload and underload situations and scaling policies. The distributed and stateful nature of big data platforms (e.g., Hadoop-based cluster) makes the management of these platforms a challenging task. This paper aims to design and implement a scalable cloud native Hadoopbased big data platform using MiCADO, an open-source, and a highly customisable multi-cloud orchestration and auto-scaling framework for Docker containers, orchestrated by Kubernetes. The proposed MiCADO-based big data platform automates the deployment and enables an automatic horizontal scaling (in and out) of the underlying cloud infrastructure. The empirical evaluation of the MiCADO-based big data platform demonstrates how easy, efficient, and fast it is to deploy and undeploy Hadoop clusters of different sizes. Additionally, it shows how the platform can automatically be scaled based on user-defined policies (such as CPU-based scaling).

Abstract Typical cloud applications require high-level policy driven orchestration to achieve efficient resource utilisation and robust security to support different types of users and user scenarios. However, the efficient and secure utilisation of cloud resources to run applications is not trivial. Although there have been several efforts to support the coordinated deployment, and to a smaller extent the run-time orchestration of applications in the Cloud, no comprehensive solution has emerged until now that successfully leverages applications in an efficient, secure and seamless way. One of the major challenges is how to specify and manage Quality of Service (QoS) properties governing cloud applications. The solution to address these challenges could be a generic and pluggable framework that supports the optimal and secure deployment and run-time orchestration of applications in the Cloud. A specific aspect of such a cloud orchestration framework is the need to describe complex applications incorporating several services. These application descriptions must specify both the structure of the application and its QoS parameters, such as desired performance, economic viability and security. This paper proposes a cloud technology agnostic approach to application descriptions based on existing standards and describes how these application descriptions can be processed to manage applications in the Cloud.

There are many scientific and commercial applications that require the execution of a large number of independent jobs resulting in significant overall execution time. Therefore, such applications typically require distributed computing infrastructures and science gateways to run efficiently and to be easily accessible for end-users. Optimising the execution of such applications in a cloud computing environment by keeping resource utilisation at minimum but still completing the experiment by a set deadline has paramount importance. As container-based technologies are becoming more widespread, support for job-queuing and auto-scaling in such environments is becoming important. Current container management technologies, such as Docker Swarm or Kubernetes, while provide auto-scaling based on resource consumption, do not support job queuing and deadline-based execution policies directly. This paper presents JQueuer, a cloud-agnostic queuing system that supports the scheduling of a large number of jobs in containerised cloud environments. The paper also demonstrates how JQueuer, when integrated with a cloud application-level orchestrator and auto-scaling framework, called MiCADO, can be used to implement deadline-based execution policies. This novel technical solution provides an important step towards the cost-optimisation of batch processing and job submission applications. In order to test and prove the effectiveness of the solution, the paper presents experimental results when executing an agent-based simulation application using the open source REPAST simulation framework.