In this article, we present a timing analysis of orchestration times for containerized services, revealing the inability of current container orchestrators to fully prioritize services under concurrent requests. The analysis identifies the sources of orchestration delays that impact services to be prioritized potentially violating their Service Level Objectives (SLOs). Based on the findings of the timing analysis, we highlight three alternative SLO-aware orchestration system designs aimed at preventing and/or mitigating delays for high-priority services. We provide principles and guidelines that must drive the implementation of these designs. We then introduce Ulysses , a Kubernetes -based prototype embodying the simplest of the three designs. Ulysses modifies the core Kubernetes control plane components to manage events synchronously and with fixed priority. Through experiments conducted with both synthetic workloads and a containerized cloud-native 5G core network, we demonstrate that Ulysses ensures stable orchestration times for high-priority services, with a reduction of up to 78% under high orchestration load.

In this paper, we design and implement a fixedpriority and fully preemptable controller for Kubernetes. The controller is designed to manage mixed-criticality services, handling orchestration requests and cluster events with a priority level that matches each service’s criticality. The controller aims at providing predictable and schedulable orchestration times according to services’ priority, even in the presence of interfering orchestration events. Experimental results show a reduction of up to 99% of the time spent in the control plane to handle highpriority requests.

Containers and microVMs are ubiquitous solutions to virtualize components and foster flexible and elastic edge/cloud settings. However, they can suffer from timing and failure interferences, jeopardizing the adoption in mixed-criticality systems.This paper presents a framework for zero-interference containers (ZICs), i.e., applications running in an isolated partition handled by a partitioning hypervisor but managed as containers. The framework includes: i) a container runtime for ZICs; ii) an image manager that automates the creation and seamlessly downloads ZIC images; iii) a tool to build replicable environments based on partitioning hypervisors. The container runtime allows integrating software components with strict real-time and dependability requirements into orchestration platforms (e.g., Kubernetes), fostering novel industrial use cases. The building tool and image manager facilitate ZICs development and testing through replicable environments and transparent image management. Experimental results show that ZICs survive a management VM crash and can guarantee timeliness despite heavy co-located stresses.

In this paper, we i) analyze and classify real-world failures of Kubernetes (the most popular container orchestration system), ii) develop a framework to perform a fault/error injection campaign targeting the data store preserving the cluster state, and iii) compare results of our fault/error injection experiments with real-world failures, showing that our fault/error injections can recreate many real-world failure patterns. The paper aims to address the lack of studies on systematic analyses of Kubernetes failures to date. Our results show that even a single fault/error (e.g., a bit-flip) in the data stored can propagate, causing cluster-wide failures (3% of injections), service networking issues (4%), and service under/overprovisioning (24%). Errors in the fields tracking dependencies between object caused 51% of such cluster-wide failures. We argue that controlled fault/error injection-based testing should be employed to proactively assess Kubernetes' resiliency and guide the design of failure mitigation strategies.

The evolution of industrial environments makes the reconfigurability and flexibility key requirements to rapidly adapt to changeable market needs. Computing paradigms like Edge/Fog computing are able to provide the required flexibility and scalability while guaranteeing low latencies and response times. Orchestration systems play a key role in these environments, enforcing automatic management of resources and workloads’ lifecycle, and drastically reducing the need for manual interventions. However, they do not currently meet industrial non-functional requirements, such as real-timeliness, determinism, reliability, and support for mixed-criticality workloads. In this article, we present k4.0s, an orchestration system for Industry 4.0 (I4.0) environments, which enables the support for real-time and mixed-criticality workloads. We highlight through experiments the need for novel monitoring approaches and propose a workflow for selecting monitoring metrics, which depends on both workload requirements and hosting node guarantees. We introduce new abstractions for the components of a cluster in order to enable criticality-aware monitoring and orchestration of real-time industrial workloads. Finally, we design an orchestration system architecture that reflects the proposed model, introducing new components and prototyping a Kubernetes-based implementation, taking the first steps towards a fully I4.0-enabled orchestration system.

Real-time containers are a promising solution to reduce latencies in time-sensitive cloud systems. Recent efforts are emerging to extend their usage in industrial edge systems with mixed-criticality constraints. In these contexts, isolation becomes a major concern: a disturbance (such as timing faults or unexpected overloads) affecting a container must not impact the behavior of other containers deployed on the same hardware. In this paper, we propose a novel architectural solution to achieve isolation in real-time containers, based on real-time co-kernels, hierarchical scheduling, and time-division networking. The architecture has been implemented on Linux patched with the Xenomai co-kernel, extended with a new hierarchical scheduling policy, named SCHED_DS, and integrating the RTNet stack. Experimental results are promising in terms of overhead and latency compared to other Linux-based solutions. More importantly, the isolation of containers is guaranteed even in presence of severe co-located disturbances, such as faulty tasks (elapsing more time than declared) or high CPU, network, or I/O stress on the same machine.