This study characterizes GPU resilience in Delta, a large-scale AI system that consists of 1,056 A100 and H100 GPUs, with over 1,300 petaflops of peak throughput. We used 2.5 years of operational data (11.7 million GPU hours) on GPU errors. Our major findings include: (i) H100 GPU memory resilience is worse than A100 GPU memory, with 3.2x lower per-GPU MTBE for memory errors, (ii) The GPU memory error-recovery mechanisms on H100 GPUs are insufficient to handle the increased memory capacity, (iii) H100 GPUs demonstrate significantly improved GPU hardware resilience over A100 GPUs with respect to critical hardware components, (iv) GPU errors on both A100 and H100 GPUs frequently result in job failures due to the lack of robust recovery mechanisms at the application level, and (v) We project the impact of GPU node availability on larger-scales and find that significant overprovisioning of 5% is necessary to handle GPU failures.

In this paper, we i) analyze and classify real-world failures of Kubernetes (the most popular container orchestration system), ii) develop a framework to perform a fault/error injection campaign targeting the data store preserving the cluster state, and iii) compare results of our fault/error injection experiments with real-world failures, showing that our fault/error injections can recreate many real-world failure patterns. The paper aims to address the lack of studies on systematic analyses of Kubernetes failures to date. Our results show that even a single fault/error (e.g., a bit-flip) in the data stored can propagate, causing cluster-wide failures (3% of injections), service networking issues (4%), and service under/overprovisioning (24%). Errors in the fields tracking dependencies between object caused 51% of such cluster-wide failures. We argue that controlled fault/error injection-based testing should be employed to proactively assess Kubernetes' resiliency and guide the design of failure mitigation strategies.

Use cases for network determinism may involve the traversal of multiple network domains. A typical example is the edge-cloud virtualization of the controllers for industrial automation, where the end-to-end data path may include a private mobile network and a wired local-area network within the factory premises, a public transport network between the factory and the edge-cloud data center, and a switching fabric within the same data center. Maintaining deterministic guarantees throughout these domains, likely operated by different providers, is a daunting task when each domain relies on its own control and data-plane technologies for satisfaction of its allocated portion of the end-to-end requirements. We propose a universal model for determinism in wired packet networks that simplifies the realization of deterministic services over chains of network domains. The model builds on a novel traffic shaper where routing information constrains the packet scheduling decisions. The shaper, and therefore the provisioning of resources for every deterministic service, is only needed at the ingress edge of each wired domain. We conjecture perdomain latency bounds enforced by the shaper and corroborate them with experimental evidence. The bounds are easy to combine for multi-domain network paths.

Edge serverless computing has been widely used in mobile and web applications thanks to its simplicity and cost efficiency of network resource management. Serverless functions, event-driven and often stateless, are deployed on edge servers to reduce the latency caused by data transfer and the workload on cloud servers. However, in many edge serverless application scenarios, such as in IoT networks, the network is usually hierar-chical, with limited edge storage and computational capacity, and requests received by different edge nodes might be heterogeneous and time-varying. The function deployment policy becomes a key issue in achieving guaranteed network performances. In this paper, we focus on function allocation policy in edge serverless computing networks to help edge nodes dynamically choose the functions to deploy based on incoming requests, aiming to reduce latency and minimize request outage rate. A Deep Deterministic Policy Gradient (DDPG)-based function allocation algorithm is proposed for a multi-tier edge-serverless computing scenario where a set of functions are deployed to the edge nodes strategically. Simulation results show that our proposed method can reduce the latency of requests while also minimizing the outage rate.

One of the biggest emerging challenges for telco operators is to dynamically create new services while maintaining the network at its optimal performance/revenue break point. To this end, operators are moving to a much leaner cloud-based Software Defined Network (SDN) infrastructure to achieve a truly programmable network fabric. While cloud services can be provisioned in seconds and tested in-production, service provisioning in SDNs still lasts many weeks and requires substantial manual effort. A large part of this service creation time can be attributed to testing and tuning the control plane. In this paper we present SCP-CLUB (SDN Control Plane CLoUd-based Benchmarking), a platform for in-production performance testing of telco operator SDNs, offering a level of automation as available in deploying cloud services. Telco cloud SDN performance testing with SCP-CLUB focuses on the analysis of how design choices in the cloud and SDN control planes influence SLA metrics like throughput and latency. We describe the SCP-CLUB architecture and its performance testing support capabilities. Experiments are performed on an SDN telco cloud built to demonstrate SCP-CLUB under production load conditions.

Availability of the interconnection network in high-performance computing (HPC) systems is fundamental to sustaining the continuous execution of applications at scale. When failures occur, interconnect recovery mechanisms orchestrate complex operations to recover network connectivity between the nodes. As the scale and design complexity of HPC systems increase, so does the system's susceptibility to failures during execution of interconnect-recovery procedures. This study characterizes the recovery procedures of the Gemini interconnect network, the largest Gemini network built by Cray, on Blue Waters, a 13.3 petaflop supercomputer at the National Center for Supercomputing Applications (NCSA). We propose a propagation model that captures interconnect failures and recovery procedures to help understand types of failures and their propagation in both the system and applications during recovery. The measurements show that recovery procedures occur very frequently and that the unsuccessful execution of recovery procedures, when additional failures occur during recovery, causes system-wide outages (SWOs, 28 out of 101) and application failures (3.4 percent of all running applications).

A software-as-a-service (SaaS) needs to provide its intended service as per its stated service-level agreements (SLAs). While SLA violations in a SaaS platform have been reported, not much work has been done to empirically characterize failures of SaaS. In this paper, we study SLA violations of a production SaaS platform, diagnose the causes, unearth several critical failure modes, and then, suggest various solution approaches to increase the availability of the platform as perceived by the end user. Our approach combines field failure data analysis (FFDA) and fault injection. Our study is based on 283 days of operational logs of the platform. During this time, the platform received business workload from 42 customers spread over 22 countries. We have first developed a set of home-grown FFDA tools to analyze the log, and second implemented a fault injector to automatically inject several runtime errors in the application code written in .NET/C#, and then, collate the injection results. We summarize our finding as: first, system failures have caused 93% of all SLA violations; second, our fault injector has been able to recreate a few cases of bursts of SLA violations that could not be diagnosed from the logs; and third, the fault injection mechanism could recreate several error propagation paths leading to data corruptions that the failure data analysis could not reveal. Finally, the paper presents some system-level implication of this study and how the joint use of fault injection and log analysis may help in improving the reliability of the measured platform.

This paper presents an in-depth characterization of the resiliency of more than 5 million HPC application runs completed during the first 518 production days of Blue Waters, a 13.1 petaflop Cray hybrid supercomputer. Unlike past work, we measure the impact of system errors and failures on user applications, i.e., the compiled programs launched by user jobs that can execute across one or more XE (CPU) or XK (CPU+GPU) nodes. The characterization is performed by means of a joint analysis of several data sources, which include workload and error/failure logs. In order to relate system errors and failures to the executed applications, we developed LogDiver, a tool to automate the data pre-processing and metric computation. Some of the lessons learned in this study include: i) while about 1.53% of applications fail due to system problems, the failed applications contribute to about 9% of the production node hours executed in the measured period, i.e., the system consumes computing resources, and system-related issues represent a potentially significant energy cost for the work lost, ii) there is a dramatic increase in the application failure probability when executing full-scale applications: 20x (from 0.008 to 0.162) when scaling XE applications from 10,000 to 22,000 nodes, and 6x (from 0.02 to 0.129) when scaling GPU/hybrid applications from 2000 to 4224 nodes, and iii) the resiliency of hybrid applications is impaired by the lack of adequate error detection capabilities in hybrid nodes.

This paper provides an analysis of failures and their impact for Blue Waters, the Cray hybrid (CPU/GPU) supercomputer at the University of Illinois at Urbana-Champaign. The analysis is based on both manual failure reports and automatically generated event logs collected over 261 days. Results include i) a characterization of the root causes of single-node failures, ii) a direct assessment of the effectiveness of system-level fail over as well as memory, processor, network, GPU accelerator, and file system error resiliency, and iii) an analysis of system-wide outages. The major findings of this study are as follows. Hardware is not the main cause of system downtime. This is notwithstanding the fact that hardware-related failures are 42% of all failures. Failures caused by hardware were responsible for only 23% of the total repair time. These results are partially due to the fact that processor and memory protection mechanisms (x8 and x4 Chip kill, ECC, and parity) are able to handle a sustained rate of errors as high as 250 errors/h while providing a coverage of 99.997% out of a set of more than 1.5 million of analyzed errors. Only 28 multiple-bit errors bypassed the employed protection mechanisms. Software, on the other hand, was the largest contributor to the node repair hours (53%), despite being the cause of only 20% of the total number of failures. A total of 29 out of 39 system-wide outages involved the Lustre file system with 42% of them caused by the inadequacy of the automated fail over procedures.

This paper investigates SLA violations of a production SaaS platform by means of joint use of field failure data analysis (FFDA) and fault injection. The objective of this study is to diagnose the causes of SLA violations, pinpoint critical failure modes under realistic error assumptions and identify potential means to increase the user perceived availability of the platform and assurance of SLA requirements. We base our study on 283 days of logs obtained during the production time of the platform, while it was employed to process business data received by 42 customers in 22 countries. In this paper, we develop a set of tools that include i) a FFDA toolset used to analyze the data extracted from the platform and by the operating system event logs and ii) a. NET/C++ injector able to automate the injection of specific runtime errors in the production code and the collection of results. Major findings include i) 93% of all service level agreement (SLA) violations were due to system failures, ii) there were a few cases of bursts of SLA violations that could not be diagnosed from the logs and were revealed from the performed injections, and iii) the error injection revealed several error propagation paths leading to data corruptions that could not be detected from the analysis of failure data.

This paper presents a novel approach to assess time coalescence techniques. These techniques are widely used to reconstruct the failure process of a system and to estimate dependability measurements from its event logs. The approach is based on the use of automatically generated logs, accompanied by the exact knowledge of the ground truth on the failure process. The assessment is conducted by comparing the presumed failure process, reconstructed via coalescence, with the ground truth. We focus on supercomputer logs, due to increasing importance of automatic event log analysis for these systems. Experimental results show how the approach allows to compare different time coalescence techniques and to identify their weaknesses with respect to given system settings. In addition, results revealed an interesting correlation between errors caused by the coalescence and errors in the estimation of dependability measurements.

As Wireless Sensor Networks (WSNs) are starting to be adopted in critical scenarios, it becomes important to study the behavior of WSN software in response to errors induced by hardware faults. To this aim, in this paper we present the results of an extensive fault-injection campaign, conducted on three different WSN operating systems (OSs). Results show that, depending on the concurrency model and on the memory management, the OS reacts to injected faults differently, indicating that fault containment strategies and hang-checking assertions should be implemented to avoid spreading and activations of errors. Analysis also allowed us to define a detailed dependability model of the WSN software, to be used to simulate the expected failure behavior of a given OS when solicited by given low-level hardware faults.

Wireless Sensor Networks (WSNs) are widely recognized as a promising solution to build next-generation monitoring systems. Their industrial uptake is however still compromised by the low level of trust on their performance and dependability. Whereas analytical models represent a valid mean to assess nonfunctional properties via simulation, their wide use is still limited by the complexity and dynamicity of WSNs, which lead to unaffordable modeling costs. To reduce this gap between research achievements and industrial development, this paper presents a framework for the assessment of WSNs based on the automated generation of analytical models. The framework hides modeling details, and it allows designers to focus on simulation results to drive their design choices. Models are generated starting from a high-level specification of the system and by a preliminary characterization of its fault-free behavior, using behavioral simulators. The benefits of the framework are shown in the context of two case studies, based on the wireless monitoring of civil structures.

This paper presents an effective approach for injecting faults/errors in WSN nodes operating systems. The approach is based on the injection of faults at the assembly level. Results show that depending on the concurrency model and on the memory management, the operating systems react to injected errors differently, indicating that fault containment strategies and hang-checking assertions should be implemented to avoid spreading and activations of errors.

As the incidence of faults in real Wireless Sensor Networks (WSNs) increases, fault injection is starting to be adopted to verify and validate their design choices. Following this recent trend, this paper presents a tool, named AVR-INJECT, designed to automate the fault injection, and analysis of results, on WSN nodes. The tool emulates the injection of hardware faults, such as bit flips, acting via software at the assembly level. This allows to attain simplicity, while preserving the low level of abstraction needed to inject such faults. The potential of the tool is shown by using it to perform a large number of fault injection experiments, which allow to study the reaction to faults of real WSN software.

This paper proposes a flexible framework for dependability modeling and assessing of Wireless Sensor Networks (WSNs). The framework takes into account network related aspects (topology, routing, network traffic) as well as hardware/software characteristics of nodes (type of sensors, running applications, power consumption). It is composed of two basic elements: i) a parametric Stochastic Activity Networks (SAN) failure model, reproducing WSN failure behavior as inferred from a detailed Failure Mode Effect Analysis (FMEA), and ii) an external library reproducing network behavior on behalf of the SAN model. This library specializes the SAN model by feeding it with quantitative parameters obtained by simulation or by experimental campaigns; it is also in charge of updating the network state in response to failure events during the simulation (e.g., routing tree updated due to node failures). The framework is thus suited to evaluate the dependability of several WSNs, with different topologies, routing algorithms, hardware/software platforms, without requiring any changes to its structure. The use of the external library makes the model simpler, decoupling the network behavior from the failure behavior. Simulation experiments are discussed that provide a quantitative evaluation of WSN dependability for a sample scenario: results show how the proposed framework supports WSN developers to find proper cost-reliability trade-offs for the system being deployed.