High-accurate machine learning (ML) image classifiers cannot guarantee that they will not fail at operation. Thus, their deployment in safety-critical applications such as autonomous vehicles is still an open issue. The use of fault tolerance mechanisms such as safety monitors is a promising direction to keep the system in a safe state despite errors of the ML classifier. As the prediction from the ML is the core information directly impacting safety, many works are focusing on monitoring the ML model itself. Checking the efficiency of such monitors in the context of safety-critical applications is thus a significant challenge. Therefore, this paper aims at establishing a baseline framework for benchmarking monitors for ML image classifiers. Furthermore, we propose a framework covering the entire pipeline, from data generation to evaluation. Our approach measures monitor performance with a broader set of metrics than usually proposed in the literature. Moreover, we benchmark three different monitor approaches in 79 benchmark datasets containing five categories of out-of-distribution data for image classifiers: class novelty, noise, anomalies, distributional shifts, and adversarial attacks. Our results indicate that these monitors are no more accurate than a random monitor. We also release the code of all experiments for reproducibility.

Nanocomputing and related-enabling technologies hold the promise of higher performance and lower power consumption, as well as increased communication capabilities and functionality. In addition to the impact on today computerized systems, nanocomputing is an essential lever to foster the emerging cyberphysical system paradigm. However, the dependability and security of these unprecedentedly small devices, of their deployment, and of their interconnection remain uncertain. The main sources of concern are: • Nanometer devices are expected to be highly sensitive to process variations. The guard-bands used today for avoiding the impact of such variations will not represent a feasible solution in the future. As a consequence, timing errors and their higher frequency of occurrence have to be addressed. • New and intricate failure modes, specific to new materials, are expected to raise serious challenges to the design and test engineers. • Environment induced errors, such as single event upsets (SEU), are likely to occur more frequently than in the case of more conventional semiconductor devices. • Design of hardware architectures encompassing resilience techniques are needed to achieve the development of highly reliable energy efficient systems. • The increased complexity of the systems based on nanotechnology will require improved computer aided design (CAD) tools, as well as better validation techniques. • The security of nanocomputing systems may be threatened by malicious attacks targeting new vulnerable areas in the hardware.

Nanocomputing technologies hold the promise for higher performance, lower power consumption as well as increased functionality. However, the dependability of these unprecedentedly small scale devices remains uncertain. The main sources of concern are: • Nanometer devices are expected to be highly sensitive to process variations. The guard-bands used today for avoiding the impact of such variations will not represent a feasible solution in the future. Thus, timing errors may occur more frequently. • New failure modes, specific to new materials, are expected to raise serious challenges to the design and test engineers. • Environment induced errors, like single event upsets (SEU), are likely to occur more frequently than in the case of conventional semiconductor devices. • New hardware redundancy techniques are needed to enable development of energy efficient systems. • The increased complexity of the systems based on nanotechnology will require improved computer aided design (CAD) tools, as well as better validation techniques. • Security of nanocomputing systems may be threatened by malicious attacks targeting new vulnerable areas in the hardware.

As already witnessed today, and as foreseen, concerns attached to hardware will play an increasing role in the design and assessment of dependable and secure computing systems.

There is still a strong demand for solutions to improve dependability and security. Issues at stake include threats ranging from managing complexity and manufacturing defects to accelerated aging and susceptibility to environment disturbances. These solutions extend beyond hardware, microelectronics or physics; major computing- related issues spanning various aspects are also essential: architecture, networks, communication, synchronization, task parallelization, etc. The workshop is aimed at further characterizing these impairments and threats, as well as distinguishing possible alternative design approaches and operation control paradigms that have to be enforced and/or favored in order to keep achieving dependable and secure computing. Finally, it is worth pointing out that besides design techniques aimed at providing dependable and secure computing, assessment techniques (risk evaluation, validation, testing, etc.) should also play a fundamental role! For all these reasons, we believe that DSN is an ideal forum to address all these issues and foster fruitful exchanges on a long term basis.

The continuous advances and progress made in hardware technology makes it possible to foresee a realm of unprecedented performance levels and new application-driven architectural designs, as evidenced by the recent announcement of a 80-core chip [1]. Nevertheless, the evolution of nanotechnologies raises serious challenges with respect to both dependability and security viewpoints. Issues at stake go far beyond developing protections with respect to accidental disturbances in operation, they also relate to the unreliability and variability that will characterize emerging nanoscale devices. Accounting for malicious threats targeting hardware circuits will constitute another increasing concern.

No abstract available.

This paper presents first the context and motivation for dealing with soft errors. In order to be able to account for the various issues involved, a concerted reflection has been carried out including embedded system integrators, manufacturers, and academic researchers. The authors have summarized the main outcomes of this effort. Finally, the various contributions that are meant to address the several factors that characterize the problem posed by soft errors and provide solutions was introduced to mitigate their effects.

Drivers are becoming the larger part of operating systems kernels. Previous studies have shown that device drivers seem to be one of the most important sources of operating systems misbehavior. Their failure can have significant impact on the kernel and cause significant damages to the system as a whole. To objectively characterize the impact of faulty drivers, we have carried out a series of fault injection experiments. To conduct these experiments we have targeted the DPI (Driver Programming Interface) that implements the way device drivers interact with the kernel. Faults are injected on the parameters of these kernel core Junctions. This allows for the derivation of useful results about the failure modes induced and thus characterization of the robustness of a target kernel with respect to faulty drivers. The information gathered also enables to improve these interaction facilities.

The aim of this paper is to compare the dependability of three operating systems (Windows NT4, Windows 2000 and Windows XP) with respect to erroneous behavior of the application layer. The results show a similar behavior of the three OSs with respect to robustness and a noticeable difference in OS reaction and restart times. They also show that the application state (mainly the hang and abort states) significantly impacts the restart time for the three OSs.

We propose a dependability benchmark suitable for a general purpose operating system (OS). The specifications of the benchmark components are presented and illustrated on a benchmark prototype dedicated to Windows 2000. The important novelty, as regards OS dependability benchmarking, is threefold. First, it lies on a comprehensive and structured set of measures: outcomes are considered both at the OS level and at the application level. Second, these measures include not only robustness measures (e.g., the distribution among the observed outcomes for the OS and the application: error codes, exceptions, workload correct or erroneous completion, OS and application hang), but also the related temporal measures in the presence of faults (e.g., system call and workload execution times, as well as operating system restart time). Finally, we are considering a realistic workload (namely, TPC-C client), instead of a synthetic workload.

This paper presents an approach for building SoftWareImplemented Fault Injection (SWIFI) tools for real-time systems starting from temporal logic specifications. Temporal logic formulas are provided describing the behavior of the main SWIFI mechanisms, both for the injection of faults and for the observation of the target system. Besides conventional fault injectors (i.e., that flip bits in memory and in system calls’ parameters), we have also considered fault injectors based on the notion of saboteur. Concerning the observation mechanisms, we have defined on-line software probes aimed at observing not only the failure modes of the system under test, but also the real-time behavior of both the application and the underlying operating system. The specifications of all such mechanisms are then translated into programs that are executed on-line by a virtual machine. This approach was applied to the development of MAFALDA-RT, a SWIFI tool for real-time systems. A case study exemplifies how the proposed approach allows for the SWIFI mechanisms to be defined and applied.

This paper addresses the issue of characterizing the respective impact of fault injection techniques. Three physical techniques and one software-implemented technique that have been used to assess the fault tolerance features of the MARS fault-tolerant distributed real-time system are compared and analyzed. After a short summary of the fault tolerance features of the MARS architecture and especially of the error detection mechanisms that were used to compare the erroneous behaviors induced by the fault injection techniques considered, we describe the common distributed testbed and test scenario implemented to perform a coherent set of fault injection campaigns. The main features of the four fault injection techniques considered are then briefly described and the results obtained are finally presented and discussed. Emphasis is put on the analysis of the specific impact and merit of each injection technique.

The main goal of the experimental stud), reported in this paper is to investigate to what extent distinct fault injection techniques lead to similar consequences (errors and failures). The target system we are using to carry out our investigation is the Linux kernel as it provides a representative operating system. It is featuring full controllability and observability thanks to its open source status. Three types of software-implemented fault injection techniques are considered, namely: i) provision of invalid values to the parameters of the kernel calls, ii) corruption of the parameters of the kernel calls, and iii) corruption of the input parameters of the internal functions of the kernel. The workload being used for the experiments is tailored to activate selectively each functional component. The observations encompass typical kernel failure modes (e.g., exceptions and kernel hangs) as well as a detailed analysis of the reported error codes.

This paper presents a new prototype tool called MAFALDA-RT (Microkernel Assessment by Fault injection AnaLysis and Design Aid for Real Time systems) that is aimed at the dependability assessment of real-time systems. The tool proposes a novel method to cope with the problem of temporal intrusiveness caused by the use of SWIFI. In addition to typical failure modes (e.g., application hang, system hang, exception, etc), the observation capabilities of MAFALDA-RT extend to temporal measurements characterizing both the executive and application layers (e.g., task processing, task synchronization, context-switch, system calls, etc.). The usefulness of the tool is illustrated by a set of results of a significant case study, based on a real time system consisting of the Chorus microkernel and a mine drainage control application.

The application of fault injection in the context of dependability benchmarking is far from being straightforward. One decisive issue to be addressed is to what extent injected faults are representative of actual faults. This paper proposes an approach to analyze the effects of real and injected faults.

The dependability of CORBA systems is a crucial issue for the development of today's distributed platforms and applications. This paper analyzes various techniques that can be applied to the dependability evaluation of CORBA systems. Due to the complexity of a middleware platform like CORBA and its various types of software components, experiments using several fault injection techniques are required to obtain comprehensive dependability benchmarks. To illustrate one of these techniques, we have applied fault injection at the communication level, targeting requests to major CORBA services, such as naming and events. Experiments have been carried out on a number of off-the-shelf implementations of CORBA. We present and discuss some of the results that we have obtained. They provide objective insights into the system's behaviour in the presence of faults, and are significant inputs for the selection of a candidate for a given application domain.

The commercial offer concerning microkernel technology constitutes an attractive alternative for developing operating systems to suit a wide range of application domains. However, the integration of commercial off-the-shelf (COTS) microkernels into critical embedded computer systems is a problem for system developers, in particular due to the lack of objective data concerning their behavior in the presence of faults. This paper addresses this issue by describing a prototype environment, called MAFALDA (Microkernel Assessment by Fault injection AnaLysis and Design Aid), that is aimed at providing objective failure data on a candidate microkernel and also improving its error detection capabilities. The paper first presents the overall architecture of MAFALDA. Then, a case study carried out on an instance of the Chorus microkemel is used to illustrate the benefits that can be obtained with MAFALDA both from the dependability assessment and design-aid viewpoints. Implementation issues are also addressed that account for the specific API of the target microkemel. Some overall insights and lessons learned, gained during the various studies conducted on both Chorus and another target microkemel (LynxOS), are then depicted and discussed. Finally, we conclude the paper by summarizing the main features of the work presented and by identifying future research.

MAFALDA (Microkernel Assessment by Fault injection Analysis and Design Aid) is a generic tool providing quantitative information on COTS microkernels to support their integration into dependable systems. The main originality of MAFALDA relies on the features provided for both the analysis of the failure modes of the target microkernel and the design of error confinement wrappers. The paper illustrates: (i) how MAFALDA is organized and its user interface, and (ii) how it can be used to carry out fault injection campaigns. Finally, we present the experimental context of campaigns carried out on two commercial microkernels and draw the main lessons learnt.

This paper presents a method based on formal specifications for building robust real-time microkernels. Temporal logic is used to specify the functional and temporal properties of real-time kernels with respect to their main services (e.g., scheduling, time, synchronization, and clock interrupts). As an example of a synchronization mechanism, the specification of the Priority Ceiling Protocol is provided. The objective is to verify kernel properties at runtime in order to improve the internal kernel's detection mechanisms and complement their weaknesses. The core of this paper is a complete description of the temporal logic formulas corresponding to real-time kernel specifications. The formulas developed in this paper are the basis for the implementation of fault containment wrappers. The combination of COTS microkernels and wrappers leads to the notion of robust microkernels. The provided case study illustrates the approach on top of an instance of the Chorus microkernel.

This paper addresses the problem of estimating fault tolerance coverage through statistical processing of observations collected in fault-injection experiments. In an earlier paper, venous estimators based on simple sampling in the complete fault/activity input space and stratified sampling in a partitioned space were studied; frequentist confidence limits were derived based on a normal approximation. In this paper, the validity of this approximation is analyzed. The theory of confidence regions is introduced to estimate coverage without approximation when stratification is used. Three statistics are considered for defining confidence regions. It is shown that one-a vectorial statistic-is often more conservative than the other two. However, only the vectorial statistic is computationally tractable. We then consider Bayesian estimation methods for stratified sampling. Two methods are presented to obtain an approximation of the posterior distribution of the coverage by calculating its moments. The moments are then used to identify the type of the distribution in the Pearson distribution system, to estimate its parameters, and to obtain the coverage confidence limit. Three hypothetical example systems are used to compare the validity and the conservatism of the frequentist and Bayesian estimations.

The development and validation of fault-tolerant computers for critical real-time applications are currently both costly and time consuming. Often, the underlying technology is out-of-date by the time the computers are ready for deployment. Obsolescence can become a chronic problem when the systems in which they are embedded have lifetimes of several decades. This paper gives an overview of the work carried out in a project that is tackling the issues of cost and rapid obsolescence by defining a generic fault-tolerant computer architecture based essentially on commercial off-the-shelf (COTS) components (both processor hardware boards and real-time operating systems). The architecture uses a limited number of specific, but generic, hardware and software components to implement an architecture that can be configured along three dimensions: redundant channels, redundant lanes, and integrity levels. The two dimensions of physical redundancy allow the definition of a wide variety of instances with different fault tolerance strategies. The integrity level dimension allows application components of different levels of criticality to coexist in the same instance. The paper describes the main concepts of the architecture, the supporting environments for development and validation, and the prototypes currently being implemented.

This study addresses the use of fault injection for explicitly removing design/implementation faults in complex fault-tolerance algorithms and mechanisms (FTAM), viz, fault-tolerance deficiency faults. A formalism is introduced to represent the FTAM by a set of assertions. This formalism enables an execution tree to be generated, where each path from the root to a leaf of the tree is a well-defined formula. The set of well-defined formulas constitutes a useful framework that fully characterizes the test sequence. The input patterns of the test sequence (fault and activation domains) then are determined to fewer specific structural criteria over the execution tree (activation of proper sets of paths). This provides a framework for generating a functional deterministic test for programs that implement complex FTAM. This methodology has been used to extend a debugging tool aimed at testing fault tolerance protocols developed by BULL France. It has been applied successfully to the injection of faults in the inter-replica protocol that supports the application-level fault-tolerance features of the architecture of the ESPRIT-funded Delta-4 project. The results of these experiments are analyzed in detail. In particular, even though the target protocol had been independently verified formally, the application of the proposed testing strategy revealed two fault-tolerance deficiency faults.

This paper addresses the problem of estimating the coverage of a fault tolerance mechanism through statistical professing of observations collected in fault injection experiments. A formal definition of coverage is given in terms of the fault and system activity sets that characterize the input space. Two categories of sampling techniques are considered for coverage estimation: sampling in the whole space and sampling in a space partitioned into classes. The estimators for each technique are compared by means of hypothetical examples. Techniques for early estimations of coverage are then studied. These techniques allow unbiased estimations of coverage to be made before all classes of the sampling space have been tested. Then, the "no-reply" problem that hampers most practical fault-injection experiments is discussed and an a posteriori stratification technique is proposed that allows the scope of incomplete tests to be widened by accounting for available structural information about the target system.>

The authors describe a dependability evaluation method based on fault injection that establishes the link between the experimental evaluation of the fault tolerance process and the fault occurrence process. The main characteristics of a fault injection test sequence aimed at evaluating the coverage of the fault tolerance process are presented. Emphasis is given to the derivation of experimental measures. The various steps by which the fault occurrence and fault tolerance processes are combined to evaluate dependability measures are identified and their interactions are analyzed. The method is illustrated by an application to the dependability evaluation of the distributed fault-tolerant architecture of the Esprit Delta-4 Project.>

Dependability modeling and evaluation (encompassing reliability and safety issues) of the two major fault tolerance software approaches-recovery blocks (RBs) and N version programming (NVP)-are presented. The study is based on the detailed analysis of software fault-tolerance architectures able to tolerate a single fault (RB: two alternates and an acceptance test; NVP: three versions and a decider). For each approach a detailed model based on the software production process is established and then simplified by assuming that only a single fault type may manifest during execution of the fault-tolerant software and that no error compensation may take place within the software. The analytical results obtained make it possible to identify the improvement, compared to a non-fault-tolerant software, that could result from the use of RB (the acceptance test has to be more reliable from the alternates) and NVP (related faults among the versions and the decider have to be minimized) and to determine the most critical types of related faults. Nested RBs are studied, showing that the proposed analysis approach can be applied to such realistic software structures and that when an alternate is itself an RB, the results are analogous to the case of the addition of a third alternate. The reliability analysis shows that only a small improvement can be expected.>

The authors address the problem of validating the dependability of fault-tolerant computing systems, in particular, the validation of the fault-tolerance mechanisms. The proposed approach is based on the use of fault injection at the physical level on a hardware/software prototype of the system considered. The place of this approach in a validation-directed design process and with respect to related work on fault injection is clearly identified. The major requirements and problems related to the development and application of a validation methodology based on fault injection are presented and discussed. Emphasis is put on the definition, analysis, and use of the experimental dependability measures that can be obtained. The proposed methodology has been implemented through the realization of a general pin-level fault injection tool (MESSALINE), and its usefulness is demonstrated by the application of MESSALINE to the experimental validation of two systems: a subsystem of a centralized computerized interlocking system for railway control applications and a distributed system corresponding to the current implementation of the dependable communication system of the ESPRIT Delta-4 Project.>