Many industrial systems have specific requirements derived from the applications they execute. Specifically, the interaction of a distributed embedded control system (DECS) with the real-world imposes strict real-time (RT) and reliability requirements. For a system to be RT, it has to produce a proper result in a bounded time. On top of that, for a system to be reliable, it has to operate continuously during its mission time, and in cases in which very high reliability is needed, fault tolerance (FT) techniques are used. Moreover, these systems are often deployed in dynamic environments where the operational conditions may change in an unpredictable manner. Therefore, there is an increasing interest in creating DECSs that are capable of modifying their behavior autonomously and dynamically in response to unexpectedly changing requirements or conditions. In recent years, there is a growing trend toward using Ethernet as the network technology for DECSs. Unfortunately, the original specification of this technology lacks appropriate services to fulfill the most demanding requirements of industrial systems. In this regard, many Ethernet-based protocols and standards have been proposed along the past years to deal with these limitations. In this paper, we survey solutions that have been proposed to achieve FT in Ethernet-based DECSs, considering faults both in their nodes and communication subsystem. In addition, we discuss adaptive FT techniques that can be used to increase the flexibility of adaptive DECS. Finally, we identify future trends and open challenges to build highly reliable DECS in the future.

Distributed embedded systems (DESs) that perform critical tasks in unpredictable environments must be reliable, hard real-time, and adaptive. Since a DES comprises nodes that rely on a network, the network must provide adequate support: it must be reliable, convey messages on time, and meet new real-time requirements as the nodes adapt. Ethernet is ill-suited for such hard real-time adaptive systems, but it can be made suitable. The flexible time-triggered (FTT) paradigm already supports hard real-time message exchanges and the necessary flexibility to meet evolving hard real-time requirements, but its Ethernet implementations had reliability limitations. To address these, we designed FTT replicated star for Ethernet (FTTRS), a communication subsystem that tolerates permanent and transient faults, even if they occur simultaneously, while keeping the paradigm's key features: support for both the timely exchange of periodic and sporadic real-time messages, and support for updating the real-time parameters of these messages at runtime. In this paper, we present FTTRS, the first Ethernet-based communication subsystem specifically designed for highly reliable hard real-time adaptive DESs.

Distributed embedded systems typically have real-time and dependability requirements. Moreover, they must also be flexible to changing conditions when they are deployed in dynamic environments. The FT4FTT project aims at providing a switched Ethernet architecture that can support distributed control applications that are predictable, highly-reliable and adaptive. FT4FTT relies on the Flexible Time-Triggered Replicated Star for Ethernet (FTTRS) to tolerate channel faults. Moreover, nodes' hardware faults are tolerated by means of active node replication with majority voting. In order to coordinately trigger the execution of the tasks in the replicas, we designed the CD4NR mechanism, in which the network assists in deciding what to execute and when. This paper presents the first implementation of the CD4NR mechanism on a real prototype of FTTRS and the first testing of the complete system. For this we developed an experimental setup, based on the hardware-in-the-loop technique, running a real-time control application.

Distributed embedded systems (DES) have traditionally been designed assuming that the requirements they need to satisfy are known in advance. If this is not the case, and a DES should operate autonomously without interruption, it needs to be adaptive. For this, flexible approaches are necessary and this applies in particular to the network of the DES. However, if the probability of faults occurring is non-negligible, then flexibility alone is not enough and fault tolerance is also necessary. The Flexible Time-Triggered Replicated Star for Ethernet (FTTRS) is a set of protocols and mechanisms together with a specific network topology that builds on a switched-Ethernet implementation of the Flexible Time-Triggered (FTT) communication paradigm by enhancing it to not only provide flexibility, but also fault tolerance. This paper describes our efforts towards a layered architecture for FTTRS to benefit from the well-known advantages of these architectures, such as making the complexity manageable and easier to communicate, and making the design more future proof by allowing changes in one layer without affecting other layers.

A distributed embedded system (DES) is made up of a set of computing nodes interconnected by a network. If we want the DES to continue to operate even if a subset of its network elements fail, the network must be fault-tolerant. In particular, this requires that the architecture of the network provides redundant paths between nodes and that any elements critical for the operation of the network are replicated. In the context of DES that must not only be highly reliable, but also provide sufficient flexibility to adapt to unpredictable requirement changes, the Flexible Time-Triggered Replicated Star for Ethernet (FTTRS) has been proposed. One of the core features of FTTRS is precisely its fault-tolerant network architecture. In this paper we present a proof-of-concept prototype of FTTRS and a series of fault-injection experiments. These experiments show that FTTRS can tolerate the crash of any single network element, as well as the crash of various combinations of multiple network elements. A variety of omission failures affecting the most critical FTTRS message (called the trigger message) are also tolerated.

For a distributed embedded system (DES) to operate continuously in a dynamic environment, it must be flexible and highly reliable. This applies in particular to its communication subsystem. The Flexible Time-Triggered Replicated Star for Ethernet (FTTRS) aims at providing such a subsystem by means of a highly-reliable switched-Ethernet architecture based on the Flexible Time-Triggered paradigm (FTT), a master/slave communication paradigm where the master periodically polls the slaves using so-called trigger messages (TMs). In particular, FTTRS interconnects nodes by redundant communication paths provided by two switches, each embedding an FTT master that manages the communication. This allows FTTRS to tolerate the failure of one switch without interrupting the communication as long as the masters are replica determinate, i.e., provide identical service to the slaves. The master replica determinism entails the masters broadcasting their TMs in a lockstep fashion: when one master broadcasts a TM, the other should do the same quasi-simultaneously. In this paper we present a solution inspired by the Precision Time Protocol (PTP) for achieving this lockstep transmission and preliminary results showing the precision with which we can synchronize the masters on a software prototype.

The communication subsystem of distributed embedded systems (DES) that must operate continuously and satisfy unpredictable requirement changes must be reliable and flexible. Recently the Flexible Time-Triggered Replicated Star for Ethernet (FTTRS) has been proposed as a communication subsystem that satisfies these two attributes. It is based on the master/multi-slave Flexible-Time Triggered (FTT) communication paradigm and relies on two custom switches, each with its own embedded FTT master. Both masters are active simultaneously and provide the same service. Specifically, they simultaneously and periodically broadcast so-called trigger messages (TMs) in a redundant manner to make them robust to transient channel faults. One of the functions of these TMs is to divide the communication time into rounds called elementary cycles (ECs). For the correct operation of FTTRS, it is important that all slaves agree when each EC starts and ends. A mechanism to achieve this has been recently proposed. This paper presents a first implementation of this mechanism and a series of experimental tests that constitute a first step towards building a prototype of an FTTRS network.

Fault Tolerance for Flexible Time-Triggered Ethernet-based systems (FT4FTT) is a project to devise an architecture for distributed embedded systems that provides both flexibility to changing real-time requirements and high reliability through fault tolerance. One of the key parts of such an architecture is the communication subsystem. When designing such a subsystem many decisions have to be made. To understand how such decisions impact the reliability of the final design, in this paper we present a framework to evaluate the reliability of a large number of potential designs. The approach is based on storing a finite subset of the design space for the communication subsystem of FT4FTT in an undirected graph and then generating a continuous-time Markov chain from the graph to evaluate the reliability of each design belonging to the subset.

Flexible Time-Triggered communication (FTT) allows a distributed embedded system (DES) to adapt to changing real-time requirements at runtime. This facilitates the continuous operation of the DES under dynamic environments that change over time. However, for continuous operation, high reliability in the nodes of the DES is also crucial. This can be achieved using node replication, as long as failure independence between replicas is ensured, which calls for preventing the propagation of errors. Our goal is to prevent the propagation of Byzantine node behaviours and to ensure that local errors in the channel cannot disturb the global communication. For this, we construct the HaRTES/PG switch, a new switch based on the HaRTES implementation of FTT for Ethernet. This paper presents as a first step a study of the possible errors that may lead to Byzantine node behaviours and a global communication disturbance in HaRTES, as well as some ideas on how to prevent the propagation of these errors in HaRTES/PG.

Distributed embedded systems (DES) have been traditionally designed to operate in static environments that do not change over time. Flexible designs are increasingly being introduced to achieve continuous and correct operation under dynamic environments. Some designs, such as the Flexible Time-Triggered communication paradigm (FTT), are focused on being able to modify the real-time operation upon changing requirements imposed by the environment. The on-going project Fault Tolerance for FTT (FT4FTT) purports to increase the reliability of a DES based on the FTT protocol by introducing static fault tolerance. In this paper we give some hints on how to go one step beyond by adding dynamic fault tolerance to the DES. This would result in new systems that would combine the qualities of flexible real-time operation and flexible and adaptive fault tolerance, much enlarging their sphere of applicability.

Distributed embedded systems have traditionally been designed using static approaches, i.e., assuming a static environment. Such approaches, however, cannot guarantee continuous operation under dynamic environments that impose new requirements upon a system as time passes. As a solution, flexible approaches have been proposed. One such approach that allows a system to adapt to changing real-time requirements is the Flexible Time-Triggered (FTT) communication paradigm. Nevertheless, if continuous operation under dynamic environments is desired, then flexibility is not enough. Indeed, it is also crucial for the system to be sufficiently reliable. In this paper we therefore explore some design ideas to make FTT highly reliable through fault tolerance by using replication. As a starting point we will use the switch of the Hard Real-Time Ethernet Switching (HaRTES) implementation of FTT.

Some of the severe dependability limitations of Controller Area Network (CAN) can be overcome by replacing its bus topology with a star topology. Thus, a replicated star topology with advanced error-containment and fault-tolerance mechanisms for CAN, called ReCANcentrate, has been proposed. Its two hubs are coupled with each other and create a single logical broadcast domain. This allows each node to easily manage the replicated star by means of a software driver, called reCANdrv, that abstracts away the details of this replication. The goal of reCANdrv is to manage the star's media redundancy transparently for a CAN application, allowing it to exchange information through the star while tolerating faults. This paper describes the design of reCANdrv, the specification as properties of reCANdrv's correct redundancy management, and the verification of these properties by means of model checking.

Highly dependable distributed embedded systems (DES) have traditionally been developed using static approaches, i.e., assuming a mostly constant environment. However, the little flexibility of such approaches does not allow continuous operation under dynamic environments. The Flexible Time-Triggered (FTT) communication paradigm is a promising approach to introduce the required flexibility. However, for continuous operation reliability is also crucial. Replicated star topologies are particularly well-suited to provide an increased reliability. Nevertheless, for FTT-CAN, the implementation of FTT for CAN, no replicated star topology that takes advantage of FTT-CAN's features to increase reliability and error containment exists. This paper discusses important design questions that need to be solved to create such a novel solution.

Despite the significant advantages of the Controller Area Network (CAN) there is an extended belief that CAN is not suitable for critical applications, mainly because of several dependability limitations. During the CANbids project each one of these limitations has been addressed and a complete architecture for CAN-based fault-tolerant systems has been devised. This architecture allows building highly-reliable systems. This paper describes the design of such an architecture and the prototyping of its fundamental parts.

There is a growing interest in making the CAN field-bus more suitable for dependable applications. In the past years, several dependability limitations of CAN have already been addressed and a significant number of solutions are available. Nevertheless, the integration of these solutions into a single communication infrastructure is still an open issue. In this paper we discuss the integration of two specific solutions: FTT-CAN and ReCANcentrate. FTT-CAN is a higher-layer protocol that guarantees flexible real-time scheduling of CAN messages; whereas ReCANcentrate is a duplicated star topology for CAN that includes several enhanced mechanisms for media fault tolerance. We show how they are integrated into a single architecture that preserves the properties of each solution.

This paper presents the design and a preliminary implementation of sfiCAN: a physical fault injector for the CAN field-bus that allows the creation of a great variety of complex fault scenarios. The fault injector replaces the CAN bus topology with a star topology, whose central element is a hub with fault injection mechanisms. The fault injector is easily configured, with great flexibility, from a PC connected to a dedicated port of the hub. For this it uses a fault-injection specification, which is translated to a configuration protocol on top of CAN. This protocol is only used in-between fault injection tests and therefore does not interfere with the execution of any test. The purpose of the fault injector is to test the behavior of the nodes of a CAN network in the presence of channel errors, in particular, of the nodes' CAN controllers and the software executing on them, for which the star topology is transparent.

Although the use of star topologies to improve dependability in field-buses is gaining in importance, as in TTP/C and FlexRay, a mature technology such as the Controller Area Network (CAN) remained essentially a bus-only network. Thus, we proposed a CAN-compliant replicated star topology called ReCANcentrate, which has advanced error-containment and fault-tolerance mechanisms. Its two hubs are coupled with each other and create a single logical broadcast domain that allowed us to propose, in a previous work, a strategy for each node to easily manage the replicated star by means of a software driver that abstracts away the details of the replication. This paper describes the main functionalities of this driver, as well as the first tests we have conducted, on a real ReCANcentrate prototype, to verify the correctness and the performance of the driver in the absence and in the presence of faults.

Star topologies are rising the interest of newer field-bus communication technologies like TTP/C and FlexRay, given the dependability advantages stars can offer. However, it is also possible to take advantage of a mature technology such as Controller Area Network (CAN), while benefiting from stars. For that, we developed a CAN-compliant replicated star called ReCANcentrate. It includes two hubs that are coupled with each other, thereby forcing a single broadcast domain that allowed us to define, in a previous work, a strategy for each node to easily manage the replicated star. To demonstrate the feasibility of this management, this paper presents its on-going implementation as a driver to be executed at each node.