Many industrial systems have specific requirements derived from the applications they execute. Specifically, the interaction of a distributed embedded control system (DECS) with the real-world imposes strict real-time (RT) and reliability requirements. For a system to be RT, it has to produce a proper result in a bounded time. On top of that, for a system to be reliable, it has to operate continuously during its mission time, and in cases in which very high reliability is needed, fault tolerance (FT) techniques are used. Moreover, these systems are often deployed in dynamic environments where the operational conditions may change in an unpredictable manner. Therefore, there is an increasing interest in creating DECSs that are capable of modifying their behavior autonomously and dynamically in response to unexpectedly changing requirements or conditions. In recent years, there is a growing trend toward using Ethernet as the network technology for DECSs. Unfortunately, the original specification of this technology lacks appropriate services to fulfill the most demanding requirements of industrial systems. In this regard, many Ethernet-based protocols and standards have been proposed along the past years to deal with these limitations. In this paper, we survey solutions that have been proposed to achieve FT in Ethernet-based DECSs, considering faults both in their nodes and communication subsystem. In addition, we discuss adaptive FT techniques that can be used to increase the flexibility of adaptive DECS. Finally, we identify future trends and open challenges to build highly reliable DECS in the future.

Distributed Embedded Control Systems (DECSs) used for critical applications must usually abide by strict real-time and dependability requirements. Correspondingly, the FT4FTT project proposes a complete fault-tolerant (FT) architecture for RT DECSs. The Flexible Time-Triggered Ethernet (FTT-Ethernet) communication protocol fulfills the RT requirements, while the FT mechanisms added on top of it, which are based on channel duplication and active replication of nodes, provide the FT behaviour. Temporary faults affecting the channel or the nodes, which are the most probable type of faults in DESs, can manifest in such a way that a node replica loses its coordination with the others and, thereby, it also loses its communication and/or computation capability from then on, leading to attrition of the redundancy initially provided by the active replication of nodes. This paper describes the implementation and test of specific mechanisms that are devised to determine which replicas are temporarily faulty and to promptly reintegrate them.

Distributed embedded systems typically have real-time and dependability requirements. Moreover, they must also be flexible to changing conditions when they are deployed in dynamic environments. The FT4FTT project aims at providing a switched Ethernet architecture that can support distributed control applications that are predictable, highly-reliable and adaptive. FT4FTT relies on the Flexible Time-Triggered Replicated Star for Ethernet (FTTRS) to tolerate channel faults. Moreover, nodes' hardware faults are tolerated by means of active node replication with majority voting. In order to coordinately trigger the execution of the tasks in the replicas, we designed the CD4NR mechanism, in which the network assists in deciding what to execute and when. This paper presents the first implementation of the CD4NR mechanism on a real prototype of FTTRS and the first testing of the complete system. For this we developed an experimental setup, based on the hardware-in-the-loop technique, running a real-time control application.

Distributed Embedded Control Systems (DECSs) used for Real-Time (RT) critical applications must satisfy stringent time requirements and attain high reliability. FTT-Ethernet provides nodes of DECSs with real-time communication capabilities, but does not include Fault Tolerance (FT) mechanisms. The FT4FTT project aims at proposing a complete FT architecture for RT critical DECSs. It uses a duplicated switched FTT-Ethernet star and active node replication with consistent distributed majority voting to respectively tolerate channel and node faults. However, FT4FTT, in its current state, still lacks mechanisms to prevent node redundancy attrition due to temporary faults affecting the nodes and channel, which are the most likely types of faults in DESs. This paper presents our ongoing work to complete the FT4FTT architecture with appropriate fault-diagnosis and reintegration mechanisms that overcome this limitation.

Distributed embedded systems (DESs) that operate in dynamic environments require emerging flexibility and adaptivity communication requirements. When those DESs are deployed for critical applications, they must also employ appropriate fault-tolerance (FT) mechanisms to attain a high level of reliability. The FTT-Ethernet communication protocol supports the flexibility needed in dynamic environments, but does not provide adequate fault tolerance. In order to overcome this limitation the ongoing FT4FTT project proposes a communication architecture that includes fault-tolerance capabilities at different levels of DESs relying on FTT-Ethernet. In particular, it provides communication and execution mechanisms to tolerate node failures by means of active node replication with majority voting. This paper builds upon a previous OMNET++ model of an FTT-Ethernet-based DES in order to add, simulate and assess those mechanisms. Specifically, it models the communication mechanisms envisaged to enforce replica determinism in the voting procedure, as well as to trigger and coordinate the tasks executed in the replicas.

Distributed Embedded Systems (DESs) typically have dependability and real-time requirements. Moreover, when they are deployed in dynamic environments, they must be flexible enough to adapt to changes in the operation requirements. The Fault Tolerance for Flexible Time-Triggered Ethernet (FT4FTT) project aims at providing a Switched-Ethernet architecture, based on the Flexible Time-Triggered communication paradigm (FTT), that is flexible and highly reliable. In particular, FT4FTT provides node fault-tolerance by means of active replication with majority voting. In this sense, FT4FTT includes the Consistent Replicated Voting (CRV) protocol to enforce replica determinism, even in presence of faults, while maximizing the reliability that can be achieved thanks to the node redundancy and the communication subsystem itself. This papers presents a first implementation of this protocol in a real prototype, and shows the on-going experimental evaluation been carried out to asses its correctness.

In the context of critical applications there is an increasing interest in having Distributed Embedded Systems (DESs) that are able to operate in dynamic environments, while at the same time reaching a high reliability. The Flexible Time-Triggered communication paradigm (FTT) is designed to support the QoS and real-time requirements of the traffic of these systems. However, FTT does not provide fault tolerance. This paper explains our on-going work towards designing a consistent and highly-reliable voting protocol which supports node replication on DESs that use FTT switched Ethernet. In particular, we propose a protocol for the node replicas to vote consistently on messages exchanged through an FTT Ethernet network that uses time redundancy, while trying to maximize the reliability that can be achieved thanks to the redundancy of the nodes and the communication subsystem itself.

The Flexible Time Triggered (FTT) paradigm provides online flexible scheduling for distributed embedded systems but it does not present adequate fault tolerance mechanisms so as to reach a very high reliability. Adding the adequate fault tolerance mechanisms to FTT-based architectures would open room for adaptive yet highly dependable systems. In this work we present a fault-tolerant system architecture for control applications that adds a node replication scheme with voting on top of an FTT-based system. Using a previously proposed network-centric approach we show how to coordinate the execution of the different phases for a typical control application in our system architecture, i.e. we show how to trigger the execution of tasks in node replicas and the transmission of messages in the communication channel, using the underlying FTT protocol. At the end, we demonstrate how to apply this idea of coordinated dispatching to one concrete control application, ball-on-plate.

Distributed embedded systems (DES) have been traditionally designed to operate in static environments that do not change over time. Flexible designs are increasingly being introduced to achieve continuous and correct operation under dynamic environments. Some designs, such as the Flexible Time-Triggered communication paradigm (FTT), are focused on being able to modify the real-time operation upon changing requirements imposed by the environment. The on-going project Fault Tolerance for FTT (FT4FTT) purports to increase the reliability of a DES based on the FTT protocol by introducing static fault tolerance. In this paper we give some hints on how to go one step beyond by adding dynamic fault tolerance to the DES. This would result in new systems that would combine the qualities of flexible real-time operation and flexible and adaptive fault tolerance, much enlarging their sphere of applicability.