Wired real-time network deployments based on time-sensitive networking (TSN) are becoming increasingly popular in several application fields including industrial automation and the automotive area. This is to a great extent due to the scheduled traffic (ST; IEEE 802.1Qbv), a TSN mechanism that precisely triggers transmissions at pre-defined instants from a set of independent hardware queues. Unfortunately, the wireless counterparts based on IEEE 802.11 (Wi-Fi) currently lack standardized or functional solutions that would be able to fulfill strict real-time requirements. This work proposes the use of ST over IEEE 802.11, taking advantage of an existing but overlooked mechanism from commercial off-the-shelf (COTS) equipment. An extensive set of tests, spanning different implementation configurations, traffic patterns, and levels of interference evaluates the proposal in terms of reliability and delivery delays and serves as a proof of concept. The tests also evaluate the much-needed clock synchronization quality of an out-of-the-box software-only implementation of the precision time protocol (PTP) due to its relevance in enabling ST.

Wired networks are deployed in scenarios requiring the highest levels of performance in terms of reliability and timeliness. Unfortunately, broken wires might permanently compromise the network operation unless fault tolerance mechanisms are in place. Fault tolerance is commonly achieved by replicating the wired network components, but this paper examines the use of a wireless backup network, since the wireless physical layer (PHY) is not expected to display permanent failures due to broken wires. Two mechanisms at medium access control (MAC) level are presented to take advantage of the wireless backup network: one allocating redundancy statically and one dynamically. Without loss of generality, redundancy is applied using the standard mechanisms from IEEE 802.3 (Ethernet) and IEEE 802.11 (Wi-Fi). The performance increase added by the backup network is studied both analytically and by simulation, showing considerable improvements in a very compromised midsize wired network.

Automated vehicles come with promises for higher comfort and safety compared to standard human-driven vehicles. Various demonstrator vehicles with fully automated driving capabilities have been already presented with success. Yet, there is a large number of technical challenges to be solved until the safety levels comply with those required from safety standards, and most importantly with those for public acceptance. In this paper, we introduce the technical challenges resulting from the need for fault-tolerant capabilities of automated vehicles with no fallback-ready drivers. We then propose a concrete solution to these challenges. This includes a fault-tolerant architecture for automated driving systems. Furthermore we introduce, the safety co-pilot, that is a safety mechanism that ensures the coordinated operation of two or more redundant automated driving systems, by means of novel application-specific verification methods. We conclude our work with experimental proof of concept results of the proposed solution.

Networks are a core element of many industrial and automation systems at present. Often these networks transport time- and safety-critical messages that control physical processes. Thus, timely and guaranteed delivery is an essential property of networks for such critical systems. Over the past decade, a variety of network solutions have evolved to satisfy said properties. However, these solutions are largely incompatible with each other and many system architects are forced to deploy different solutions in parallel due to their different capabilities. IEEE 802.1 Time-Sensitive Networking (TSN) is a standardization group that enhances IEEE networking standards, most prominently Ethernet-based networks, with said properties and has the unique potential to evolve as a cross-industry mainstream networking technology. In this survey paper, we give an overview of TSN in industrial communication and automation systems and discuss specific TSN standards and projects in detail as well as their applicability to various industries.

Time-Sensitive Networks are becoming the standard in real-time networking over IEEE 802 Ethernet. The set of TSN standards currently being defined by the IEEE TSN task group offer mechanisms introducing real-time behavior over specialized hardware network nodes. Out of these standards we identify those enabling the integration of deterministic real-time communication with the legacy best-effort traffic. In this demonstration we show the management of a simple TSN hardware setup via the Slate XNS tool-chain. We demonstrate the ease-of-use of the tools in charge of the entire network as well as the provided scheduling capabilities allowing one-click deployment of deterministic communications with properly configured IEEE 802.1Qbv and IEEE 802.1ASrev sub-standards.

Time Sensitive Networks (TSN) emerge as the set of sub-standards incorporating real-time support as an extension of standard Ethernet. In particular, IEEE 802.1Qbv defines a time-triggered communication paradigm with the addition of a time-aware shaper governing the selection of frames at the egress queues according to a predefined schedule, encoded in so-called Gate Control Lists (GCL). Nonetheless, the design of compositional systems with real-time demands requires a proper configuration of these mechanisms to truly achieve the temporal isolation of communication streams with end-to-end timeliness guarantees. In this paper we address how the synthesis of communication schedules for GCLs defined in IEEE 802.1Qbv can be formalized as a system of constraints expressed via first-order theory of arrays (T_A). We formulate the necessary constraints showing the suitability of the theory of arrays and discuss optimization opportunities arising from the underlying scheduling problem. Our evaluation using general-purpose SMT/OMT solvers proves the validity of the approach, scaling well for small-to medium-networks, and exposing trade-offs for the time needed to synthesize a schedule. Furthermore, we conduct a comparison against previous work and conclude the appropriateness of the method as the basis for future TSN scheduling tools.

Configuration processes of real-time networks are costly both in terms of time and engineering effort and require the system to be shutdown during the reconfiguration phase thus resulting in significant down time as well. The convergence of IT/OT technologies is bringing a whole world of possibilities for the configuration and management of real-time networks for the automation industry. With software defined networking (SDN) features like the separation of the data and control plane and standards like IEEE 802.1 developed with the goal of adding deterministic capabilities to traditionally dynamic switched Ethernet networks, the plug and play paradigm is almost around the corner. In this paper, we go one step further and start looking into the self-configuration of real-time networks. To achieve that we propose to introduce a Configuration Agent in the network, an entity that continuously monitors the network to detect changes and automatically update the configuration to adapt to such changes while maintaining desired quality of service. We present here the complete framework for the Configuration Agent that will provide self-configuration capabilities to real-time networks. The proposed framework has IEEE 802.1 as its core, but also shows how the set of standards need to be extended in order to achieve the self-configuration requirements. Concretely we examine the role of existing communication protocols like NETCONF and OPC-UA and propose the essential ingredients (managed objects) for a YANG model for the learning aspects in the bridges, including different working modes.

Wireless communications enables introduction of Internet of Things (IoT) in industrial networks. Unfortunately, real-time guarantees required for many IoT applications, may be compromised in wireless networks due to an unreliable transmission medium. A key component in enabling real-time communications is the medium access control (MAC) layer and its ability to effectively avoid concurrent transmissions that causes deadline misses. Also, deploying the network in a harsh interference environment can lead to low reliability. Time diversity, based on transmitting several copies of the same data at different instants, increases reliability but at the expense of increased jitter and bandwidth. A more efficient resource utilization is expected from cognitive radio, which dynamically takes into account the status of the wireless environment before performing transmissions. This paper proposes a wireless MAC protocol based on scheduled timeslots to avoid concurrent transmissions, combined with two different mechanisms to increase reliability, one based on time diversity and another on cognitive radio. The protocol and its mechanisms to enhance reliability are compared in different interference scenarios, and show that cognitive radios achieves better performance than time diversity, especially when the interference is produced by a jammer.

Over the last decades the field of dependable computer systems has gained tremendous significance in our modern society. We rely on the dependability of automobiles, railways, airplanes, medical devices, critical infrastructures, like the electrical grid or industrial production facilities, and many more. These dependable systems frequently implement non-trivial mechanisms, for example, to coordinate between redundant components, and a guarantee of correctness of these mechanisms is therefore crucial to avoid catastrophic incidents. Consequently, formal methods are frequently used in industrial dependable system design and in this talk I will discuss the various aspects in which formal methods are and have been deployed for specification, verification, and configuration at TTTech for critical networking products.

Industry 4.0 and Industrial Internet of Things projects work towards adoption of standard IT technologies for real-time control networks in industrial automation. For this the IEEE 802.1 Time-Sensitive Networking (TSN) Task Group has developed and continues to develop a set of standards. One of these standards is the IEEE 802.1AS clock synchronization protocol. IEEE 802.1AS can be used to enable time-triggered communication as well as to coordinate distributed actions in industrial networks. In this paper we study the synchronization quality of IEEE 802.1AS and we are interested in whether the clocks can be synchronized with sufficiently low error such that the protocol can be used for demanding industrial automation applications. In particular, we study the protocol behavior in large-scale networks while considering critical implementation details. We report analytical worst-case results as well as probabilistic results based on simulations, that show that implementation details such as the PHY jitter and the clock granularity have a big impact on the time synchronization precision.

Supporting real-time applications over wireless networks is challenging for several reasons. The medium access control (MAC) layer has a key role in enabling real-time guarantees by providing bounded access time to the transmission medium. Unfortunately, even when timely access is granted, transmissions can still be compromised, specially in the wireless domain, due to the unreliable transmission medium. A common way to increase the reliability of a communication system is to apply redundancy in the form of time diversity, i.e., transmitting several copies of the same message at different points in time. In this paper we propose a wireless MAC protocol capable of handling traffic with different levels of criticality and where transmissions and retransmissions are tailored to deadlines, so that the reliability of the communication system is increased and real- time guarantees can be provided. We focus on the traffic scheduling problem, and extend an already existing solution, Time-Triggered Ethernet, suitable for real-time traffic over wired multi- hop networks. The extension enables transmission of heterogeneous traffic over hybrid wired- wireless networks, and provides timely retransmissions in the wireless segment, such that the real-time guarantees are met. Further, we are able to schedule traffic to better cope with interference patterns of different duration, frequency and level of persistence. The results show that the MAC protocol can be successfully applied to combat different kinds of interference while meeting real-time deadlines.

Ethernet-based networks have found their way into industrial communication more than a decade ago. However, while industry and academia developed Ethernet variants to also meet real-time and fault-tolerant requirements, recent standardization efforts within the IEEE 802 will broadly bring standard IT switched Ethernet in future industrial communication networks. As first standards of IEEE 802.1 time-sensitive networking (TSN) are becoming published at the time of this writing, we review these standards and formulate further research challenges that still go beyond current standard developments. Furthermore, we report on recent research results from the RetNet project that target these research challenges.

The communication in-between embedded systems present in cars and planes, requires real-time networks. Up to now, fieldbus technologies like PROFIBUS and CAN have covered the demand for predictable communications in embedded systems. However, these fieldbuses do not suit some of the emerging application domains, that need more flexibility, support for dynamic traffic flows, different traffic classes, high throughput, and the inclusion of wireless capabilities. To this end, we propose several different medium access control (MAC) schemes with support for traffic with diverse time and safety requirements. We have calculated the worst case channel access delay for each proposal, and also simulated them in OMNeT++ to analyse and compare their performance in terms of average access delay and packet collisions as a function of different protocol settings and traffic patterns e.g., the channel load, data traffic emerging from one sender only versus evenly distributed between all senders. Our results indicate that the more that is known about the data traffic, the better performance can be achieved by selecting an appropriate MAC protocol. Conversely, when nothing is known, one MAC protocol emerges as the best trade-off.

Time-triggered offline scheduling is a cost-efficient way to guarantee low communication end-to-end latency and minimal jitter for communication networks in real-time systems. The schedule is generated pre-runtime and indicates the transmission times of time-triggered frames such that contention is prevented. The synthesis of such offline schedules is a bin-packing problem, known to be NP-complete, with complexity driven by the constraints on frame transmissions, and the number of frames in the schedule. Satisfiability Modulo Theories combined with segmented approaches have been successfully used for synthesizing schedules of large networks. However, such synthesis did not take into account frames periods that are much shorter than the time to execute the schedule cycle. This paper presents a period-aware segmented approach that takes into account the frame periods in order to allocate various instances of a frame within a single cycle. We describe three different synthesis strategies and evaluate them with different synthetic experiments. The results show better performance for one of the strategies, which can synthesize schedules of large networks with high communication loads in less than one hour. We also report how the synthesis time and the schedule quality can change with different parameter configurations.

Real-time networks have tight communication latency and minimal jitter requirements. One way to ensure these requirements is the implementation of a static schedule, which defines the transmission points in time of time-triggered frames. Synthesizing such static schedules is known to be an NP-complete problem where the complexity is driven by the large number of constraints imposed by the network. Satisfiabily Modulo Theories (SMT) have been proven powerful tools to synthesize schedules of medium-to-large industrial networks. However, the schedules of new extremely large networks, such as integrated multi-machine factory networks, are defined by an extremely large number of constraints exceeding the capabilities of being synthesized by the tool alone. This paper presents a decomposition approach that will allow us to improve to synthesize schedules with up to two orders of magnitude in terms of the number of constraints that can be handled. We also present an implementation of a dependency tree on top of the decomposition approach to address application-imposed constraints between frames.

An important aspect of any communication technology is its medium access method, being responsible for sharing the medium among users. For delay-sensitive applications, such as industrial control systems, timely and reliable channel access is of essence. Hence, existing protocols like HART and TTEthernet use a time-triggered medium access approach. However, as the nature of industrial control systems change and evolve, there is a growing need to provide wireless access and support data traffic with mixed requirements. While technologies like WirelessHART can offer timely and reliable access to the wireless medium, only one type of data traffic is usually supported. In this paper, we therefore propose and evaluate three different medium access methods for wireless communications, all capable of supporting three different data traffic classes: time-triggered, rate-constrained and best-effort traffic. In particular, different options on how to handle best-effort traffic, using scheduled time-slots or contention, are evaluated, showing for all the proposals different drawbacks and benefits depending on additional requirements on e.g., hardware, protocol overhead and resistance to interference.

Ethernet is a low-cost communication solution offering high transmission speeds. Although its applications extend beyond computer networking, Ethernet is not suitable for real-time and safety-critical systems. To alleviate this, several real-time Ethernet-based communication protocols have been proposed, such as TTEthernet, which is the focus of this paper. TTEthernet is suitable for mixed-criticality systems both in the safety and temporal domain. TTEthernet offers three traffic classes: static time-triggered (TT) traffic, dynamic traffic with bounded transmission rate (called "Rate Constrained", RC), and unbounded dynamic traffic ("Best-Effort", BE). In this paper we propose a novel worst-case end-to-end delay analysis of the RC traffic for TTEthernet systems. The proposed technique considerably reduces the pessimism of the analysis, compared to existing approaches. We have evaluated the new analysis using several test cases.

Distributed cyber-physical systems are growing in size and functionality and deterministic communication is an important requirement for those systems. The existing solutions based on the time-triggered paradigm pose certain limitations in regards to the configuration. Usually configuration is seen as a one-time event during the installation of the network. Future realtime networks need to be able to adapt more easily to changes in the network. Thus, the configuration becomes an ongoing service, e.g., as for network maintenance and re-configuration to add and remove new, respectively old, equipment. We postulate that configuration will emerge to a continued service that accompanies a real-time network throughout its different life-cycle phases. In this context of evolving and dynamic networks, we introduce the concept of a configuration agent for real-time networks and demonstrate the concept by a realization based on the time triggered paradigm.

TTEthernet is a general purpose communication infrastructure for applications with real-time and/or fault-tolerance requirements. It is based on Ethernet as standardized by the IEEE and allows the integration of mixed-criticality applications in a single physical network. However, currently TTEthernet only operates in wired network settings. With growing industrial demand of the design freedom that comes with wireless communication solutions we are interested in extending TTEthernet to a wireless communication paradigm. In this paper we review the state of the art in deterministic wireless communication approaches. We deduce quality criteria for wireless networks from industrial use cases and outline candidates for a wireless complement to wired TTEthernet.

We present a methodology for the probabilistic modeling of failure dependencies in large, complex systems using Markov Logic Networks (MLNs), a state-of-the-art probabilistic relational modeling technique in machine learning. We illustrate this modeling methodology on example system architectures, and show how the the Probabilistic Consistency Engine (PCE) tool can create and analyze failure-dependency models. We compare MLN-based analysis with analytical symbolic analysis to validate our approach. The latter method yields bounds on the expected system behaviors for different component-failure probabilities, but it requires closed-form representations and is therefore often an impractical approach for complex system analysis. The MLN-based method facilitates techniques of early design analysis for reliability (e.g., probabilistic sensitivity analysis). We analyze two examples - a portion of the Time-Triggered Ethernet (TTEthernet) communication platform used in space, and an architecture based on Honeywell's Cabin Air Compressor(CAC) - that highlight the value of the MLN-based approach for analyzing failure dependencies in complex cyber-physical systems.

Fault-tolerant clock synchronization is the foundation of synchronous architectures such as the Time-Triggered Architecture (TTA) for dependable cyber-physical systems. Clocks are typically local counters that are increased with a given rate according to real time, and clock synchronization algorithms ensure that any two clocks in the system read about the same value at about the same point in real time. This is achieved by a clock synchronization algorithm that changes the current values of the clocks, the clocks' rate, or both. This paper presents a diagnosis algorithm and a clock-rate correction algorithm as layered services on top of the TTEthernet clock synchronization algorithm, which itself is a clock-state correction algorithm. We analyze the algorithms' properties and explore and understand their behavior using a bounded model checker for infinite data types. We use our formal framework for both simulation and formal proof. To the best knowledge of the authors this has been the first time that formal methods, should they be theorem provers or model checkers, have been applied to the problem of rate-correction for fault-tolerant clock synchronization. Furthermore, the formal development process itself demonstrates how easily existing models can be utilized in the development of new algorithms and their formal verification.

Networks for real-time systems have stringent end-to-end latency and jitter requirements. One cost-efficient way to meet these requirements is the time-triggered communication paradigm which plans the transmission points in time of the frames off-line. This plan prevents contentions of frames on the network and is called a time-triggered schedule (tt-schedule). In general the tt-scheduling is a bin-packing problem, known to be NP-complete, where the complexity is mostly driven by the freedom in topology of the network, its associated hardware restrictions, and application-imposed constraints. Multi-hop networks, in particular, require the synthesis of path-dependent tt-schedules to maintain full determinism of time-triggered communication from sender to receiver. Our experiments using the YICES SMT solver show that the scheduling problem can be solved by YICES out-of-the-box for a few hundred random frame instances on the network. A customized tt-scheduler using YICES as a back-end solver allows to increase this number of frame instances up to tens of thousands. In terms of scheduling quality, the synthesis produces up to ninety percent maximum utilization on a communication link with schedule synthesis times of about half an hour for the biggest examples we have studied. As a nice side-effect the YICES out-of-the-box approach is immediately applicable for the verification of existing (even large-scale) tt-schedules and for debugging more sophisticated tt-schedulers.

TTEthernet is a novel communication infrastructures that allows using a single physical communication infrastructure for distributed applications with mixed-criticality requirements, e.g. the command and control systems and audio/video systems. This is achieved via a fault-tolerant self-stabilizing synchronization strategy, which establishes temporal partitioning and, hence, ensures isolation of the critical dataflows from the non-critical dataflows. The focus in this paper is on the dataflow in TTEthernet. For this we take the synchronization as a given and discuss from a TTEthernet user perspective which communication options TTEthernet provides and how they are aligned and realized. While this paper uses TTEthernet as reference communication infrastructure, the methods and strategies presented are valid for any message-based network.

We give an introduction and survey of a formal modeling and verification approach that has been successfully applied to time-triggered protocols. This method allows us to capture and reason about real-time properties of distributed systems. It relies on the modeling concept of calendar similar to what has been used for a long time in discrete event simulation. It is also supported by efficient symbolic verification tools provided by the SAL environment. We present the basis of the modeling method and discuss two related verification approaches for analyzing complex, real-time distributed systems.

Fault-tolerant time-triggered communication relies on the synchronization of local clocks. The startup problem is the problem of reaching a sufficient degree of synchronization after power-on of the system. The complexity of this problem naturally depends on the system assumptions. The system assumptions in this paper were compiled from cooperation with partners in the automotive and aeronautic industry. We present a general startup strategy for safety-critical systems that discusses the solution to the startup problem from an abstract point of view. From this abstract view we derive and analyze a new startup algorithm that is used in a TTP/C research derivative protocol (LTTP). We also analyze the FlexRay startup algorithm and discuss its behavior in presence of simple failures. The analyses were done by exhaustive fault simulation using the SAL model checker. While LTTP was found to tolerate the arbitrary failure of one node, the FlexRay startup shows to be vulnerable to simple failure modes

The increasing performance of modern model-checking tools offers high potential for the computer-aided design of fault-tolerant algorithms. Instead of relying on human imagination to generate taxing failure scenarios to probe a fault-tolerant algorithm during development, we define the fault behavior of a faulty process at its interfaces to the remaining system and use model checking to automatically examine all possible failure scenarios. We call this approach "exhaustive fault simulation". In this paper we illustrate exhaustive fault simulation using a new startup algorithm for the time-triggered architecture (TTA) and show that this approach is fast enough to be deployed in the design loop. We use the SAL toolset from SRI for our experiments and describe an approach to modeling and analyzing fault-tolerant algorithms that exploits the capabilities of tools such as this.

When time-triggered (TT) systems are to be deployed for large embedded real-time (RT) control systems in cars and airplanes, one way to overcome bandwidth limitations and achieve complexity reduction is the organization in clusters of strongly interacting computing nodes with well-defined interfaces. In this case, clock synchronization of different cluster times supports meaningful exchange of time-related data between clusters and allows coordinated control. This paper addresses fault-tolerant clock synchronization of clusters for TT systems that are already internally synchronized. By addressing systematic and stochastic errors of cluster times differently, the influence of systematic errors is eliminated and the quality of synchronization only depends on stochastic errors. Since systematic errors of cluster times are at least an order of magnitude larger than stochastic errors for typical RT embedded control systems, the presented algorithm achieves a significant improvement to known synchronization algorithms. An implementation of the proposed clock synchronization algorithm on top of the Time-Triggered Architecture and experiments show that synchronization is achieved with accuracy values of less than one microsecond.

This paper discusses measures to make a distributed system based on the time-triggered architecture resistant to arbitrary node failures. The presented approach introduces a central guardian as part of the interconnection network. This guardian acts as a supervising unit to node computers by checking for compliance to the fault hypothesis at the network interface of the node. By implementing appropriate algorithms the guardian is able to transform failure modes (of nodes) that cannot be tolerated by the fault hypothesis of the TTP/C protocol. This transformation ensures that at the interface to correct nodes, even an arbitrarily faulty node will be compliant to the fault hypothesis of the TTP/C communication protocol.

Immediately after power-up, synchronous distributed systems need some time until essential timing properties, which are required to operate correctly, are established. We say that synchronous systems are initially in asynchronous operation. In this paper, we present an algorithm and architectural guidelines that assure the transition from asynchronous to synchronous operation within a bounded duration even in case of failures.