The transition to connected vehicles (CVs) is shaping the future of autonomous and highly connected vehicles. As the vehicles’ reliance on increasingly complex software increases, more central points of failure are introduced. This necessitates more thorough and scalable anomaly and intrusion detection systems. Conventional approaches can struggle with the complex and dynamic structures of CVs while not providing any automatic response actions. Therefore, machine learning-based anomaly detection methods are gaining popularity. Reinforcement Learning especially has shown the potential to increase configurability and provide end-to-end solutions. Thus, this paper presents a Systematic Literature Review (SLR) on Reinforcement Learning (RL) for anomaly and intrusion detection, applicable for CVs. In it, different RL algorithms and the analysis of how they can be used for training and detecting anomalies are presented. The problem formulations for different approaches are compared, their advantages and challenges are discussed. Additionally, the potential of using RL to react to anomalies is presented, discussed and advocated for. The results of the survey indicate that many of the identified RL formulations are not inherently superior from supervised learning approaches. However, RL techniques can result in better configurability of anomaly detection system, enabling site reliability engineers to efficiently adjust the model via reward functions. Furthermore, it becomes evident that the strengths of RL-based anomaly detection are best leveraged when used for reacting to anomalies. Therefore, the authors propose to focus on the contextual or sequential decision making formulation including corrective actions, compared to the classification-based detection formulations using RL.

AI-controlled robotic systems can introduce significant risks to both humans and the environment. Traditional reliability assessment methods fall short in addressing the complexities of these systems, particularly when dealing with black-box or dynamically changing control policies. The traditional approaches are applied manually and do not consider frequent software updates. In this paper, we present RelAIBotiX, a new methodology that enables dynamic and continuous reliability assessment, specifically tailored for robotic systems controlled by AI algorithms. RelAIBotiX combines four methods: (i) Skill Detection that automatically identifies executed skills using deep learning techniques, (ii) Behavioral Analysis that creates an operational profile of the robotic system containing information about the skill execution sequence, active components for each skill, and their utilization intensity that influence their failure rate, (iii) Reliability Model Generation that automatically transforms the operational profile and reliability data of robotic hardware components into quantitative hybrid reliability models, and (iv) Reliability Model Solver for the numerical evaluation of the generated reliability models. Our evaluation included computing the reliability of the system, the probability of failure of individual skills, and component sensitivity analysis. We validated the applicability of the proposed framework across five simulative and real-world setups.

Advancements in the Internet of Things (IoT) applied by automation systems lead to an increasing and flexible interconnection of various devices and the generation of voluminous data. In the domain of home automation for instance, having a multitude of interconnected data leveraged to knowledge could enhance user-centeredness and energy efficiency, among other use cases. However, the data as well as data sources are dynamic and heterogeneous, posing a great challenge to unfold their full capabilities. Thus, a unifying and correspondingly dynamic architecture is needed for supporting the flexibility and extensibility of such systems and data sources. In this paper, we present the concept and implementation of a context-aware INTElligent LIVing (inteLiv) architecture. Consisting of a data layer, a context layer and a service layer, the architecture abstracts the heterogeneous data of the linked devices via a middleware. The resulting system enables the collection of data across networked heterogeneous devices and models the collected data by using a unified dynamic context model. A hybrid approach was chosen for the designed context model: A combination of an ontology-based and property graph-based model represent a core aspect of this work as opposed to common static ontological representations and pre-defined context-based use cases. In the following contribution, the inteLiv ontology is first defined to describe the concepts of the intelligent living system. Based on the modeled context, reasoning algorithms also enable the derivation of new context information. The context is stored as a knowledge graph and can be used for context-aware applications, which can be designed during runtime. Furthermore, a user interaction with the system is possible via a web interface.

Since the introduction of the term Cyber-Physical Systems (CPS) in 2006, they came to a long way. CPS are now autonomous and networked systems of systems with state-space exceeding the capabilities of conventional risk analysis methods. Model-based fault injection methods allow assessment of a system’s fault tolerance not only during its design phase but also in the course of operation. This allows the evaluation of updates and new modules before deploying such changes to a real system. Such operational model-based fault injection on a system’s digital twin can ensure continuous safety throughout all system life cycles.Modern risk analysis tools and Machine Learning-based safety methods require vast amounts of representative input and training data. Such methods not only will require mountains of erroneous time-series data from a myriad of operational cycles, but also corresponding fault parameter labels. As the state space of the system component explodes in complexity, it becomes problematic to cover all possible component fault combinations. As such, only those faults that could lead to potential failures or increased risk scenarios are of interest for automated safety assessment methodologies. It is clear that an intelligent and effective model-based fault injection method is required for the operational safety assessment of industrial CPS.Recently we introduced a new model-based fault injection method implemented as a highly customizable Simulink block called FIBlock. It supports the model-based injection of typical faults of CPS components such as sensors, software, computing, and network hardware. In this paper, we proposed a Deep Learning-based approach for model-based fault injection called FIDGET. It extends the FIBlock with Deep Reinforcement Learning capabilities. We employed a Deep Deterministic Policy Gradient algorithm with Long Short-Term Memory (LSTM) architecture to train the Reinforcement Learning agent to per-form the automated search of fault parameters that yield the biggest system response. It allows automatic generation of labeled training data for further use in risk analysis tools or to train fault classifiers. The generated training data consists of errors that lead to the biggest response (i.e., disturbance) of the system.

The current trend in production systems development is shifted towards the software part. This is emphasized by the concepts of digital twins and Software-Defined Manufacturing (SDM). These software-intensive and safety-critical systems have more frequent software updates to address higher system flexibility and adjustable production processes. SDM-systems bring new challenges to the reliability assessment. Each update can change the system behavior significantly. This leads to the necessity to reconduct reliability assessment automatically before each software update.In this paper, we introduce the concept of an automated model-based reliability assessment method sensible to the software update. The paper describes the key idea of the method and demonstrates its application on a model of a robotic manipulator.