The strategic implementation of Artificial Intelligence (AI) technologies in the industry requires extending conventional engineering disciplines to include AI-specific considerations. This allows the management and evaluation of risks associated with AI technologies, thereby unlocking their potential to improve system autonomy. Moreover, this allows to ensure a high level of confidence among stakeholders, such as regulatory authorities and clients. In this context, this paper provides an overview of the findings from the confiance.ai research program, which aims to develop methodological guidelines/ processes for engineering trustworthy AI systems. These processes are the result of collaborative efforts by a large group of experts focused on AI system trustworthiness. Data trustworthiness assessment and risk analysis are examples of these methodological processes.

Validating the safety of automated systems is a highly complex task that cannot be done effectively through one validation methodology alone. As a result, current trends recommend adopting a multi-pillar approach for the validation of such systems. In this paper, we share our experience in applying a combined safety approach for the safety evaluation of an automated vehicle. The evaluation approach couples Model-Driven Engineering paradigm and simulation for a detailed assessment of critical scenarios. Based on a system model, we perform analytical safety analysis to identify the critical failures that may lead to undesired events. The analytical analysis is complemented by extensive simulation experiments to assess finer the impact of the identified malfunctions. The overall approach builds upon a tool chain consisting of Physistem as a modeling framework, Papyrus-Sophia for dysfunctional analysis support, and Phisim as a simulation environment. We report on the experiment results and discuss the advantages and limitations that the proposed approach brings for the evaluation of safety-critical automated systems.

This paper presents the Ontology Language for the Dependability of Automated Systems (OLDAS), a modeling language based on Unified Modeling Language (UML) that aims to support dependability assessment for Automated Systems (ASs), i.e., systems intended to perform a function with minimal or no human intervention. OLDAS extends the Unified Foundational Ontology (UFO) and embeds validation rules to prevent constraint violations in ASs analysis. Specifically, the paper presents how OLDAS can support different activities during the design of ASs, from the definition of the Operational Design Domain to scenario-based analysis. OLDAS is available as a plugin of the open-source Papyrus for Robotics framework.

To allow the large-scale deployment of automated and connected vehicles, their safety must be ensured. The Operational Design Domain (ODD) aims to define under which conditions an Automated Driving System (ADS) can operate safely: speed, type of road, weather conditions, etc. Clearly identifying the characteristics and boundaries of the ODD is an important issue today for ADS. To this end, the design and use of an ODD taxonomy seems to be a relevant approach considered by both the industrial and academic worlds. Therefore, in this paper, we propose an analysis and comparison of the main existing taxonomies. We also define a new generic taxonomy, combining the different approaches proposed in the literature, applicable to both vehicles and road sections. Then, this taxonomy is applied to a specific use case (Bus Station Automated Service). Finally, we identify potential directions for an extended ODD taxonomy.

Dynamic Dependability Management (DDM) is a promising approach to guarantee and monitor the ability of safety-critical Automated Systems (ASs) to deliver the intended service with an acceptable risk level. However, the non-interpretability and lack of specifications of the Learning-Enabled Components (LECs) used in ASs make this mission particularly challenging. Some existing DDM techniques overcome these limitations by using probabilistic environmental perception knowledge associated with predicting behavior changes for the agents in the environment. We propose to improve these techniques with a supervisory system that considers hazard analysis and risk assessment from the design stage. This hazard analysis is based on a characterization of the AS's operational domain (i.e., its scenario space, including unsafe ones). The proposed supervisory system also considers the uncertainty estimation and interaction between AS components through the whole perception-planning-control pipeline. Our framework then proposes leveraging and handling uncertainty from LEC components toward building safer ASs.

We are interested in the safety of critical systems whose development is based on models. Implementing failure analyses for this kind of system requires modeling the failures and conditions of their appearances. The failure analysis approaches are mainly based on the structures of systems where boolean equations depict the propagation of faults. The objective of the analysis is to calculate Minimal Cut Sets (MCS), i.e., the smallest sets of basic faults that may cause a feared event and their probabilities. The most efficient MCS resolution method is based on Binary Decision Diagrams (BDD). In this paper, we present a model-based toolset to construct from SysML structural models of systems, the fault trees, and their BDD-representation enabling us to compute MCS. Faults in our approach are not limited to boolean variables; they can be expressed by constraints coming from an arbitrary decidable theory. We validate the toolset capabilities with an oil burner system use case.

The use of Artificial Intelligence (AI)-based systems, using particularly Machine Learning (ML) classifiers, is growing rapidly and finding uses in many industries. Most of these industries have critical safety, security, and dependability requirements. Despite this rapid growth, interest in the security of these systems has only arisen in the last few years and it is not yet well-studied. There is a want for a formal notion of security for ML systems, similar to that used in classical information security. We took this statement toward security threat modeling and analysis in ML-based systems, focusing on the adversarial example threat. An adversarial example threat is an input of the classifier that was maliciously modified to induce a misclassification. Identifying this threat at the architecture design stage before proceeding with system development is a critical milestone in the development process of secure ML systems. In this paper, we propose an approach to adversarial example threat specification and detection in component-based software architecture models. We use first-order and modal logic as an abstract and technology-independent formalism. The general idea of the approach is to specify the threat as property of a modeled system such that the violation of the specified property indicates the presence of the threat. We demonstrate the applicability of the method through a classifier used in a recommendation system.

This paper presents the coupling between systems engineering (SE) and safety analysis (SA), by proposing an approach that allows integrating a safety analysis methodology to the ‘Papyrus 4 robotics’ (P4R) framework. We are especially interesting by automatic generation of fault trees in Open-PSA format from a UML Models.

Albeit Artificial Intelligent (AI) based systems are nowadays deployed in a variety of safety critical domains, current engineering methods and standards are barely applicable for their development and assurance.The lack of common criteria to assess safety levels as well as the dependency of certain development phases w.r.t. the chosen technology (e.g., machine learning modules) are among the identified drawbacks.In addition, the development of such engineering methods has been hampered by the emerging challenges in AI-based systems design mainly regarding autonomy, correctness and prevention of catastrophic risks.In this paper we propose an approach to conduct a safeby-design development process for AI based systems.The approach relies upon a method which benefits from a reference AI architecture and safety principles.This contribution helps to address safety concerns and to comprehend current AI architectures diversity and particularities.

Nowadays standards provide recommendations for system development process in various phases and activities, aiming to address and reduce the risks associated to poor or flawed designs. Indeed, to increase systems assurance levels, a variety of concerns like safety, security and reliability are currently considered as critical and targeted by standards. However, to ensure such assurance levels, all stakeholders involved in the product development cycle (manufacturers, regulators, etc.) need to have a clear understanding of the standards contents. This paper presents an approach that aims to ease the common understanding of standards and their adoption by automatically generating models from them. The approach mostly relies upon the Business Process Model and Notation (BPMN) language. The potential benefits of this structured graphical view of standards are to facilitate their comprehension, visualization and navigation by the stakeholders involved in product development cycle.

The implementations and testing in real conditions of Autonomous Vehicles (AV) for private usage show important advances. However, a lack still exists in addressing the particularities of AVs for Public Transportation. Such particularities range from limited safety mechanisms aboard, risky situations associated to particular users and complex self-driving situations up to the limited passengers-vehicle interactions possible. Since, to our knowledge, no comprehensive safety assessment actually exists and the current automotive related standards do not address identified aspects, in this paper, we propose to conduct a minimal but representative safety assessment based upon a local but real autonomous vehicle implementation. To conduct our study, the Hazard Analysis and Risks Assessment introduced in the ISO 26262 standard is taken as a basis. Initial outcomes suggest that critical autonomy aspects, like machine learning of complex operational situations, the metrics for quantitative assessment of autonomy, and potential conflicts between autonomy principles and external safety fences can have critical safety impacts and demand further discussions.

Legal markup (metadata) is an important prerequisite for the elaboration of legal requirements. Manually encoding legal texts into a markup representation is laborious, specially for large legal corpora amassed over decades and centuries. At the same time, automating the generation of markup in a fully accurate manner presents a challenge due to the flexibility of the natural-language content in legal texts and variations in how these texts are organized. Following an action research method, we successfully collaborated with the Government of Luxembourg in transitioning five major legislative codes from plain-text to a legal markup format. Our work focused on generating markup for the structural elements of the underlying codes. The technical basis for our work is an adaptation and enhancement of an academic markup generation tool developed in our prior research [1]. We reflect on the experience gained from applying automated markup generation at large scales. In particular, we elaborate the decisions we made in order to strike a cost-effective balance between automation and manual work for legal markup generation. We evaluate the quality of automatically-generated structural markup in real-world conditions and subject to the practical considerations of our collaborating partner.

When elaborating compliance requirements, analysts need to follow the cross references in the underlying legal texts and consider the additional information in the cited provisions. To enable easier navigation and handling of cross references, automation is necessary for recognizing the natural language patterns used in cross reference expressions (cross reference detection), and for interpreting these expressions and linking them to the target provisions (cross reference resolution). In this paper, we propose a solution for automated detection and resolution of legal cross references. We ground our work on Luxembourg's legislative texts, both for studying the natural language patterns in cross reference expressions and for evaluating the accuracy and scalability of our solution.

A major activity in many industries is to integrate software artifacts such that the functional and performance requirements are properly taken care of. In this paper, we focus on the problem of minimizing the risk of CPU time shortage in integrated embedded systems. In order to minimize this risk, we manipulate the start time (offset) of the software executables such that the system real-time constraints are satisfied, and further, the maximum CPU time usage is minimized. We develop a number of search-based optimization algorithms, specifically designed to work for large search spaces, to compute offsets for concurrent software executables with the objective of minimizing CPU usage. We evaluated and compared our algorithms by applying them to a large automotive software system. Our experience shows that our algorithms can automatically generate offsets such that the maximum CPU usage is very close to the known lower bound imposed by the domain constraints. Further, our approach finds limits on the maximum CPU usage lower than those found by a random strategy, and is not slower than a random strategy. Finally, our work achieves better results than the CPU usage minimization techniques devised by domain experts.

The documentation of customer needs from the source specifications in a modeling environment for allocating them to architectural elements needs efficient tools and techniques in requirement engineering. Once requirements are present in models, enhancement with suitable properties, classification, prioritization. and allocation on system architecture are then possible. A downside is that the customer needs are likely to evolve over time, and then, we would need to manually redo the modeling of the requirements. Then, what we want to avoid is the manual definition of the customer needs from the source documents as a requirements model in the target environment. We propose in this paper a solution to import and export in Papyrus MDT, a UML modeling tool, the customers' needs from Microsoft documents using the Requirement Interchange Format, ReqIF.