In its quest for approaches to taming uncertainty in self-adaptive systems (SAS), the research community has largely focused on solutions that adapt the SAS architecture or behaviour in response to uncertainty. By comparison, solutions that reduce the uncertainty affecting SAS (other than through the blanket monitoring of their components and environment) remain underexplored. Our previous work proposed Parley , a more nuanced, adaptive approach to SAS uncertainty reduction. To that end, we introduced an SAS architecture comprising an uncertainty reduction controller that drives the adaptive acquisition of new information within the SAS adaptation loop and a tool-supported method that uses probabilistic model checking to synthesise such controllers. The controllers generated by our method deliver optimal tradeoffs between SAS uncertainty reduction benefits and new information acquisition costs with guarantees for the satisfaction of requirements. In this article, we extend Parley to Parley+ by improving the synthesis of these controllers and by expanding the formalisation of Parley+ to prove the validity of the synthesis. We illustrate the use and extend the evaluation of the effectiveness of our approach for mobile robot navigation and service-based system SAS. The evaluation results show that Parley+ can synthesise controllers that help achieve the system’s objectives significantly better than Parley in 88.1% of the cases.

Safe exploration of reinforcement learning (RL) agents is a critical activity for empowering their deployment in many real-world scenarios. When prior knowledge of the target domain or task is unavailable, training RL agents in unknown, black-box environments unavoidably yields significant safety risks. Our ADVICE (Adaptive Shielding with a Contrastive Autoencoder) novel post-shielding approach operates in continuous state and action spaces, distinguishing safe and unsafe features of state-action pairs during training, and uses this knowledge to safeguard the RL agent from executing actions that yield likely hazardous outcomes. Our comprehensive experimental evaluation shows that ADVICE significantly reduces safety violations (≈50%) compared to state-of-the-art safe RL exploration approaches, while maintaining a competitive outcome reward for the synthesised safe policy.

Ensuring the safety of automated driving vehicles is particularly challenging due to the wide range of their operating conditions. This paper introduces CoCoSaFe, a Co mpositional Co de-level formal Sa fety verification F ram e work for automated driving controllers. Unlike traditional verification methods, such as model-based analysis, counterexample detection by guided simulation, or runtime verification through online monitoring, our approach verifies controller implementations directly at code level in an offline setting. Compositional contracts and bounded model checking are employed to assess the implementation of subsystem controllers against invariant sets. For neural network-based controllers, we introduce a scalable three-step decomposition method that utilizes a neural network verifier. CoCoSaFe is applied to adaptive cruise and lane-keeping controllers, for which we derive formal specifications and analytical models of the desired longitudinal and lateral behaviors, amenable for decoupled invariant sets. Various types of traditional and neural network controllers are verified in the order of minutes, showcasing its broad applicability and effectiveness in ensuring behavioral safety of software for automated driving and similar cyber–physical systems.

Abstract The formal safety analysis of automated driving vehicles poses unique challenges due to their dynamic operating conditions and significant complexity. This paper presents a case study of applying formal safety verification to adaptive cruise controllers. Unlike the majority of existing verification approaches in the automotive domain, which only analyze (potentially imperfect) controller models, employ simulation to find counter-examples or use online monitors for runtime verification, our method verifies controllers at code level by utilizing bounded model checking. Verification is performed against an invariant set derived from formal specifications and an analytical model of the required behavior. For neural network controllers, we propose a scalable three-step decomposition, which additionally uses a neural network verifier. We show that both traditionally implemented as well as neural network controllers are verified within minutes. The dual focus on formal safety and implementation verification provides a comprehensive framework applicable to similar cyber-physical systems.

As software systems increasingly interact with humans in application domains such as transportation and healthcare, they raise concerns related to the social, legal, ethical, empathetic, and cultural (SLEEC) norms and values of their stakeholders. Normative non-functional requirements (N-NFRs) are used to capture these concerns by setting SLEEC-relevant boundaries for system behavior. Since N-NFRs need to be specified by multiple stakeholders with widely different, non-technical expertise (ethicists, lawyers, regulators, end users, etc.), N-NFR elicitation is very challenging. To address this difficult task, we introduce N-Check, a novel tool-supported formal approach to N-NFR analysis and debugging. N-Check employs satisfiability checking to identify a broad spectrum of N-NFR well-formedness issues, such as conflicts, redundancy, restrictiveness, and insufficiency, yielding diagnostics that pinpoint their causes in a user-friendly way that enables non-technical stakeholders to understand and fix them. We show the effectiveness and usability of our approach through nine case studies in which teams of ethicists, lawyers, philosophers, psychologists, safety analysts, and engineers used N-Check to analyse and debug 233 N-NFRs, comprising 62 issues for the software underpinning the operation of systems, such as, assistive-care robots and tree-disease detection drones to manufacturing collaborative robots.

We present DeepDECS, a new method for the synthesis of correct-by-construction software controllers for autonomous systems that use deep neural network (DNN) classifiers for the perception step of their decision-making processes. Despite major advances in deep learning in recent years, providing safety guarantees for these systems remains very challenging. Our controller synthesis method addresses this challenge by integrating DNN verification with the synthesis of verified Markov models. The synthesised models correspond to discrete-event software controllers guaranteed to satisfy the safety, dependability and performance requirements of the autonomous system, and to be Pareto optimal with respect to a set of optimisation objectives. We evaluate the method in simulation by using it to synthesise controllers for mobile-robot collision limitation, and for maintaining driver attentiveness in shared-control autonomous driving.

Abstract Deep neural networks (DNNs) are increasingly used in safety-critical autonomous systems as perception components processing high-dimensional image data. Formal analysis of these systems is particularly challenging due to the complexity of the perception DNNs, the sensors (cameras), and the environment conditions. We present a case study applying formal probabilistic analysis techniques to an experimental autonomous system that guides airplanes on taxiways using a perception DNN. We address the above challenges by replacing the camera and the network with a compact abstraction whose transition probabilities are computed from the confusion matrices measuring the performance of the DNN on a representative image data set. As the probabilities are estimated based on empirical data, and thus are subject to error, we also compute confidence intervals in addition to point estimates for these probabilities and thereby strengthen the soundness of the analysis. We also show how to leverage local, DNN-specific analyses as run-time guards to filter out mis-behaving inputs and increase the safety of the overall system. Our findings are applicable to other autonomous systems that use complex DNNs for perception.

The paradox of choice refers to the observation that numerous choices can have a detrimental effect on the quality of decision making. We study this effect in swarms in the context of a resource foraging task. We simulate the evolution of swarms gathering energy from a number of energy sources distributed in a two-dimensional environment. As the number of sources increases, the evolution of the swarm results in reduced levels of efficiency, despite the sources covering more space. Our results indicate that this effect arises because the simultaneous detection of multiple sources is incompatible with an evolutionary scheme that favours greedy energy consumption. In particular, the communication among the agents tends to reduce their efficiency by precluding the evolution of a clear preference for an increasing number of options. The overabundance of explicit information in the swarm about fitness-related options cannot be exploited by the agents lacking complex planning capabilities. If the sensor ranges evolve in addition to the behaviour of the agents, a preference for reduced choice results, and the average range approaches zero as the number of sources increases. Our study thus presents a minimal model for the paradox of choice, which implies several options of experimental verification.