The existing approaches to enforcing runtime properties and handling failures in autonomous agents primarily focus on single-agent systems, responding to violations by immediately rejecting unsafe actions. In this paper, we extend the notion of safety shields for Belief-Desire-Intention agents by proposing a revised version where a shield can observe and respond not only to what the agent itself does, but also to changes from other agents, when these affect the shielded agent’s behaviour. Our shields suspend intentions that would break a formal specification and resume them once it is safe to do so. To support this, we introduce recovery shields, a new mechanism that defines when a suspended intention can be safely resumed. Our main contributions are extensions to the reasoning cycle and operational semantics of AgentSpeak(L), as well as an implementation in the JaCaMo platform.

Runtime Verification is a lightweight formal verification technique used to verify whether a system behaves as expected at runtime. Expected behaviour is typically formally specified using properties, which are used to automatically synthesise monitors. Properties that can be verified at runtime by a monitor are called monitorable , while those that cannot are termed non-monitorable . In this paper, we revisit the notion of monitorability and demonstrate how non-monitorable properties can still be used to generate partial monitors. We tackle this from two different perspectives: (i) by recognising that a monitor can give up on monitoring the property under analysis if it recognises that the monitoring will never conclude the satisfaction or violation of the property; (ii) by recognising that a monitor can give up on events that are not necessary for successful monitoring of the property under analysis. By considering these two aspects, we present how to achieve partial monitoring of Linear Temporal Logic properties by building upon the standard monitor construction. Finally, we present a prototype implementation of our approach and its application to a remote inspection case study, as well as a set of evaluation experiments to stress test our approach using synthetic properties. • How to extend standard monitor construction to handle non-monitorable properties. • Non-monitorable properties can be partially monitored. • Tackling partial monitorability directly on the monitor makes the approach formalism-agnostic. • A monitor can give up on a property if it recognises it will never conclude its verification. • A monitor can give up on events if such events are not of interest for the verification of the property.

Autonomous robotic systems systems are both safety- and security-critical, since a breach in system security may impact safety. In such critical systems, formal verification is used to model the system and verify that it obeys specific functional and safety properties. Independently, threat modelling is used to analyse and manage the cyber security threats that such systems may encounter. Both verification and threat analysis serve the purpose of ensuring that the system will be reliable, albeit from differing perspectives. In prior work, we argued that these analyses should be used to inform one another and, in this paper, we extend our previously defined methodology for security-minded verification by incorporating runtime verification. To illustrate our approach, we analyse an algorithm for sending Cooperative Awareness Messages between autonomous vehicles. Our analysis centres on identifying STRIDE security threats. We show how these can be formalised, and subsequently verified, using a combination of formal tools for static aspects, namely Promela/SPIN and Dafny, and generate runtime monitors for dynamic verification. Our approach allows us to focus our verification effort on those security properties that are particularly important and to consider safety and security in tandem, both statically and at runtime.

Engineering a software system can be a complex process and prone to failure. This is exacerbated when the system under consideration presents some degree of autonomy, such as in cognitive agents. In this paper, we use runtime verification as a way to enforce safety properties on Belief-Desire-Intention (BDI) agents by enveloping certain plans in safety shields. These shields function as a failure handling mechanism, they can detect and avoid violations in shielded plans. The safety shields also provide automated failure recovery by attempting alternative execution paths to avoid violations.

Abstract This paper presents an extension of the Predictive Runtime Verification (PRV) paradigm to consider multiple models of the System Under Analysis (SUA). We call this extension Multi-Model PRV. Typically, PRV attempts to predict the satisfaction or violation of a property based on a trace and a (single) formal model of the SUA. However, contemporary node- or component-based systems (e.g. robotic systems) may benefit from monitoring based on a model of each component. We show how a Multi-Model PRV approach can be applied in either a centralised or a compositional way (where the property is compositional), as best suits the SUA. Crucially, our approach is formalism-agnostic. We demonstrate our approach using an illustrative example of a Mars Curiosity rover simulation and evaluate our contribution via a prototype implementation.

When applying formal verification to a system that interacts with the real world, we must use a model of the environment. This model represents an abstraction of the actual environment, so it is necessarily incomplete and hence presents an issue for system verification. If the actual environment matches the model, then the verification is correct; however, if the environment falls outside the abstraction captured by the model, then we cannot guarantee that the system is well behaved. A solution to this problem consists in exploiting the model of the environment used for statically verifying the system’s behaviour and, if the verification succeeds, using it also for validating the model against the real environment via runtime verification. The article discusses this approach and demonstrates its feasibility by presenting its implementation on top of a framework integrating the Agent Java PathFinder model checker. A high-level Domain Specific Language is used to model the environment in a user-friendly way; the latter is then compiled to trace expressions for both static formal verification and runtime verification. To evaluate our approach, we apply it to two different case studies: an autonomous cruise control system and a simulation of the Mars Curiosity rover.