The runtime verification of multi-domain software applications implementing the behaviors of modern robots is a challenging task. On the one hand, assertion-based verification (ABV) has shown great potential to check the correctness of complex systems at runtime. On the other hand, the computational overhead introduced by runtime ABV can be substantial, variable and non-deterministic. As a consequence, applying accurate ABV at runtime to autonomous robots, which are often characterized by resource-constrained computing architectures, can lead to severe slowdowns of the software execution and failures of temporal constraints, thus compromising the overall system’s correctness. We address this challenge by proposing a platform for runtime ABV that implements monitor synthesis from signal temporal logic assertions and dynamic monitor migration across edge devices and the cloud. The synthesized monitors are wrapped into ROS-compliant nodes and connected to the system under verification. The overall ABV framework and the related migration mechanism are then containerized with Docker for both edge and cloud computing. To evaluate the proposed platform, we present the results obtained with a set of synthetic benchmarks and with an industrial case study, which implements the mission of a Robotnik RB-Kairos mobile robot in a smart manufacturing production line. Note to Practitioners . This article was motivated by the need for accurate and runtime verification of robotic systems software. Verification and validation of intelligent systems are often incomplete, as they cannot anticipate all potential scenarios, including errors or unexpected events. On top of this, assertion-based verification can also be resource-intensive; therefore, careful use of resources is required to avoid overloading the robot’s computational resources with the monitors. To achieve this, we used signal temporal logic, a widely accepted solution to monitor robotic and distributed applications. The main contribution of this work is a framework that can automatically synthesize the monitors that interface with the Robot Operating System (ROS) and also the capability of optimizing the end-to-end latency of verification at runtime by exploiting a distributed computing architecture (i.e., edge-cloud). In future work, we will address not only the minimization of end-to-end latency but also the timing upper bound of monitors to achieve runtime deterministic verification.

Over the past few decades, the verification community has developed several specification miners as an alternative to manual assertion definition. However, assessing their effectiveness remains a challenging task. Most studies evaluate these miners using predefined ranking metrics, which often fail to ensure the quality of the inferred specifications, especially when no fixed ground truth exists and the relevance of the specifications varies depending on the use case. This paper presents a comprehensive framework aimed at facilitating the evaluation and comparison of Linear Temporal Logic (LTL) specification miners. Unlike traditional approaches, which struggle with subjective analyses and complex tool configurations, our framework provides a structured method for assessing and comparing the quality of specifications generated by multiple sources, using both semantic and syntactic techniques. To achieve this, the framework offers users an easy-to-extend environment for installing, configuring, and running third-party miners via Docker containers. Additionally’ it supports the inclusion of new evaluation methods through a modular design. Miner comparison can be based either on user-defined designs or on synthetic benchmarks, which are automatically generated to serve as a non-subjective ground truth for the evaluation of the miners. We demonstrate the utility of our framework through comparative analyses with four well-known LTL miners, illustrating its ability to standardize and enhance the specification mining evaluation process.

Several approaches have been proposed over the years to automatically generate specifications of digital systems by means of dynamic techniques, which are now ripe to be applied in large-scale industrial scenarios. On the other hand, the automatic extraction of specifications for the hybrid domain, where systems express both discrete and continuous behaviours, remains mainly unexplored. Therefore, in this paper, we propose a tool for dynamically mining the specifications of hybrid systems in the form of assertions compliant with the Signal Temporal Logic (STL), which has been proven to be effective at capturing the behaviours of such systems. Our approach takes as input a set of execution traces of the target system and mixes clustering and decision-tree algorithms to generate STL assertions that describe what has been actually implemented.

Approximate Computing (AxC) paradigm was introduced to achieve higher power efficiency, lower area and better performances w.r.t. a “classical” computing system at the cost of a degraded, but still acceptable, output accuracy [1]. AxC can be applied at several abstraction levels of a given computing system: from circuit to algorithm [1], leading to a wide design exploration space that quickly became the bottleneck for successfully deploying AxC. Indeed, the literature proposes many works to automatically trade-off between output accuracy and performances [2]. However, most of them lack the capability to identify resilient elements (e.g, HW component, HDL statements, etc.) of the design to be approximated. Consequently, exploring the design for AxC generally results in a long and tedious procedure. Existing approaches generate approximate variants of the Design Under Exploration (DUE). Every variant is then executed/simulated in order to determine the accuracy degradation [3], which depends on the application and requires a specific metric to be computed (e.g., similarity index, hamming distance, etc.).

State-of-the-art mining tools generate assertions that either describe the temporal relations between Boolean expressions, or extract non-temporal logic formulas between more complex propositions involving arithmetic and/or relational operators. This paper presents a method for generating Linear Temporal Logic (LTL) assertions containing both the temporal dimension and the complexity of non-Boolean expressions. Starting from a set of simulation traces of the design under verification (DUV), our approach generates LTL assertions where the Boolean layer contains expressions of the form c = ne, c ≤ ne, c ≥ ne, cl≤ ne ≤ cr, where c, cl, crare constants of numeric type, and ne is a numerical expression predicating over the variables of the DUV. The method exploits a clustering algorithm to mine a meaningful set of propositions following the aforementioned structure. After that, the propositions are used to generate temporal assertions through a decision tree algorithm. Experimental results show real improvements with respect to the state-of-the-art in terms of fault coverage and readability of the mined assertions.

This article presents HARM, a tool to generate linear temporal logic (LTL) assertions starting from a set of user-defined hints and the simulation traces of the design under verification (DUV). The tool is agnostic with respect to the design from which the trace was generated, thus the DUV source code is not necessary. The user-defined hints involve LTL templates, propositions, and ranking metrics that are exploited by the assertion miner to reduce the search space and improve the quality of the generated assertions. This way, the tool supports the work of the verification engineer by including his/her insights in the process of automatically generating assertions. The experimental results show real improvements with respect to the state-of-the-art in terms of assertion coverage and scalability.

This paper proposes an architecture and a related automatic flow to generate, orchestrate and deploy a ROS-compliant verification environment for robotic systems. The architecture enables assertion-based verification by exploiting monitors automatically synthesized from LTL assertions. The monitors are encapsulated in plug-and-play ROS nodes that do not require any modification to the system under verification (SUV). To guarantee both verification accuracy and real-time constraints of the system in a resource-constrained environment even after the monitor integration, we define a novel approach to move the monitor evaluation across the different layers of an edge-to-cloud computing platform. The verification environment is containerized for both cloud and edge computing using Docker to enable system portability and to handle, at run-time, the resources allocated for verification. The effectiveness and efficiency of the proposed architecture have been evaluated on a complex distributed system implementing a mobile robot path planner based on 3D simultaneous localization and mapping.

Many verification strategies have been proposed to identify unexpected behaviours in system-level implementations. However, when one of such behaviours is found, the verification engineer still has to understand its cause to fix the originating error. This requires a tedious and long process, generally based on manual inspection of the execution traces of the design under verification. Nevertheless, usually only a few instructions in the execution traces are relevant to understand the cause of the unexpected behaviour. To help the verification engineers identifying and focusing on such instructions, we present a tool that automatically removes the irrelevant ones. It works by combining dynamic program slicing with a clustering procedure on the execution traces corresponding to unexpected behaviours. Firstly, program slicing is applied to remove instructions not belonging to the cone of influence of the unexpected behaviour. Then, clusters of instructions based on store operations at the LLVM intermediate representation are created to guide the heuristic in removing further irrelevant instructions.

This paper presents MIST, an all-in-one tool capable of generating a complete environment to verify C/C++ firmwares starting from informal specifications. Given a set of specifications written in natural language, the tool guides the user in translating each specification into an XML formal description, capturing a temporal behavior that must hold in the design. Our XML format guarantees the same expressiveness of linear temporal logic, but it is designed to be used by designers that are not familiar with formal methods. Once each behavior is formalized, MIST automatically generates the corresponding test-bench and checker to stimulate and verify the design. In order to guide the verification process, MIST employs a clustering procedure that classifies the internal states of the firmware. Such classification aims at finding an effective ordering to check the expected behaviors and to advise for possible specification holes. MIST has been fully integrated into the IAR System Embedded Workbench. Its effectiveness and efficiency have been evaluated to formalize and check a complex test-plan for an industrial firmware.