Abstract As robotic systems such as autonomous cars and delivery drones assume greater roles and responsibilities within society, the likelihood and impact of catastrophic software failure within those systems is increased. To aid researchers in the development of new methods to measure and assure the safety and quality of robotics software, we systematically curated a dataset of 221 bugs across 7 popular and diverse software systems implemented via the Robot Operating System (ROS). We produce historically accurate recreations of each of the 221 defective software versions in the form of Docker images, and use a grounded theory approach to examine and categorize their corresponding faults, failures, and fixes. Finally, we reflect on the implications of our findings and outline future research directions for the community.

Effective testing of message-oriented software requires describing the expected behaviour of the system and the causality relations between messages. This is often achieved with formal specifications based on temporal logics that require both first-order and metric temporal constructs - to specify constraints over data and real time. This paper proposes a technique to automatically generate tests for metric first-order temporal specifications that match well-understood specification patterns. Our approach takes in properties in a high-level specification language and identifies test schemas (strategies) that are likely to falsify the property. Schemas correspond to abstract classes of execution traces, that can be refined by introducing assumptions about the system. At the low level, concrete traces are successively produced for each schema using property-based testing principles. We instantiate this approach for a popular robotic middleware, ROS, and evaluate it on two systems, showing that schema-based test generation is effective for message-oriented software.

Abstract Model-driven engineering (MDE) addresses central aspects of robotics software development. MDE could enable domain experts to leverage the expressiveness of models, while implementation details on different hardware platforms would be handled by automatic code generation. Today, despite strong MDE efforts in the robotics research community, most evidence points to manual code development being the norm. A possible reason for MDE not being accepted by robot software developers could be the wide range of applications and target platforms, which make all-encompassing MDE IDEs hard to develop and maintain. Therefore, we chose to leverage a large corpus of open-source software widely adopted by the robotics community to extract common structures and gain insight on how and where MDE can support the developers to work more efficiently. We pursue modeling as a complement, rather than imposing MDE as separate solution. Our previous work introduced metamodels to describe components, their interactions, and their resulting composition. In this paper, we present two methods based on metamodels for automated generation of models from manually written artifacts: (1) through static code analysis and (2) by monitoring the execution of a running system. For both methods, we present tools that leverage the potentials of our contributions, with a special focus on their application at runtime to observe and diagnose a real system during its execution. A comprehensive example is provided as a walk-through for robotics software practitioners.

Robots are currently deployed in safety-critical domains but proper techniques to assess the functional safety of their software are yet to be adopted. This is particularly critical in ROS, where highly configurable robots are built by composing third-party modules. To promote adoption, we advocate the use of lightweight formal methods, automatic techniques with minimal user input and intuitive feedback. This paper proposes a technique to automatically verify system-wide safety properties of ROS-based applications at static time. It is based in the formalization of ROS architectural models and node behaviour in Electrum, over which system-wide specifications are subsequently model checked. To automate the analysis, it is deployed as a plug-in for HAROS, a framework for the assessment of ROS software quality aimed at the ROS community. The technique is evaluated in a real robot, AgRob V16, with positive results.

In principle, Model-Driven Engineering (MDE) addresses central aspects of robotics software development. Domain experts could leverage the expressiveness of models; implementation details over different hardware could be handled by automatic code generation. In practice, most evidence points to manual code development as the norm, despite several MDE efforts in robotics. Possible reasons for this disconnect are the wide ranges of applications and target platforms making all-encompassing MDE IDEs hard to develop and maintain, with developers reverting to writing code manually. Acknowledging this, and given the opportunity to leverage a large corpus of open-source software widely adopted by the robotics community, we pursue modeling as a complement, rather than an alternative, to manually written code. Our previous work introduced metamodels to describe components, their interactions, and their resulting composition, as inspired by, but not limited to, the de-facto standard Robot Operating System (ROS). In this paper we put such metamodels into use through two contributions. First, we automate the generation of models from manually written artifacts through extraction from source code and runtime system monitoring. Second, we make available an easy-to-use web infrastructure to perform the extraction, together with a growing database of models so generated. Our aim with this tooling, publicly available both as-a-service and as source code, is to lower the MDE barrier for practitioners and leverage models to 1) improve the understanding of manually written code; 2) perform correctness checks; and 3) systematize the definition and adoption of best practices through large-scale generation of models from existing code. A comprehensive example is provided as a walk-through for robotics software practitioners.

The Robot Operating System (ROS) is nowadays one of the most popular frameworks for developing robotic applications. To ensure the (much needed) dependability and safety of such applications we forecast an increasing demand for ROS-specific coding standards, static analyzers, and tools alike. Unfortunately, the development of such standards and tools can be hampered by ROS modularity and configurability, namely the substantial number of primitives (and respective variants) that must, in principle, be considered. To quantify the severity of this problem, we have mined a large number of existing ROS packages to understand how its primitives are used in practice, and to determine which combinations of primitives are most popular. This paper presents and discusses the results of this study, and hopefully provides some guidance for future standardization efforts and tool developers.

Robots are being increasingly used in safety-critical contexts, such as transportation and health. The need for flexible behavior in these contexts, due to human interaction factors or unstructured operating environments, led to a transition from hardware- to software-based safety mechanisms in robotic systems, whose reliability and quality is imperative to guarantee. Source code static analysis is a key component in formal software verification. It consists on inspecting code, often using automated tools, to determine a set of relevant properties that are known to influence the occurrence of defects in the final product. This paper presents HAROS, a generic, plug-in-driven, framework to evaluate code quality, through static analysis, in the context of the Robot Operating System (ROS), one of the most widely used robotic middleware. This tool (equipped with plug-ins for computing metrics and conformance to coding standards) was applied to several publicly available ROS repositories, whose results are also reported in the paper, thus providing a first overview of the internal quality of the software being developed in this community.