Humans and autonomous cyber-physical systems increasingly share physical space, for example, in industrial manufacturing, autonomous taxis, warehouses, and unmanned package delivery. This makes such autonomous CPS safety-critical because design errors can harm the people in their shared space. To enhance their own safe operation and the safety of humans around them, these CPSs typically use multiple sensor modalities to perceive the environment. Such sensor systems include RADAR, LIDAR, ultra-wideband, SONAR, odometry, GPS, and camera-based sensors to make estimations about their own state and observations of the environment. Traditionally, the observations made by different sensor streams are fused using probabilistic models such as Bayesian filters (e.g., Kalman filters). These filters make assumptions about the distribution of error between the observation and the ground truth for a given sensor and, using such assumptions, attempt to reconstruct a state estimate by computing some weighted combination of observations from multiple sensors (with possibly different error distributions). However, such assumptions can be challenging to model as environments become more complex. Furthermore, such algorithms typically do not account for sensor failures or shifts in the error distribution during deployment. This paper presents an algorithmic framework that defines a notion of spatio-temporal consistency across sensor streams. We eschew the idea of computing a fused state estimate and instead focus on producing a consistent state estimate if the multiple sensor observations are deemed consistent. If we detect an inconsistency in the state estimate, we propose a conservative over-approximation of the state estimate based on the last known consistent estimate. We demonstrate how such a framework can be deployed in an industrial manufacturing case study. We show that such a framework can provide probabilistic runtime assurance using conformal prediction techniques for statistical analyses.

The IEEE International Conference on Software Quality, Reliability, and Security celebrated its 20th anniversary in 2020.We received 156 submissions to the regular and short articles tracks. After the process of desk rejection, all remaining submissions were reviewed by three or more program committee members. Each submission was discussed online by the respective reviewers. Finally, the program chairs recommended six submissions of top quality out of the accepted articles to the IEEE TRANSACTIONS ON RELIABILITY for potential publication. We invited the authors of the six submissions to submit the corresponding revised versions that had addressed the reviewers’ comments to the IEEE TRANSACTIONS ON RELIABILITY for further evaluation. We invited the original reviewers of these QRS submissions to review these revised articles using the standard of the journal for article acceptance. In addition, the abstracts of these six selected articles were published in the QRS 2020 conference proceedings. The six articles we invited span the topics of both reliability and security in hardware and software with a new focus on AI-assisted techniques. The first three articles were published in theMarch 2021 issue; the fourth and fifth articles appear in this issue; and the last article will be included in the September 2021 issue.

This paper rigorously introduces the concept of model-based mutation testing (MBMT) and positions it in the landscape of mutation testing. Two elementary mutation operators, insertion and omission , are exemplarily applied to a hierarchy of graph-based models of increasing expressive power including directed graphs, event sequence graphs, finite-state machines and statecharts. Test cases generated based on the mutated models ( mutants ) are used to determine not only whether each mutant can be killed but also whether there are any faults in the corresponding system under consideration (SUC) developed based on the original model. Novelties of our approach are: (1) evaluation of the fault detection capability (in terms of revealing faults in the SUC) of test sets generated based on the mutated models, and (2) superseding of the great variety of existing mutation operators by iterations and combinations of the two proposed elementary operators. Three case studies were conducted on industrial and commercial real-life systems to demonstrate the feasibility of using the proposed MBMT approach in detecting faults in SUC, and to analyze its characteristic features. Our experimental data suggest that test sets generated based on the mutated models created by insertion operators are more effective in revealing faults in SUC than those generated by omission operators. Worth noting is that test sets following the MBMT approach were able to detect faults in the systems that were tested by manufacturers and independent testing organizations before they were released.

In our daily life we increasingly depend on software-based systems deployed as embedded software control systems in the automotive domain, or the numerous health or government applications. Software-based systems are more and more developed by reusable components available as commercial off-the-shelf components or open source components. The successful introduction of such integrated systems into businesses however does depend whether we trust the system or not. Trust and therewith the quality of software-based systems is determined by many properties such as completeness, consistency, maintainability, security, safety, reliability, and usability, among others. However during the development of software-based systems there are many opportunities to introduce errors in the different phases of the software development lifecycle. Testing is commonly applied as the predominant activity in industry to ensure high software quality providing a wide variety of methods and techniques to detect different types of errors in software-based systems. The panel goal is to discuss software testing strategy and techniques to improve the quality of the software and at the same time to build trust with customers. The panel will discuss the experts view on what the key factors are in developing high quality software-based systems. Through the panel, the discussions shall include the impact of testing on software quality within several domains and their businesses.

Mechatronic systems integrate mechanical, electrical and software subsystems. They are increasingly important in mission-critical infrastructures in different domains including automotive, healthcare, energy and transportation sectors. As such, the adequate testing and validation of such infrastructures are of prime importance. Mechatronic systems testing has been supported by different isolated industrial and research approaches including automated test infrastructures, model-based testing, and test execution engines. While these approaches have individual benefits, they are usually applied in isolation, suffering from different weaknesses. This paper reports on an integrated model-driven approach for mechatronic systems testing, where existing approaches are combined and enhanced. The result is unprecedented levels of automation and testability. We discuss the design and implementation of our approach, illustrating its use with a hybrid calculator example.

In software development practice, testing often accounts for as much as 50% of the total development effort. It is therefore imperative to reduce the cost and improve the effectiveness of software testing by automating the testing process. In the past decades, a substantial amount of research effort has been invested into the development and study of automatic test case generation, automatic test oracles, and other (semi-)automated testing techniques. As the theory and practice of software testing becomes more mature, a deeper and more meaningful automation of the testing process is possible. Therefore, the automation of various testing activities is now becoming an integral part of industrial practice. In response to and in support of these exciting developments, the 5th Workshop on the Automation of Software Test provides a publication forum that bridges the gap between the theory and practice of automated testing.

Testing of graphical user interfaces is important due to its potential to reveal faults in operation and performance of the system under consideration. Most existing test approaches generate test cases as sequences of events of different length. The cost of the test process depends on the number and total length of those test sequences. One of the problems to be encountered is the determination of the test sequence length. Widely accepted hypothesis is that the longer the test sequences, the higher the chances to detect faults. However, there is no evidence that an increase of the test sequence length really affect the fault detection. This paper introduces a reliability theoretical approach to analyze the problem in the light of real-life case studies. Based on a reliability growth model the expected number of additional faults is predicted that will be detected when increasing the length of test sequences.

A wide divergence is observed in projects between test activities planned in the test plan and the actual tests that can be executed. Estimates for test execution computed during the planning are inaccurate without test design. The actual time and resources available are usually less than planned. Assuming that time and resources cannot be changed, a dynamic selection of tests for execution that maximizes quality is required.

With the growing complexity of computer-based systems, their graphical user interfaces have also become more complex. Accordingly, the test and analysis process becomes more tedious and costly. This paper introduces a holistic view of fault modelling that is carried out as a complementary step to system modelling, enabling a scalability of the test process, and providing considerable potential for automation. Event-based notions and tools are used to generate and select test cases systematically. The elements of the approach are illustrated and validated by a case study. This paper does not claim to introduce a novel theoretic approach; rather, it makes use of graph-theoretic results for a practical and simple, but nevertheless powerful, view of modelling, analysis and testing of graphical user interfaces. Copyright © 2005 John Wiley & Sons, Ltd.

It is widely accepted that conventional test methods are not necessarily adequate for testing of component-based software (CBS). As a consequence, also conventional test tools cause similar problems for the test automation of CBS based on their graphical user interfaces (GUI), because for any level of user-focused testing domain knowledge and knowledge about the implementation of the CBS are essential to run the tests. The component manufacturer, on the other side, is usually not willing to deliver the code to protect his, or her, commercial interest. For solving this conflict, this paper introduces a framework for the automation of user-oriented component testing that significantly reduces the test costs. The concept is based on black-box testing techniques and utilizes the common features of commercial capture/replay test tools.