Vehicle and traffic simulation is a common practice for testing and evaluating advanced driver-assistance systems (ADAS) and autonomous driving (AD). As a result, the literature mentions numerous simulators capable of simulating ADAS/AD implementations. In this study, we investigate previous surveys that cover multiple scenarios and initiate a systematic review targeting simulators for testing ADAS/AD. Our results show that the literature mentions, in total, 181 simulators capable of evaluating one or more ADAS/AD implementations. Furthermore, according to previous surveys and reviews, the most popular simulators are CARLA, Airsim, and SUMO. Finally, our results uncover that every five years, the number of novel simulators added to the literature grows at least quadratically, showing that further review is necessary to address the differences between these simulators and understand the simulator landscape from an ADAS/AD testing perspective.

Summary With the ever‐growing Android graphical user interface (GUI) application market, there have been many studies on automated test generation for Android GUI applications. These studies successfully demonstrate how to detect fatal exceptions and achieve high coverage with fully automated test generation engines. However, it is unclear how many GUI functions these engines manage to test. The current best practice for the functional testing of Android GUI applications is to design user interface (UI) test scenarios with a non‐technical and human‐readable language such as Gherkin and implement Java/Kotlin methods for every statement of all the UI test scenarios. Writing tests for UI test scenarios is hard, especially when some scenario statements are high‐level and declarative, so it is not clear what actions should the generated test perform. We propose the Fully Automated Reinforcement LEArning‐Driven specification‐based test generator for Android (FARLEAD‐Android). FARLEAD‐Android first translates the UI test scenario to a GUI‐level formal specification as a linear‐time temporal logic (LTL) formula. The LTL formula guides the test generation and acts as a specified test oracle. By dynamically executing the application under test (AUT), and monitoring the LTL formula, FARLEAD‐Android learns how to produce a witness for the UI test scenario, using reinforcement learning (RL). Our evaluation shows that FARLEAD‐Android is more effective and achieves higher performance in generating tests for UI test scenarios than three known engines: Random, Monkey and QBEa. To the best of our knowledge, FARLEAD‐Android is the first fully automated mobile GUI testing engine that uses formal specifications.

In system-on-chip design, resources for verification is limited by time-to-market and cost. In order to allocate verification resources effectively, managers need to rely on their experience backed by design related metrics. However, often there are also other aspects of development process, such as bug history and developer information that can improve the effectiveness of verification. Software bug prediction is a machine learning (ML)-based technique which predicts whether a given software module is bug-prone by using product and process metrics of the module. Therefore, it can help direct verification effort, reduce costs, and improve the quality of software. Although there is a plethora of work in software bug prediction, no such work exists for SystemC. We propose an ML-based software bug prediction solution for verification of SystemC models used in virtual prototypes that takes into account system level design metrics and demonstrate its effectiveness on several open source system level designs. We find that 96% of modules could be correctly predicted as buggy or clean.

GUI testing of mobile applications gradually became a very important topic in the last decade with the growing mobile application market. We propose Test Case Mutation (TCM) which mutates existing test cases to produce richer test cases. These mutated test cases detect crashes that are not previously detected by existing test cases. TCM differs from the well-known Mutation Testing (MT) where mutations are inserted in the source code of an Application Under Test (AUT) to measure the quality of test cases. Whereas in TCM, we modify existing test cases and obtain new ones to increase the number of detected crashes. Android applications take the largest portion of the mobile application market. Hence, we evaluate TCM on Android by replaying mutated test cases of randomly selected $$100$$ AUTs from F-Droid benchmarks. We show that TCM is effective at detecting new crashes in a given time budget.

Android applications are used extensively around the world. Many of these applications contain potential crashes. Black-box testing of Android applications has been studied over the last decade to detect these crashes. In this paper, we propose QLearning-Based Exploration (QBE), a fully automated black-box testing methodology, which explores GUI actions using a well-known reinforcement learning technique called QLearning. QBE performs automata learning to obtain a model of the AUT, and generates replayable test suites. Specifically, QBE learns from a set of existing applications the kinds of actions that are most useful in order to reach a particular objective such as detecting crashes or increasing activity coverage. To the best of our knowledge, ours is the first machine learning based approach in Android GUI Testing. We conduct experiments on a test set of 100 AUTs obtained from the commonly used F-Droid benchmarks to show the effectiveness of QBE. We show that QBE performs better than all compared black-box tools in terms of activity coverage and number of distinct detected crashes. We make QBE and our experimental data available online.