Vehicles controlled by autonomous driving software (ADS) are expected to bring many social and economic benefits, but at the current stage not being broadly used due to concerns with regard to their safety. Virtual tests, where autonomous vehicles are tested in software simulation, are common practices because they are more efficient and safer compared to field operational tests. Specifically, search-based approaches are used to find particularly critical situations. These approaches provide an opportunity to automatically generate tests; however, system-atically producing bug-revealing tests for ADS remains a major challenge. To address this challenge, we introduce DoppelTest, a test generation approach for ADSes that utilizes a genetic algorithm to discover bug-revealing violations by generating scenarios with multiple autonomous vehicles that account for traffic control (e.g., traffic signals and stop signs). Our extensive evaluation shows that DoppelTest can efficiently discover 123 bug-revealing violations for a production-grade ADS (Baidu Apollo) which we then classify into 8 unique bug categories.

Autonomous Vehicles (AVs) leverage advanced sensing and networking technologies (e.g., camera, LiDAR, RADAR, GPS, DSRC, 5G, etc.) to enable safe and efficient driving without human drivers. Although still in its infancy, AV technology is becoming increasingly common and could radically transform our transportation system and by extension, our economy and society. As a result, there is tremendous global enthusiasm for research, development, and deployment of AVs, e.g., self-driving taxis and trucks from Waymo and Baidu. The current practice for testing AVs uses virtual tests—where AVs are tested in software simulations—since they offer a more efficient and safer alternative compared to field operational tests. Specifically, search-based approaches are used to find particularly critical situations. These approaches provide an opportunity to automatically generate tests; however, systematically creatingvalidandeffectivetests for AV software remains a major challenge. To address this challenge, we introducescenoRITA, a test generation approach for AVs that uses an evolutionary algorithm with (1) a novel gene representation that allows obstacles to befully mutable, hence, resulting in more reported violations and more diverse scenarios, (2) 5 test oracles to determine both safety and motion sickness-inducing violations and (3) a novel technique to identify and eliminate duplicate tests. Our extensive evaluation shows thatscenoRITAcan produce test scenarios that are more effective in revealing ADS bugs and more diverse in covering different parts of the map compared to other state-of-the-art test generation approaches.

The problem of performance ranking in configurable systems is to find the optimal (near-optimal) configurations with the best performance. This problem is challenging due to the large search space of potential configurations and the cost of manually examining configurations. Existing methods, such as the rank-based method, use a progressive strategy to sample configurations to reduce the cost of examining configurations. This sampling strategy is guided by frequent and random trials and may fail in balancing the number of samples and the ranking difference (i.e., the minimum of actual ranks in the predicted ranking). In this paper, we proposed a sampling method, namely MoConfig, which uses multi-objective optimization to minimize the number of samples and the ranking difference. Each solution in MoConfig is a sampling set of configurations and can be directly used as the input of existing methods of performance ranking. We conducted experiments on 20 datasets from real-world configurable systems. Experimental results demonstrate that MoConfig can sample fewer configurations and rank better than the existing rank-based method. We also compared the results by four algorithms of multi-objective optimization and found that NSGA-II performs well. Our proposed method can be used to improve the ranking difference and reduce the number of samples in building predictive models of performance ranking.