Obstacle detection is crucial to the operation of autonomous driving systems, which rely on multiple sensors, such as cameras and LiDARs, combined with code logic and deep learning models to detect obstacles for time-sensitive decisions. Consequently, obstacle detection latency is critical to the safety and effectiveness of autonomous driving systems. However, the latency of the obstacle detection module and its resilience to various changes in the LiDAR point cloud data are not yet fully understood. In this work, we present the first comprehensive investigation on measuring and modeling the performance of the obstacle detection modules in two industry-grade autonomous driving systems, i.e., Apollo and Autoware. Learning from this investigation, we introduce ADPerf, a tool that aims to generate realistic point cloud data test cases that can expose increased detection latency. Increasing latency decreases the availability of the detected obstacles and stresses the capabilities of subsequent modules in autonomous driving systems, i.e., the modules may be negatively impacted by the increased latency in obstacle detection.We applied ADPerf to stress-test the performance of widely used 3D obstacle detection modules in autonomous driving systems, as well as the propagation of such tests on trajectory prediction modules. Our evaluation highlights the need to conduct performance testing of obstacle detection components, especially 3D obstacle detection, as they can be a major bottleneck to increased latency of the autonomous driving system. Such an adverse outcome will also further propagate to other modules, reducing the overall reliability of autonomous driving systems.

Autonomous driving systems (ADSs) rely on real-time sensor data, such as cameras and LiDARs, for time-critical decisions using deep neural networks. The accuracy of these decisions is crucial for the widespread adoption of ADSs, as errors can have serious consequences. 3D obstacle detection, in particular, is sensitive to point cloud data (PCD) noise from various sources. However, the robustness of current 3D obstacle detection models against specification-based perturbations remains unevaluated. These perturbations are derived from the specification of LiDAR sensors and previous research on LiDAR’s ability to capture objects of different colors and materials. They can manifest as very subtle sensor-based noises or obstacle-specific perturbations. Hence, we propose SORBET, a framework that tests the robustness of 3D obstacle detection models in ADS against such perturbations to the PCD to evaluate their robustness. We applied SORBET to evaluate the robustness of five classic 3D obstacle detection models, including one from an industry-grade Level 4 ADS (Baidu’s Apollo). Furthermore, we studied how the deviated obstacle detection results would propagate and negatively impact trajectory prediction. Our evaluation emphasizes the importance of testing 3D obstacle detection against specification-based perturbations. We find that even very subtle changes in the PCD (i.e., removing two points) may introduce a non-trivial decrease in the detection performance. Furthermore, such a negative impact will further propagate to other modules and endanger the safety of the ADS.