Sensor fusion algorithms combine data from multiple sensors to produce more accurate and reliable results. However, temporal misalignment between sensors, caused by factors such as clock drift, jitter or networking delays, can significantly degrade fusion quality. Prior work on modeling temporal misalignment in sensor fusion algorithms assumes that in the ideal case all samples should be aligned with the same reference time point. We show that this assumption limits its applicability when samples are intentionally taken at different time points, e.g., when a single sensor is sampled multiple times or when sensors operate at different frequencies. In this paper, we introduce alignment sets, which allow system designers to explicitly specify the intended alignment between samples. This flexibility enables more precise temporal misalignment measures that better reflect the actual requirements of sensor fusion scenarios. We prove that alignment sets generalize the prior definitions of temporal misalignment of sensor fusion algorithms. We also provide an evaluation on a camera-LiDAR fusion pipeline for 3D object detection, showing that alignment sets provide more accurate misalignment measures and robustness estimates.

We present a post-hoc approach for scenario-based testing of automated driving systems, enabling the analysis of safety and correctness for (cooperative) automated driving systems in many scenarios without conducting tests for individual scenarios. The system under test is operated in its physical environment’ and data is recorded during operation. Then, driving scenarios are identified in this data and functional requirements are checked, yielding pass or fail verdicts for individual scenarios. We validate the envisioned post-hoc approach in a single-case mechanism experiment by the example of a platooning controller, identifying a previously unknown bug in the tested system, as well as a functional insufficiency concerning the intended operational design domain.

The Robot Operating System 2 (ROS 2) is a widely used middleware that provides software libraries and tools for developing robotic systems. In these systems, tasks are scheduled by ROS 2 executors. Since the scheduling behavior of the default ROS 2 executor is inherently different from classical real-time scheduling theory, dedicated analyses or alternative executors requiring substantial changes to ROS 2 have been developed. In 2023, the events executor was introduced into ROS 2. It features an events queue and allows the possibility to make scheduling decisions immediately after a job is completed. In this paper, we show that with minor modifications of the events executor, a large body of research results from classical real-time scheduling theory becomes directly applicable to ROS 2. This enables analytical bounds on the worst-case response time and the end-to-end latency, outperforming bounds for the default ROS 2 executor in many scenarios. Our solution is easy to integrate into existing ROS 2 systems since it requires only minor modifications of the events executor, which is natively included in ROS 2. The evaluation results show that our ROS 2 events executor with minor modifications can have significant improvement in terms of dropped jobs, worst-case response time, end-to-end latency, and performance compared to the default ROS 2 executor.

Abstract We propose two complementary research directions, “Time for ML” and “ML for Time”, that we believe to be critical for the deployment of machine-learning (ML) applications in time-sensitive applications. “Time for ML” refers to ML systems that are aware of and can adapt to dynamic time constraints regarding their execution, while “ML for Time” refers to ML systems that are aware of and can deal with data’s temporal aspects, such as misalignment. We believe these two directions are complementary and can be combined to provide more robust and reliable machine learning systems.

In many applications of cyber-physical systems, a sequence of tasks is necessary to perform a certain functionality. For example, from a sensor to an actuator, the first task reads the sensor value (cause), the second task processes the data, and the third task produces an output for the actuator (an effect is triggered). For such scenarios, the end-to-end timing properties (the so-called end-to-end latency) of the sequence of tasks (the so-called cause-effect chain) are of importance. This tutorial recaps different metrics for the end-to-end latency of cause-effect chains, and summarizes fundamental properties and existing analytical results in a systematic manner. To that end, this tutorial has a special focus on the reaction time (how fast can a reaction be in the worst case) and the data age (how old is the data source of an actuation in the worst case). The goal of this tutorial is to provide a systematic view of the fundamental end-to-end timing properties of cause-effect chains and offer an outlook of possible research directions in the near future. Furthermore, we extend the proof of one fundamental property in the literature to comply with the current state-of-the-art definition of end-to-end latencies.

Sensor fusion is the process of combining data from multiple sensors for acquiring a more accurate and comprehensive understanding of the observed environment. However, temporal misalignments between sensors can lead to incorrect fusion results, while the temporal robustness of sensor fusion algorithms is still a relatively unexplored research topic. To address this gap, we define three types of temporal robustness for sensor fusion: reference-point-based, strong sample-point-based, and weak sample-point-based temporal robustness. These definitions provide a framework to quantitatively evaluate the temporal robustness of sensor fusion functions. We also investigate the case where only a part of the sensors are misaligned. Furthermore, we consider potential probabilistic aspects for the proposed definitions. We assess the temporal robustness of a state-of-the-art fusion method in the context of 3D object detection, where camera and LiDAR data are fused. Our empirical evaluation shows that the examined fusion methods exhibit moderate robustness against temporal misalignment of images, but are especially sensitive to LiDAR misalignment. Our findings call attention to the necessity of providing robustness guarantees for sensor fusion functions against temporal misalignment.

Modern robot systems, like autonomous vehicles, are complex, distributed systems that consist of many interacting components. End-to-end timing latency guarantees are key properties of such systems. They upper bound the data processing time and provide a predictable timing behavior. The Robot Operating System 2 (ROS 2) is a widely used and highly configurable set of software libraries for creating and deploying robot systems. It features a custom scheduler to execute time-triggered and event-triggered tasks and uses Data Distribution Services (DDS) for the communication between different system components. The data propagations between ROS 2 system components form cause-effect chains, which can be analyzed to determine the maximum reaction time (longest time between occurrence of an external cause and the earliest time when this external cause is fully processed) and maximum data age (longest time between the moment of a sensor measurement and the latest moment where an effect is based on this sensor measurement). In this paper, we provide an analysis of the end-to-end latencies in multi-executor ROS 2 systems to upper bound the end-to-end latencies of cause-effect chains in ROS 2 systems. Furthermore, we introduce an optimization using constrained programming that determines the optimal system configuration to minimize the end-to-end latencies for ROS 2 systems. We evaluate our upper-bound analysis to determine the end-to-end latencies of cause-effect chains in an autonomous driving-software stack for oval racing used in the Indy Autonomous Challenge and apply our optimization method to reduce the end-to-end latency upper bound, measured maximum, and measured mean by up to 50.2 %, 19.8 %, and 7.2 %, respectively.

The robot operating system 2 (ROS 2) is a widely used collection of tools and libraries for building robot applications. It is designed to be flexible and easy to use when creating complex robot systems with many interacting components.Since its alpha version release in 2015, ROS 2 provides two options in a multithreading operating system, namely the single-threaded executor and the multithreaded executor. The single-threaded executor is starvation-free by design (i.e., every task is eventually executed) even in over-utilized systems, since the set of eligible task instances (called wait set) is only refilled once all the task instances in the wait set are executed. The multithreaded executor extends this mechanism to multiple threads that manage the wait set collaboratively. While intuitively this extension preserves the starvation-free property, and analyses for the multithreaded executor even build upon this assumption, the multithreaded executor has not been shown to be starvation-free.In this work, we examine the mechanism of the multithreaded executor in ROS 2 and demonstrate that it is prone to starvation, i.e., some tasks may never be executed even in under-utilized systems. This indicates risks for multithreaded executors in the current ROS 2 design and further leads to counterexamples to the state-of-the-art response-time analyses by Jiang et al. (RTSS 2022) and Sobhani et al. (RTAS 2023). We propose a minimal change in the software architecture of the ROS 2 multithreaded executor to enable starvation- and deadlock-free behavior. We empirically test that we prevent starvation in concrete ROS 2 system configurations, and show that our solution incurs a negligible overhead using the autoware reference benchmark. Moreover, we prove that our solution is starvation- and deadlock-free using formal proofs and model checking.

Violations of real-time properties and high latencies have emerged as crucial issues in autonomous vehicles since they can lead to unwanted vehicle behavior and critical maneuvers. Our study aims to provide a comprehensive understanding of latencies in a software stack for autonomous vehicles. In this paper, we present an evaluation workflow to inspect software and the occurring latencies for ROS 2 applications. This workflow was used to analyze the open-source autonomous driving stack Autoware. Universe by showing the influence of different soft- and hardware configurations. Our focus is on the evaluation of end-to-end, communication, computation, and idle latencies. Based on the results, we show the bottlenecks and motivate future directions to optimize ROS 2 autonomous driving software.

ROS 2 is a framework consisting of software libraries for developing robot systems, such as autonomous driving systems, that consist of multiple interacting components. In ROS 2, each component is implemented as a node, which contains time-triggered and event-triggered tasks. These tasks communicate with each other via ROS 2 topics or shared memory, and are scheduled by a ROS 2 executor. In ROS 2 systems, the system configuration and callback execution can have a significant impact on system performance, including end-to-end latencies, message loss, and memory usage. In this paper, we provide a bound on the timer period of ROS 2 timers to prevent sensor undersampling, and a subscription buffer size limit to prevent message loss and minimize memory usage. Furthermore, we explain the occurrence of message loss and high end-to-end latencies in ROS 2 systems, which are caused by the system configuration and subscription buffer size choice. Based on our observations, we propose a callback-prioritization heuristic to reduce end-to-end latencies and subscription buffer sizes. We demonstrate our findings using case studies based on Autoware.Universe and provide further evaluation to highlight the benefits of our heuristic.

Modern autonomous vehicle platforms feature many interacting components and sensors, which add to the system complexity and affect their performance. A key aspect for such platforms are end-to-end timing guarantees, which are required for safe and predictable behavior in every situation. One widely used tool to develop such autonomous systems is the Robot Operating System 2 (ROS2), which allows creating robot applications composed of several components that communicate with each other to form complex systems. Furthermore, it guarantees real-time constraints and provides reliable timing behavior using a custom scheduler design that manages the execution of all components. These components and their data propagation form multiple cause-effect chains that can be analyzed to determine two key metrics: maximum reaction time (which is the maximum time for the system to react to an external input) and maximum data age (which equals the maximum time between sampling and the output of the system being based on that sample). However, an end-to-end analysis for cause-effect chains in ROS2 systems has not been provided yet. In this paper, we provide a theoretical upper bound for the end-to-end timing of a ROS2 system on a single electronic control unit (ECU). Additionally, we show how to simulate a ROS2 system to get a lower bound for the timing analysis and introduce an online end-to-end timing measurement method for existing ROS2 systems. We evaluate our methods with a basic autonomous navigation system and determine the timing behavior for different components and sensor configurations.

Numerical simulations often play an important role when evaluating and comparing the performance of schedulability tests, as they allow to empirically demonstrate their applicability using synthesized task sets under various configurations. In order to provide a fair comparison of various schedulability tests, von der Brüggen et al. presented the first version of an evaluation framework for self-suspending task sets. In this work-in-progress, we further enhance the framework by providing more features to ease the use, e.g., Python 3 support, an improved GUI, multiprocessing, Gurobi optimization, and external task evaluation. In addition, we integrate the state-of-the-arts we are aware of into the framework. Moreover, the documentation is improved significantly to simplify the application in further research and development. To the best of our knowledge, the framework contains all suspension-aware schedulability tests for uniprocessor systems and we aim to keep it up-to-date.