In the foreseeable future, highly automated mobile systems, such as vehicles, robots, UAVs, or trains, will be confronted with difficult situations that require external support. The availability of such external support corresponds to level 4 driving automation and is an essential feature in current robotaxis and automated public transportation. While the first generation of level 4 prototypes relied on safety driver support, commercial systems are gradually moving towards support by teleoperation. Designing teleoperation support for level 4 systems is an end-to-end problem involving two main research and practical challenges, the teleoperation function defining the remote human interface with its scene representation and available control functions, and the real-time communication channel involving wired and wireless segments, which must provide reliable end-to-end data transport. Both challenges are tightly linked, and combined solutions are needed to reach the required safe teleoperation. Solutions can make use of the rich sensing and control system of a level 4 vehicle, which however can only be exploited if the communication channel provides adequate real-time access.

Misperception of surrounding objects and traffic participants can lead to a critical situation. Autonomous Driving systems must be able to assess the safety impact of a degraded sensing and perception pipeline at any time. This assessment should be based on an independent risk evaluation framework. Introduced in this work is a residual risk that quantifies the potential risk originating from misperception compared to a response with non-degraded information. Evaluations are possible online at an average rate of 10Hz. Further, exemplary scenarios are analyzed and provided in this paper for discussion.

Teleoperation is a key enabler for future mobility, supporting Automated Vehicles in rare and complex scenarios beyond the capabilities of their automation. Despite ongoing research, no open source software currently combines Remote Driving, e.g., via steering wheel and pedals, Remote Assistance through high-level interaction with automated driving software modules, and integration with a real-world vehicle for practical testing. To address this gap, we present a modular, open source teleoperation software stack that can interact with an automated driving software, e.g., Autoware, enabling Remote Assistance and Remote Driving. The software features standardized interfaces for seamless integration with various real-world and simulation platforms, while allowing for flexible design of the human-machine interface. The system is designed for modularity and ease of extension, serving as a foundation for collaborative development on individual software components as well as realistic testing and user studies. To demonstrate the applicability of our software, we evaluated the latency and performance of different vehicle platforms in simulation and real-world. The source code is available on GitHub11https://github.com/TUMFTM/teleoperated_driving/tree/ros2.

Teleoperation enables remote human support of automated vehicles in scenarios where the automation is not able to find an appropriate solution. Remote assistance concepts, where operators provide discrete inputs to aid specific automation modules like planning, is gaining interest due to its reduced workload on the human remote operator and improved safety. However, these concepts are challenging to implement and maintain due to their deep integration and interaction with the automated driving system. In this paper, we propose a solution to facilitate the implementation of remote assistance concepts that intervene on planning level and extend the operational design domain of the vehicle at runtime. Using arbitration graphs, a modular decision-making framework, we integrate remote assistance into an existing automated driving system without modifying the original software components. Our simulative implementation demonstrates this approach in two use cases, allowing operators to adjust planner constraints and enable trajectory generation beyond nominal operational design domains.

Implementing a teleoperation system with its various actors and interactions is challenging and requires an overview of the necessary functions. This work collects all tasks that arise in a control center for an automated vehicle fleet from literature and assigns them to the two roles Remote Operator and Fleet Manager. Focusing on the driving-related tasks of the remote operator, a process is derived that contains the sequence of tasks, associated vehicle states, and transitions between the states. The resulting state diagram shows all remote operator actions available to effectively resolve automated vehicle disengagements. Thus, the state diagram can be applied to existing legislation or modified based on prohibitions of specific interactions. The developed control center framework and included state diagram should serve as a basis for implementing and testing remote support for automated vehicles to be validated on public roads.

Teleoperation is emerging as a crucial fallback for automated vehicles when facing unforeseen edge cases that exceed their capabilities. This work focuses on the remote assistance concept Perception Modification, where a remote operator resolves perception-related issues (e.g., neglecting false-positive detections) by modifying the vehicle’s environmental model without assuming full control. Effective perceptual data visualization is essential, enabling the remote operator to achieve sufficient situational awareness to identify perception-related issues while maintaining an acceptable mental workload. However, for Perception Modification it remains unclear which perceptual data should be visualized and how. This research implements and evaluates three visualization approaches tailored to Perception Modification. The conducted online user study shows a preference for integrating perceptual data into a single view, though further development is needed for real-world deployment.

Teleoperation serves as the fallback solution to autonomous driving but reliable functions of the teleoperation require a certain amount of mobile network resources, which cannot be guaranteed at all times. Therefore, predictive quality of service (pQoS) is introduced as a concept to increase the resilience of the teleoperation. In this paper, based on a data measurement campaign, we propose a prediction framework to prediction two important network KPIs of teleoperation: uplink data-rate and round-trip latency. Furthermore, we introduce a method to alleviate the performance degradation of machine-learning-based prediction models on previously unseen data due to concept drift by incorporating historic data into the prediction pipeline. Additionally, we introduce the metric of critical scenario detection to evaluate the prediction performance specifically for teleoperation.

Extensive testing is necessary to ensure the safety of autonomous driving modules. In addition to component tests, the safety assessment of individual modules also requires a holistic view at system level, which can be carried out efficiently with the help of simulation. Achieving seamless compatibility between a modular software stack and simulation is complex and poses a significant challenge for many researchers. To ensure testing at the system level with state-of-the-art AV software and simulation software, we have developed and analyzed a bridge connecting the CARLA simulator with the AV software Autoware Core/Universe. This publicly available bridge enables researchers to easily test their modules within the overall software. Our investigations show that an efficient and reliable communication system has been established. We provide the simulation bridge as open-source software at https://github.com/TUMFTM/Carla-Autoware-Bridge.

Despite the rapid technological progress, autonomous vehicles still face a wide range of complex driving situations that require human intervention. Teleoperation technology offers a versatile and effective way to address these challenges. The following work puts existing ideas into a modern context and introduces a novel technical implementation of the trajectory guidance teleoperation concept. The presented system was developed within a high-fidelity simulation environment and experimentally validated, demonstrating a realistic ride-hailing mission with prototype autonomous vehicles and onboard passengers. The results indicate that the proposed concept can be a viable alternative to the existing remote driving options, offering a promising way to enhance teleoperation technology and improve overall operation safety.

Since Automated Driving Systems are not expected to operate flawlessly, Automated Vehicles will require human assistance in certain situations. For this reason, teleoperation offers the opportunity for a human to be remotely connected to the vehicle and assist it. The Remote Operator can provide extensive support by directly controlling the vehicle, eliminating the need for Automated Driving functions. However, due to the physical disconnection to the vehicle, monitoring and controlling is challenging compared to driving in the vehicle. Therefore, this work follows the approach of simplifying the task for the Remote Operator by separating the path and velocity input. In a study using a miniature vehicle, different operator-vehicle interactions and input devices were compared based on collisions, task completion time, usability and workload. The evaluation revealed significant differences between the three implemented prototypes using a steering wheel, mouse and keyboard or a touchscreen. The separate input of path and velocity via mouse and keyboard or touchscreen is preferred but is slower compared to parallel input via steering wheel.

Teleoperation technology aims to support auto-mated vehicles in situations where no solution to the present scenario can be found. In these situations, decision making is handled by a remote human operator. To overcome safety impairments caused by latencies in data transmission and reduced situational awareness of the remote operator, this work proposes a shared control framework that assists the operator. To enable teleoperation in dynamic environments, risk assessment and prediction methods are integrated into the framework. As a proof of concept, two system implementations are shown. One implementation uses the time-to-collision metric to enhance teleoperation safety in presence of dynamic objects. The second implementation integrates an open-source occlusion awareness module that allows to consider risk arising from traffic participants potentially emerging from occluded areas. The risk and criticality of situations with dynamic objects are reduced by both implementations.

Online corner case detection is crucial for ensuring safety in autonomous driving vehicles. Current autonomous driving approaches can be categorized into modular approaches and end-to-end approaches. To leverage the advantages of both, we propose a method for online corner case detection that integrates an end-to-end approach into a modular system. The modular system takes over the primary driving task and the end-to-end network runs in parallel as a secondary one, the disagreement between the systems is then used for corner case detection. We implement this method on a real vehicle and evaluate it qualitatively. Our results demonstrate that end-to-end networks, known for their superior situational awareness, as secondary driving systems, can effectively contribute to corner case detection. These findings suggest that such an approach holds potential for enhancing the safety of autonomous vehicles.

The development of autonomous and remote-operated driving systems requires extensive stakeholder analyses, requirement engineering, and formalized system descriptions. This is necessary to guarantee the success of the final product after the expensive and time-consuming development phase. To integrate a formalized description of the required abilites of the system, ability graphs have been proposed in the literature. Up to this date, however, this ability graph has only been used to model less complicated driver assistance systems in the literature. This work aims to introduce the value of an ability graph-based description of complex driving systems. This is achieved by successfully demonstrating and discussing a method for constructing a holistic ability graph capable of describing the entirety of abilities required for any driving system.

Driverless road maintenance could potentially be highly beneficial to all its stakeholders, with the key goals being increased safety for all road participants, more efficient traffic management, and reduced road maintenance costs such that the standard of the road infrastructure is sufficient for it to be used in Automated Driving (AD). This paper addresses how the current state of the technology could be expanded to reach those goals. Within the project ‘System for Teleoperated Road-marking’ (SToRM), using the road-marking machine as the system, different operation modes based on teleoperation were discussed and developed. Furthermore, a functional system overview considering both hardware and software elements was experimentally validated with an actual road-marking machine and should serve as a baseline for future efforts in this and similar areas.

In parallel with the advancement of Automated Driving (AD) functions, teleoperation has grown in popularity over recent years. By enabling remote operation of automated vehicles, teleoperation can be established as a reliable fallback solution for operational design domain limits and edge cases of AD functions. Over the years, a variety of different teleoperation concepts as to how a human operator can remotely support or substitute an AD function have been proposed in the literature. This paper presents the results of a literature survey on teleoperation concepts for road vehicles. Furthermore, due to the increasing interest within the industry, insights on patents and overall company activities in the field of teleoperation are presented.

In this paper, a steering action-aware Adaptive Cruise Control (ACC) approach for teleoperated road vehicles is proposed. In order to keep the vehicle in a safe state, the ACC approach can override the human operator’s velocity control commands. The safe state is defined as a state from which the vehicle can be stopped safely, no matter which steering actions are applied by the operator. This is achieved by first sampling various potential future trajectories. In a second stage, assuming the trajectory with the highest risk, a safe and comfortable velocity profile is optimized. This yields a safe velocity control command for the vehicle. In simulations, the characteristics of the approach are compared to a Model Predictive Control-based approach that is capable of overriding both, the commanded steering angle as well as the velocity. Furthermore, in teleoperation experiments with a 1:10-scale vehicle testbed, it is demonstrated that the proposed ACC approach keeps the vehicle safe, even if the control commands from the operator would have resulted in a collision.

Truck overtaking maneuvers on freeways are inefficient, risky and promote high potential for conflict between road users. Collective perception based on V2X communication allow coordination with all parties to reduce the negative impact and could be installed in a timely manner compared to automation. However, the prerequisite for the success of this system is a human-machine interface that the driver can easily operate, trusts and accepts. In this approach, a user-centered conception and design of a human-machine interface for cooperative truck overtaking maneuver on freeways is presented. The development process is separated in two steps: After a prototype is build based on task analysis it is initially evaluated and improved iteratively with a heuristic evaluation by experts. The final prototype is tested in a simulator study with 30 truck drivers. The study provides initial feedback regarding the drivers' attitudes towards such a system and how it can be further improved.

Personal assistants like Alexa, Siri, and co. persuade users to treat technology like a friend. Studies indicate that humanized features like voice, the display of emotions and empathy help people to have more trust and pleasure and feel companionship when interacting with the technology. The professional truck driver has various challenges such as being isolated for several hours while performing a monotonous driving task, which is exacerbated by having to monitor increasing automation, growing pressure due to increasing demand, and just-in-time delivery, we believe that humanized human-machine interaction can improve the working situation of truck drivers. Therefore, we created a socially interactive device called ICo (the Intelligent Co-driver), which supports and accompanies the driver while driving. In a driving simulator study with 34 professional drivers, we investigated within a Wizard of Oz setup to what extent the addition of human characteristics through mimicry and gesture changes the driving performance, user experience, and technology acceptance. No differences could be shown in the subjective data, but the objective data captured in the study suggests that humanization affects driving performance and driver engagement.

With on-board sensor technology, the environment can only be perceived to a limited extent. This can lead to energy-inefficient driving maneuvers due to the late perception of objects. The fuel consumption of heavy trucks is a major cost factor for transport companies, which is why energy-efficient systems are being sought. With collective perception, perceived objects are exchanged via Vehicle-to-Everything (V2X) and merged to a common environment model. Therefore, it is possible to achieve a greater awareness, which allows for improved planning for automated vehicles. In this publication, a system with collective perception and energy-efficient maneuver planning is presented. The functioning of the collective perception is presented using real vehicle data. A vehicle simulation shows the positive effect of collective perception in combination with an energy-efficient maneuver planner for determining the fuel consumption of heavy trucks.

Teleoperation is becoming an essential feature in automated vehicle concepts, as it will help the industry overcome challenges facing automated vehicles today. Teleoperation follows the idea to get humans back into the loop for certain rare situations the automated vehicle cannot resolve. Teleoperation therefore has the potential to expand the operational design domain and increase the availability of automated vehicles. This is especially relevant for concepts with no backup driver inside the vehicle. While teleoperation resolves certain issues an automated vehicle will face, it introduces new challenges in terms of safety requirements. While safety and regulatory approval is a major research topic in the area of automated vehicles, it is rarely discussed in the context of teleoperated road vehicles. The focus of this paper is to systematically analyze the potential hazards of teleoperation systems. An appropriate hazard analysis method (STPA) is chosen from literature and applied to the system at hand. The hazard analysis is an essential part in developing a safety concept (e.g., according to ISO26262) and thus far has not been discussed for teleoperated road vehicles.

This paper demonstrates how a cooperative truck overtaking maneuver can be coordinated and synchronized via V2X. This is relevant because the classical truck overtaking maneuver imposes high stress on truck drivers, which can lead to work absences or accidents. We define which abstract/atomic tasks are involved in the truck overtaking maneuver and assign them to a distributed state machine. With the help of a V2X message we then synchronize this state machine and exchange all information relevant for the overtaking maneuver. The simulation of 600 overtaking scenarios demonstrates that the developed concept is adequate and that a transmission frequency of 5 Hz offers the best trade-off between channel load and maneuver quality.

Due to the challenges of autonomous driving, backup options like teleoperation become a relevant solution for critical scenarios an automated vehicle might face. To enable teleoperated systems, two main problems have to be solved: Safely controlling the vehicle under latency, and presenting the sensor data from the vehicle to the operator in such a way, that the operator can easily understand the vehicles environment and the vehicles current state. While most of the teleoperation systems face similar challenges, the teleoperation of automated vehicles is unique in its scale, safety requirements and system constraints. Two major constraints are the round-trip-latency and the maximum upload-bandwidth. While the latency mainly influences the controllability and safety of the vehicle, the upload-bandwidth affects the amount of transmittable sensor data and therefore operators situation awareness, as well as the running costs of the whole system. The focus of this paper is measuring and reducing the end-to-end latency for a teleoperation setup. Therefore the latency is separated into actuator and sensor latency. For each part the different components and settings are analyzed in order to find a realistic minimal end-to-end latency for the teleoperation of automated vehicles. Therefore new measurement methods are developed and existing methods adapted.

The vision of connected driving is a driving force in current research within automotive engineering. Trucks in particular, with their limitations due to mass and size, benefit from the cooperation of other road users. In order to be able to agree on maneuvers, however, communication between the involved vehicles must be established. This paper investigates in real tests where the antennas are best placed on the truck for vehicle to vehicle communication with ITS-G5 and what ranges are to be expected. Furthermore, the influence of the truck trailer with different materials and bending angles on the signal propagation is analyzed.

For the offline safety assessment of automated vehicles, the most challenging and critical scenarios must be identified efficiently. Therefore, we present a new approach to define challenging scenarios based on a sensor setup model of the ego-vehicle. First, a static optimal approaching path of a road user to the ego-vehicle is calculated using an A* algorithm. We consider a poor perception of the road user by the automated vehicle as optimal, because we want to define scenarios that are as critical as possible. The path is then transferred to a dynamic scenario, where the trajectory of the road user and the road layout are determined. The result is an optimal road geometry, so that the ego-vehicle can perceive an approaching object as poorly as possible. The focus of our work is on the highway as the Operational Design Domain (ODD).

Due to the challenges of autonomous driving, for the near future, automated vehicles will not be able to drive in all conditions without any human intervention. The challenge arises when no human driver is inside the vehicle to resolve the challenging situation. One solution for this might be teleoperation, here a remote operator takes control over the car and resolves the situation from a distance. But teleoperation technology itself comes with certain challenges, one of them being creating a good situational awareness at the operator site based on the sensor data transmitted from the automated vehicle. To understand this challenge better, in this paper a five-week long-time study is conducted with the goal of measuring the impact of different displays, video-canvases and -streaming quality on situation awareness, workload and decision making. The objective results show a significant impact of video streaming quality on various factors of situation awareness. On the other hand the subjective results such as workload, immersion, usability and presence indicate that video streaming quality only has an impact in situations with high contrasts and over all scenes the impact of video quality on subjective satisfaction is not significant. Between the three display modes no significance regarding quality was found. However, the participants preferred the head mounted display over the other options even though the results indicate that the head mounted display is most sensitive to changes in video streaming quality.

The objective of this paper is to propose a systematic analysis of the sensor coverage of automated vehicles. Due to an unlimited number of possible traffic situations, a selection of scenarios to be tested must be applied in the safety assessment of automated vehicles. This paper describes how phenomenological sensor models can be used to identify system-specific relevant scenarios. In automated driving, the following sensors are predominantly used: camera, ultrasonic, Radar and Lidar. Based on the literature, phenomenological models have been developed for the four sensor types, which take into account phenomena such as environmental influences, sensor properties and the type of object to be detected. These phenomenological models have a significantly higher reliability than simple ideal sensor models and require lower computing costs than realistic physical sensor models, which represents an optimal compromise for systematic investigations of sensor coverage. The simulations showed significant differences between different system configurations and thus support the system-specific selection of relevant scenarios for the safety assessment of automated vehicles.

Due to the challenges of autonomous driving backup options like teleoperation become a relevant solution for critical scenarios an automated vehicle might face. To enable teleoperated systems two main problems have to be solved: Safely controlling the vehicle under latency, and presenting the sensor data from the vehicle to the operator in such a way, that the operator can easily understand the vehicle's environment and the vehicles current state. The focus of this paper is solving the second problem and therefore the development of a novel human-machine-interface for the teleoperation of automated vehicles. For the development, the human-centered-design process is used, in which the whole teleoperation system is analyzed in the context of autonomous driving and requirements for the new interface are derived. Based on those requirements the new concept is designed and implemented. Finally, first results from real driving scenarios and novel features, like adaptive camera projection or scene configuration, are presented.

To enable the development of driver assistant systems in a driving simulator, a realistic modelling of the driving dynamics is required. A simple approach to the dynamics is using a single-track model or a double-track model. A more detailed and more realistic approach is a multi-body model. To this end, a multi-body model was integrated in the dynamic truck driving simulator and evaluated. The requirements are real-time capability, realistic driving behaviour and simulator compatibility. We increased the immersion into the simulation via realistic dynamic behaviour. Due to the multi-body model, the dynamics at starting, cornering and braking are accurately computed. Against the background of automated driving, we created opportunities for further functional extensions such as automated longitudinal and lateral control.

Despite impressive developments in automated driving technology, several technical, economic and social challenges hinder the large-scale deployment of highly or full automated vehicles. We present teleoperated driving - where in-car drivers are replaced by tele-drivers located at a control center- as a transient technology to enable a driverless, door-to-door taxi service. In this novel service, the transmission of video and audio streams of the vehicle surroundings via wireless networks to the taxi dispatch center allows a human operator to remotely sense the environment through a virtual windshield and to remotely operate the vehicle controls through an emulated cockpit. This safe and cost-effective transport service merges together aspects of taxi transport with car sharing services if the passenger drives part of the route. A large-scale empirical evaluation study proves the feasibility of this novel taxi operation mode and shows that the implementation of the system can reduce, on average, the number of drivers to between 15% and 27% when considering teleoperation during pickup/dropoff and service, respectively. A premium service where passengers are remotely also driven from their origin to the destination also presents considerable gains for taxi operators. Teleoperation of taxi fleets could revolutionize urban mobility by offering a cost-effective and safe door-to-door transportation service.