This paper introduces a novel track-to-track sensor fusion framework that integrates event-based camera tracks and radar detection tracks for robust multi-object tracking. Due to the lack of an available dataset combining real event-camera data with millimeter-wave (mmWave) radar in the automotive context, we utilize a state-of-the-art radar-camera dataset, adapting it to simulate event-based camera data. Event-based cameras, with their high temporal resolution and low energy consumption, provide asynchronous brightness change data, while radar sensors offer robustness to adverse environmental conditions and precise depth measurements. The proposed method leverages the complementary strengths of these two sensor modalities using a Global Nearest Neighbor (GNN) algorithm to associate tracks and maintain a single hypothesis about tracked objects. The event-camera's optical flow-based velocity and distance estimation is fused with radar's depth and lateral position measurements to enhance localization accuracy. This approach demonstrates the potential for improving tracking performance and robustness in autonomous driving scenarios.

Distribution shifts between operational domains can severely affect the performance of learned models in self-driving vehicles (SDVs). While this is a well-established problem, prior work has mostly explored naive solutions such as fine-tuning, focusing on the motion prediction task. In this work, we explore novel adaptation strategies for differentiable autonomy stacks (structured policy) consisting of prediction, planning, and control, perform evaluation in closed-loop, and investigate the often-overlooked issue of catastrophic forgetting. Specifically, we introduce two simple yet effective techniques: a low-rank residual decoder (LoRD) and multi-task fine-tuning. Through experiments across three models conducted on two real-world autonomous driving datasets (nuPlan, exiD), we demonstrate the effectiveness of our methods and highlight a significant performance gap between open-loop and closed-loop evaluation in prior approaches. Our approach improves forgetting by up to 23.33% and the closed-loop out-of-distribution driving score by 9.93% in comparison to standard fine-tuning. https://github.com/rst-tu-dortmund/LoRD

Visual observation of objects is essential for many robotic applications, such as object reconstruction and manipulation, navigation, and scene understanding. Machine learning algorithms constitute the state-of-the-art in many fields but require vast data sets, which are costly and time-intensive to collect. Automated strategies for observation and exploration are crucial to enhance the efficiency of data gathering. Therefore, a novel strategy utilizing the Next-Best-Trajectory principle is developed for a robot manipulator operating in dynamic environments. Local trajectories are generated to maximize the information gained from observations along the path while avoiding collisions. We employ a voxel map for environment modeling and utilize raycasting from perspectives around a point of interest to estimate the information gain. A global ergodic trajectory planner provides an optional reference trajectory to the local planner, improving exploration and helping to avoid local minima. To enhance computational efficiency, raycasting for estimating the information gain in the environment is executed in parallel on the graphics processing unit. Benchmark results confirm the efficiency of the parallelization, while real-world experiments demonstrate the strategy's effectiveness.

The fusion of sensor data is essential for a robust perception of the environment in autonomous driving. Learning-based fusion approaches mainly use feature-level fusion to achieve high performance, but their complexity and hardware requirements limit their applicability in near-production vehicles. High-level fusion methods offer robustness with lower computational requirements. Traditional methods, such as the Kalman filter, dominate this area. This paper modifies the Adapted Kalman Filter (AKF) and proposes a novel transformer-based high-level object fusion method called HiLO. Experimental results demonstrate improvements of 25.9 percentage points in$\mathbf{F}_{1}$score and 6.1 percentage points in mean IoU. Evaluation on a new large-scale real-world dataset demonstrates the effectiveness of the proposed approaches. Their generalizability is further validated by cross-domain evaluation between urban and highway scenarios. Code, data, and models are available at https://github.com/rst-tu-dortmund/HiLO.

Metric localization of automated vehicles using exteroceptive sensors involves finding reliable spatial features in both the sensor data and the map. In real world scenarios, methods have to deal with unknown and changing environments and noisy sensor measurements, making feature selection considerably harder. If the map is created using a different sensor modality, localization methods also have to deal with the characteristics of the available sensor. Machine learning methods promise a solution to these problems by extracting the same features from maps created from different sensors. In this work, we compare an approach for learning based cross modality localization with a classical method on different types of maps. Furthermore, we enhance the learned model by estimating its uncertainty directly from the measurement data.

The collaboration and interaction between humans and robots intensify with ongoing research and industry needs. Robots require a motion planner that contributes to a safe environment for humans. This paper provides the online trajectory planner Moving Horizon Planning for Human-Robot Interaction (MHP4HRI), customizable for various robots, considering obstacles and humans in their environment. The planner generates motion commands in a moving horizon manner, similar to Model Predictive Control. This enables robots to react to dynamic changes in the environment in real-time. Descriptions of the planner and the underlying algorithms are given, as well as details about the provided framework regarding the benefits and usage for the community. Furthermore, we aim to provide a growing framework with new features in the future regarding the optimization and interaction with the environment, especially humans. The code, implemented mainly in C++ for the Robot Operating System (ROS), is available at GitHub: https://github.com/rst-tu-dortmund/mhp4hri.

The Steer-by-Wire (SbW) steering system is a key technology for highly automated driving. For automated lateral vehicle guidance, the precise position control of the SbW Front Axle Actuator is an essential prerequisite. This paper presents the modeling, control synthesis, control loop analysis, and vehicle performance evaluation of the position control for the SbW Front Axle Actuator. Based on a nonlinear model of the plant a simplified linear system model is derived. This model yields the basis for the design of a Two-Degrees of Freedom Linear Quadratic Gaussian Control (2DOF LQG control), which allows an independent design of the command and the disturbance response. Besides a linear analysis of the control performance and stability, real vehicle tests for different driving maneuvers are conducted to verify simulation results and get a representative picture of the control performance.

Predicting the future trajectories for all vehicles relevant to the ego vehicle is a crucial, yet unsolved challenge to master automated driving. This paper proposes a combination of two lines of research for predicting all the trajectories of a group of vehicles of arbitrary size, considering the mutual interactions possible. Treating the prediction of other vehicles as a planning task for themselves enables the application of the artificial potential field approach. Modeling the driving situation as a potential field turns the trajectory prediction problem back to its original domain – utility space. Humans generate trajectories (that should be predicted) during driving by balancing costs and rewards, which lead to a total utility. The main difficulty inherent to the potential field approach is the hard problem of parameter tuning. Therefore, it is not directly used for prediction. Instead, the potential field representation is used as input for a neural network, which predicts a distribution over trajectories based on distinct maneuvers. This allows a multi-modal prediction for each vehicle and reflects the pattern recognition character.

This paper presents a purely visual semantic mapping framework using omnidirectional images. The approach rests upon the robust segmentation of the robot's local free space, replacing conventional range sensors for the generation of occupancy grid maps. The perceptions are mapped into a bird's eye view allowing an inverse sensor model directly by removing the non-linear distortions of the omnidirectional camera mirror. The system relies on a place category classifier to label the navigation relevant categories: room, corridor, doorway, and open room. Each place class maintains a separated grid map that are fused with the range-based occupancy grid for building a dense semantic map.

The paper at hand proposes an environment model for trajectory planning in structured environments. It is composed of a static and a dynamic environment model. The generated static potential field takes restrictions imposed by the static environment into account. The dynamic environment model is based on the physical interpretation of the required safety distance. By the use of an advanced obstacle trajectory prediction method, the safety distance is calculated in accordance to the predicted situation. As the safety distance affects the dynamic potential field, information provided by the obstacle trajectory prediction is directly considered in the ego vehicle trajectory planning process. On account of the predictive character of the developed environment potential field, simulation experiments demonstrate the feasibility and effectiveness of the proposed method.

This paper presents a novel generic formulation of Timed-Elastic-Bands for efficient online motion planning of car-like robots. The planning problem is defined in terms of a finite-dimensional and sparse optimization problem subject to the robots kinodynamic constraints and obstacle avoidance. Control actions are implicitly included in the optimized trajectory. Reliable navigation in dynamic environments is accomplished by augmenting the inner optimization loop with state feedback. The predictive control scheme is real-time capable and responds to obstacles within the robot's perceptual field. Navigation in large and complex environments is achieved in a pure pursuit fashion by requesting intermediate goals from a global planner. Requirements on the initial global path are fairly mild, compliance with the robot kinematics is not required. A comparative analysis with Reeds and Shepp curves and investigation of prototypical car maneuvers illustrate the advantages of the approach.

Situation understanding and assessment is one of the key features for automated driving. To enable safe and comfortable motion planning, sensing the current situation is not sufficient but maneuver predictions as accurate as possible are required. The paper presents a novel approach of predicting the remaining time to an upcoming lane change of adjacent vehicles on a highway. The prediction is performed in a probabilistic way to cope with the variety in execution and duration of lane change maneuvers. Two quantile regression techniques, namely Linear Quantile Regression and Quantile Regression Forests, are applied and compared in terms of prediction error and accuracy on data gathered with different drivers on a fixed base driving simulator. The superior technique is also evaluated on a dataset recorded with a test vehicle to demonstrate its general applicability in real world scenarios.

In this paper the sensorless control of a permanent magnet synchronous machine which is meant to be used in pump applications is discussed. The sensorless control in the middle and high speed region can be considered as state of the art. In contrast to that, sensorless control in the low speed region for use in industrial applications still is an issue. Therefore, this paper focuses on saliency based sensorless control employing high frequency signal injection which usually is applied for low speed applications. However, it is shown that the used machine does not provide the required properties for such kind of sensorless control. Therefore, a new strategy is presented which compensates for the insufficient properties regarding position estimation in order to allow a saliency based approach anyway.

Link elasticity is commonly understood to be a detrimental side-effect of imperfect mechanical designs of robotic arms and comparable machinery. In contrast to this notion, this paper demonstrates a novel approach to exploit intrinsic robot link compliance in order to estimate a priori unknown payload masses, measure and also control end effector forces. In this way, the intrinsic link elasticity can be seen as an enabler for new sensing and control capabilities instead of a purely detrimental effect.

The infinite dimensionality, varying, uncertainties or even unknown boundary conditions render the derivation and - in particular - the identification of accurate dynamics models for elastic link robots tedious and error prone. This contribution circumvents these challenges by the prior application of a model-free inner loop oscillation damping controller before modelling the robot's dynamics. Then, the damped dynamics of a multi elastic link robot arm under gravity can be modelled with high accuracy. An analytical and a data-driven model for the damped dynamics are proposed and quantitatively compared. Both models can explain motor currents as well as link strain measurements in real-time. The paper includes an experimental model validation with different payloads in the entire workspace of the robot.

This contribution is concerned with an emergency steering assist and its evaluation through subject testing in a driving simulator. In an emergency traffic situation where a rear end collision is imminent a swerving maneuver is often too difficult for most drivers. Therefore an assistance system is usefull that supports the driver by steering torque overlay. The paper presents the algorithm used for the assistance. The system was prototypically implemented in a driving simulator and testet with subjects to evaluate the benefits and challenges. The results show that the collision avoidance behaviour of the driver can be improved by the emergency steering assist.

The impact of additive periodic torque disturbances on the controlled motion of permanent magnet motors can be significant. The paper shows how an observer-based drive control can efficiently reject the harmonic torque disturbances providing smooth angular velocity. The proposed control design is based on the state-space torque harmonics representation and Luenberger observer that proved to be adequate. The designed control algorithms are verified using an experimental setup with a permanent magnet synchronous motor with well-detectable torque harmonics. The rejection of additive position periodic torque disturbances is experimentally demonstrated for two first harmonics and that for different angular velocities.

The paper experimentally compares six visual oscillation sensing approaches for a three degrees of freedom flexible link robot arm with an eye-in-hand RGB-D camera. The comparison includes five representative scenarios. Based upon the results the authors propose a novel scene adaptive camera motion reconstruction scheme. The scheme adaptively selects the best approach according to the actual scene texture and depth profile. Experiments in indoor scenarios with sparse texture, poor depth profiles as well as dynamic scene contents approve the obtained signal quality to be well suited for visual vibration damping of flexible link robot arms in a great variety of frequently observed scenarios.

Future advanced driver assistance systems will contain multiple sensors that are used for several applications, such as highly automated driving on freeways. The problem is that the sensors are usually asynchronous and their data possibly out-of-sequence, making fusion of the sensor data non-trivial. This paper presents a novel approach to track-to-track fusion for automotive applications with asynchronous and out-of-sequence sensors using information matrix fusion. This approach solves the problem of correlation between sensor data due to the common process noise and common track history, which eliminates the need to replace the global track estimate with the fused local estimate at each fusion cycle. The information matrix fusion approach is evaluated in simulation and its performance demonstrated using real sensor data on a test vehicle designed for highly automated driving on freeways.

Driver-assistance systems and automated driving applications in the future will require reliable and flexible surround environment perception. Sensor data fusion is typically used to increase reliability and the observable field of view. In this paper, a novel approach to track-to-track fusion in a high-level sensor data fusion architecture for automotive surround environment perception using information matrix fusion (IMF) is presented. It is shown that IMF produces the same good accuracy in state estimation as a low-level centralized Kalman filter, which is widely known to be the most accurate method of fusion. Additionally, as opposed to state-of-the-art track-to-track fusion algorithms, the presented approach guarantees a globally maintained track over time as an object passes in and out of the field of view of several sensors, as required in surround environment perception. As opposed to the often-used cascaded Kalman filter for track-to-track fusion, it is shown that the IMF algorithm has a smaller error and maintains consistency in the state estimation. The proposed approach using IMF is compared with other track-to-track fusion algorithms in simulation and is shown to perform well using real sensor data in a prototype vehicle with a 12-sensor configuration for surround environment perception in highly automated driving applications.

Modeling and identification of dynamic systems often is a prerequisite for the engineering of technical solutions, for example control system design. This paper presents an multi objective evolutionary approach for identification of dynamic systems of variable structure. The evolutionary algorithm employs domain specific operators in order to evolve the block oriented structure of the model and simultaneously optimize its parameters. Based on the observed inputs and outputs the multi objective method identifies an entire set of optimal compromise models which contrast model accuracy against complexity. The models are constructed from a set of basic blocks that capture phenomenons such as linear transfer functions, nonlinear gains and hysteresis that typically occur in mechanical, hydraulic and electrical systems. This representation enables the incorporation of domain knowledge in terms of building blocks and the interpretation of the identified model for further analysis and design. The feasibility of the proposed method is validated in the identification of an artificial dynamic system as well as a hydraulic proportional valve.

The design of visual robotic behaviors constitutes a substantial challenge. It requires to draw meaningful relation ships and constraints between the acquired visual perception and the geometry of the environment both empirically and programmatically. This contribution proposes a novel robot learning framework to classify and acquire scenario specific autonomous behaviors through demonstration. During demonstration, robocentric 3D range and omnidirectional images are recorded as training instances of typical robot navigation situations pertaining to different contexts in multiple indoor scenarios. A programming by demonstration approach generalizes the demonstrated trajectories to a general mapping between visual features extracted from the omnidirectional image onto a corresponding robot motion. The approach is able to distinguish among different traversing scenarios and further identifies the best matching context within the scenario to predict an appropriate robot motion. As a comparison to context matching, the behaviors are trained by means of an artificial neural network and its generalization ability is evaluated against the former. The experimental validation on the mobile robot indicates that the acquired visual behavior is robust and generalizes meaningful actions beyond the specific environments and scenarios presented during training.

This paper presents a novel approach for floor obstacle segmentation in omnidirectional images which rests upon the fusion of multiple classification generated from heterogeneous segmentation schemes. The individual naive Bayes classifiers rely on different features and cues to determine a pixel's class label. Ground truth data for training and testing the classifiers is obtained from the superposition of 3D scans captured by a photonic mixer device camera. The classification is supported by edge detection which indicate the presence of obstacles and sonar range data. The complementary expert decisions are aggregated by stacked generalization, behavior knowledge space or voting combination. The combined floor classifier achieves a classification accuracy of up to 0.96 true positive rate with only 0.03 false positive rate. A robust robot navigation is accomplished by arbitration among a reactive obstacle avoidance and a corridor following behavior using the robots local free space as perception.

Future driver assistance systems need to be more robust and reliable because these systems will react to increasingly complex situations. This requires increased performance in environment perception sensors and algorithms for detecting other relevant traffic participants and obstacles. An object's existence probability has proven to be a useful measure for determining the quality of an object. This paper presents a novel method for the fusion of the existence probability based on Dempster-Shafer evidence theory in the framework of a highlevel sensor data fusion architecture. The proposed method is able to take into consideration sensor reliability in the fusion process. The existence probability fusion algorithm is evaluated for redundant and partially overlapping sensor configurations.

The research interest in unmanned aerial vehicles (UAV) has grown rapidly over the past decade. UAV applications range from purely scientific over civil to military. Technical advances in sensor and signal processing technologies enable the design of light weight and economic airborne platforms. This paper presents a complete mechatronic design process of a quadrotor UAV, including mechanical design, modeling of quadrotor and actuator dynamics and attitude stabilization control. Robust attitude estimation is achieved by fusion of low-cost MEMS accelerometer and gyroscope signals with a Kalman filter. Experiments with a gimbal mounted quadrotor testbed allow a quantitative analysis and comparision of the PID and Integral-Backstepping (IB) controller design for attitude stabilization with respect to reference signal tracking, disturbance rejection and robustness.

This paper describes a novel approach for purely vision based mobile robot navigation. The visual obstacle avoidance and corridor following behavior rely on the segmentation of the traversable floor region in the omnidirectional robocentric view. The image processing employs a supervised approach in which the segmentation optimal with respect to the appearance of the local environment is determined by cross validation over 3D scans captured by a photonic mixer device (PMD) camera. The range data in the front view provides the seeds and validation data to supervise the appearance based segmentation in the omniview. Segmentation relies on histogram backprojection which maintains separate appearance models for floor, obstacles and background. A naive Bayes classifier predicts the occupancy of the robots local environment by fusing the evidence provided by different segmentations and models. The classification error is analyzed on ground truth data generated by a PMD camera and manually segmented scenes. The scheme is highly robust with respect to ambiguous and misleading visual appearances of obstacles and floor, thus enabling the robot to navigate safely in unstructured environments of diverse appearance, texture and illumination. The proposed vision algorithm and the navigation behavior demonstrate a robust performance in extensive robotic experiments across several hours of autonomous operation.

Flexible, lightweight manipulators offer some advantages in contrast to rigid arms, such as compact and lighter drives, energy efficiency, reduced masses and costs. This paper presents a novel approach for vibration damping of a multi-link flexible arm. The strain of the elastic arms is measured with Fiber-Bragg-Grating (FBG) sensors and provides the feedback signal to dampen their flexural dynamics. A dynamic model of a three link arm is derived that accounts for the rigid and flexural dynamics including gravity. The arm vibrations are damped by nonlinear strain feedback. The controller is general and robust and its design does not require a model of the flexural dynamics. In the context of closed loop vibration control FBG sensors offer a better signal to noise ratio compared to strain gauges, which allows a higher static gain in the feedback loop with more efficient dissipation of vibrational energy. The feasibility and effectiveness of the proposed vibration control scheme in conjunction with FBG sensors is verified and analyzed in simulations and confirmed in experiments with a flexible three link robot arm.

Evolutionary algorithms perform robust search in complex and high dimensional search spaces, but require a large number of fitness evaluations to approximate optimal solutions. These characteristics limit their potential for hardware in the loop optimization and problems that require extensive simulations and calculations. Evolutionary algorithms do not maintain their knowledge about the fitness function as they only store solutions of the current generation. In contrast, model assisted evolutionary algorithms utilize the information contained in previously evaluated solutions in terms of a data based model. The convergence of the evolutionary algorithm is improved as some selection decisions rely on the model rather than to invoke expensive evaluations of the true fitness function. The novelty of our scheme stems from the preselection of solutions based on an instance based fitness model, in which the selection pressure is adjusted to the quality of model. This so-called lambda-control adapts the number of true fitness evaluations to the monitored model quality. Our method extends the previous approaches for model assisted scalar optimization to multi-objective problems by a proper redefinition of model quality and preselection pressure control. The analysis on multi-objective benchmark optimization problems not only confirms the superior convergence of the model assisted evolution strategy in comparison with a multi-objective evolution strategy but also the positive effect of regulated preselection in contrast to merely static preselection.

For efficient software engineering in automotive applications the executed design processes must separate general properties and characteristics of the created system from implementation and realisation details as long as possible. This short paper shows how software architectures can be displayed on an abstract level by using the Unified Modeling Language (UML) and how these abstract depictions can be transferred to commonly used design and code generation tools either manually or by automation. Formal methods and transformation rules can be applied by using the Extensible Markup Language (XML). The methods are exemplarily shown for ASCET SE by ETAS.

Increasing demands concerning safety, economic impact, fuel consumption and comfort result in growing utilization of mechatronic components and networking of up to now widely independent systems in vehicles. The development of such a complex and networked system requires a coordinated, systematic development process. In this contribution a suitable process model is presented. It supports the verification of a domain model considering functional requirements in an early stage of development. The process model takes two different types of modeling into account. Object oriented modeling is used to describe domain models and particularly supports aspects like re-use, exchangeability, scalability and distributed development. Data flow oriented modeling especially focuses on dynamic aspects and is employed to create a simulation model. Coupling points are identified allowing an automated mapping of these two types of models.