Uncertainty quantification (UQ) is critical for deploying dense perception in intelligent vehicles, yet it is rarely included in off-the-shelf regression networks because joint UQ training often couples the likelihood to the predictor and degrades point-estimate accuracy. We frame IO-CUE as a post-hoc behavioral probe for frozen, black-box predictors: an auxiliary model conditioned on both inputs and outputs, trained with a detached maximum-likelihood objective on mild, structured perturbations. These probes expose how the base model’s predictions deform off-manifold, allowing IO-CUE to convert output-space deviations into calibrated uncertainty and to capture quasi-epistemic signatures without modifying the base predictor. Across synthetic tasks and dense depth benchmarks (NYU and driving datasets), IO-CUE matches or improves calibration and OOD sensitivity relative to ensembles, Laplace approximations, concurrent MLE variants, and output-only post-hoc baselines. Crucially, it generalizes to unseen perturbations and transfers across base models, revealing stable, dataset-agnostic output-space regularities characteristic of the model family. These behavioral signatures enable practical, low-overhead OOD detection in intelligent-vehicle settings without retraining the base network. Code: https://github.com/biggzlar/IO-CUE.

Timely detection of vulnerable road users (VRU) remains one of the toughest safety challenges as driving tasks shift from humans to automated systems. Beyond factors such as environmental visibility and road conditions, specific perceptual and behavioral characteristics of vulnerable road users could also compromise driver hazard perception. This study focuses on motion camouflage, a stealth strategy seen in animals in which a pursuer moves in such a way that its image on the target’s retina mimics the motion of a stationary object, thereby minimizing the optical cues that usually draw attention. Motion camouflage in driving scenarios refers to situations where, e.g., specific VRU movements make them less detectable to drivers. A virtual reality driving simulator was used to study how driver braking responses are affected by VRU motion trajectories under varying environmental conditions. Results show that VRU motion camouflage constitutes a fundamental gap in hazard perception, posing a risk that could potentially result in delayed detection and increased risk of accidents. Based on these findings, this paper provides a basis for developing more trustworthy advanced driver-assistance systems (ADAS) capable of recognizing and mitigating risks due to such perceptual vulnerabilities.

Imbalanced datasets are a common challenge in machine learning, where rare situations can significantly impact model performance, especially in safety critical applications like autonomous driving. Addressing these imbalances during the recording stage of a dataset is crucial to ensure that models generalize well to new data and real-world scenarios. This paper proposes a novel approach to improve pose dataset distribution live during dataset production by detecting and mitigating rare human poses interactively. Our live out-of-distribution mitigation (LOODM) method uses dimensionality reduction to transform high-dimensional human pose data into a low-dimensional latent space, allowing for efficient analysis of human poses. We analyze the latent space using density and distance to identify unique rare poses and visualize them to a recorded human subject, who can then repeat each rare pose until it is better represented in the dataset. As such, our work focuses on addressing dataset imbalances at their point of origin, namely during recording. LOODM demonstrates significant improvements in the performance of a human pose estimation model on rare poses, while also reducing PCKh metric fluctuations between rare and non-rare poses, making it a promising dataset production tool, e.g., for autonomous driving.

Machine learning models, particularly those used in open-world settings like autonomous driving, require ex-tensive and diverse datasets for effective training. However, the overabundance of standard situations in large datasets often leads to underrepresentation of rare scenarios and edge cases, which are crucial for achieving high performance across all situations. This work proposes a simulation framework for capturing human-in-the-loop simulation data, enabling the creation of realistic and diverse datasets for machine learning models as well as human behavior studies. The framework combines full body motion capture with virtual reality to record scene-relevant interactions between humans and virtual objects, allowing for the generation of various ground truths in simulation. To validate the effectiveness of this approach, an action recognition model is trained on simulated data generated by the proposed framework. From the point of view of a vehicle, the model needs to determine if the vehicle is the intended target of a waving pedestrian. The investigation into different keypoint representations and spatial encodings reveals the importance of integrating spatial and contextual data for accurate action recognition. Furthermore, the results show that human-in-the-loop simulations can effectively capture complex human behaviors and interactions, enabling the creation of more realistic and diverse datasets for machine learning models.

Situational awareness of the driving environment is crucial for making safe and informed driving decisions. It can be deteriorated by distractions or environmental properties such as low light. In order to study and model the effects of environmental, perceptual, and behavioral features within the binocular field of view on the drivers’ awareness of pedestrians, an experimental setup consisting of a Virtual Reality (VR) based driving simulator, data generation, and analysis solution has been developed. An experimental study was conducted to evaluate the developed setup and analyze drivers’ awareness of pedestrians in the context of varying external conditions. Results show that driver awareness can be measured from the properties of eye tracking and a secondary detection task to detect pedestrians. It is demonstrated how this research aims towards developing more human-aware driver monitoring systems with assistive functionalities such as attention guidance, taking perceptual and cognitive factors into account.

Human pose estimation (HPE) is integral to scene understanding in numerous safety-critical domains involving human-machine interaction, such as autonomous driving or semi-automated work environments. Avoiding costly mistakes is synonymous with anticipating failure in model predictions, which necessitates meta-judgments on the accuracy of the applied models. Here, we propose a straightforward human pose regression framework to examine the behavior of two established methods for simultaneous aleatoric and epistemic uncertainty estimation: maximum a-posteriori (MAP) estimation with Monte-Carlo variational inference and deep evidential regression (DER). First, we evaluate both approaches on the quality of their predicted variances and whether these truly capture the expected model error. The initial assessment indicates that both methods exhibit the overconfidence issue common in deep probabilistic models. This observation motivates our implementation of an additional recalibration step to extract reliable confidence intervals. We then take a closer look at deep evidential regression, which, to our knowledge, is applied comprehensively for the first time to the HPE problem. Experimental results indicate that DER behaves as expected in challenging and adverse conditions commonly occurring in HPE and that the predicted uncertainties match their purported aleatoric and epistemic sources. Notably, DER achieves smooth uncertainty estimates without the need for a costly sampling step, making it an attractive candidate for uncertainty estimation on resource-limited platforms.

The research area of domain adaptation investigates methods that enable the transfer of existing models across different domains, e.g., addressing environmental changes or the transfer from synthetic to real data. Especially unsupervised domain adaptation is beneficial because it does not require any labeled target domain data. Usually, existing methods are targeted at specific tasks and require access or even modifications to the source model and its parameters which is a major drawback when only a black-box model is available. Therefore, we propose a CycleGAN-based approach suitable for black-box source models to translate target domain data into the source domain on which the source model can operate. Inspired by multi-task learning, we extend CycleGAN with an additional auxiliary task that can be arbitrarily chosen to support the transfer of task-related information across domains without the need for having access to a differentiable source model or its parameters. In this work, we focus on the regression task of 2D human pose estimation and compare our results in four different domain adaptation settings to CycleGAN and RegDA, a state-of-the-art method for unsupervised domain adaptation for keypoint detection.

We present a multitask network that supports various deep neural network based pedestrian detection functions. Besides 2D and 3D human pose, it also supports body and head orientation estimation based on full body bounding box input. This eliminates the need for explicit face recognition. We show that the performance of 3D human pose estimation and orientation estimation is comparable to the state-of-the-art. Since very few data sets exist for 3D human pose and in particular body and head orientation estimation based on full body data, we further show the benefit of particular simulation data to train the network. The network architecture is relatively simple, yet powerful, and easily adaptable for further research and applications.

Recognizing human actions, reliably inferring their meaning and being able to potentially exchange mutual social information are core challenges for autonomous systems when they directly share the same space with humans. Intelligent transport systems in particular face this challenge, as interactions with people are often required. The development and testing of technical perception solutions is done mostly on standard vision benchmark datasets for which manual labelling of sensory ground truth has been a tedious but necessary task. Furthermore, rarely occurring human activities are underrepresented in these datasets, leading to algorithms not recognizing such activities. For this purpose, we introduce a modular simulation framework, which offers to train and validate algorithms on various human-centred scenarios. We describe the usage of simulation data to train a state-of-the-art human pose estimation algorithm to recognize unusual human activities in urban areas. Since the recognition of human actions can be an important component of intelligent transport systems, we investigated how simulations can be applied for his purpose. Laboratory experiments show that we can train a recurrent neural network with only simulated data based on motion capture data and 3D avatars, which achieves an almost perfect performance in the classification of those human actions on real data.

Learning-based vision tasks are usually specialized on the sensor technology for which data has been labeled. The knowledge of a learned model is simply useless when it comes to data which differs from the data on which the model has been initially trained or if the model should be applied to a totally different imaging or sensor source. New labeled data has to be acquired on which a new model can be trained. Depending on the sensor, this can even get more complicated when the sensor data becomes more abstract and hard to be interpreted and labeled by humans. To enable reuse of models trained for a specific task across different sensors minimizes the data acquisition effort. Therefore, this work focuses on learning sensor models and translating between them, thus aiming for sensor interoperability. We show that even for the complex task of human pose estimation from 3D depth data recorded with different sensors, i.e. a simulated and a Kinect 2 TM depth sensor, human pose estimation can greatly improve by translating between sensor models without modifying the original task model. This process especially benefits sensors and applications for which labels and models are difficult if at all possible to retrieve from raw sensor data.

It is necessary to employ smart sensory systems in dynamic and mobile workspaces where industrial robots are mounted on mobile platforms. Such systems should be aware of flexible and non-stationary workspaces and able to react autonomously to changing situations. Building upon our previously presented RoPose-system [1], which employs a convolutional neural network architecture that has been trained on pure synthetic data to estimate the kinematic chain of an industrial robot arm system, we now present RoPose-Real. RoPose-Real extends the prior system with a comfortable and targetless extrinsic calibration tool, to allow for the production of automatically annotated datasets for real robot systems. Furthermore, we use the novel datasets to train the estimation network with real world data. The extracted pose information is used to automatically estimate the observing sensor pose relative to the robot system. Finally we evaluate the performance of the presented subsystems in a real world robotic scenario.

Significant advances have been achieved in mobile robot localization and mapping in dynamic environments, however these are mostly incapable of dealing with the physical properties of automotive radar sensors. In this paper we present an accurate and robust solution to this problem, by introducing a memory efficient cluster map representation. Our approach is validated by experiments that took place on a public parking space with pedestrians, moving cars, as well as different parking configurations to provide a challenging dynamic environment. The results prove its ability to reproducibly localize our vehicle within an error margin of below 1% with respect to ground truth using only point based radar targets. A decay process enables our map representation to support local updates.

Determining the viewpoint of traffic participants provides valuable high-level attributes to constrain the interpretation of their movement, and thus allows more specific predictions of alert behavior. We present a monocular object viewpoint estimation approach that is realized by a random regression forest. In particular, we address the circular and continuous structure of the problem for training the decision trees. Our approach builds on a 2D deformable part based object detector. Using detected cars on the KITTI vision benchmark, we demonstrate performance for continuous viewpoint estimation, ground point estimation, and their integration into a high-dimensional particle filtering framework. Besides location and viewpoint of cars, the filter framework considers full monocular egomotion information of the observing platform. This demonstrates the versatility of using only monocular information processing with appropriate machine learning.

A driver assistance system realizes that the driver is distracted and that a potentially hazardous situation is emerging. Where should it guide the attention of the driver? Optimally to the spot that allows the driver to make the best decision. Pedestrian detectability has been proposed recently as a measure of the probability that a driver perceives pedestrians in an image [9]. Leveraging this information allows a driver assistance system to direct the attention of the driver to the spot that maximizes the probability that all pedestrians are seen. In this paper we extend this concept to dynamic scenes. We use an annotated video dataset recorded from a moving car in an urban environment and acquire the detectabilities of pedestrians via a psychophysical experiment. Based on these measured detectabilites we train a machine learning algorithm to predict detectabilities from a set of image features. We then exploit this mapping to predict the optimal focus of attention in a second experiment, thus demonstrating the usefulness of our method in a dynamic driver assistance context.

Visual odometry has been promoted as a fundamental component for intelligent vehicles. Relying solely on monocular image cues would be desirable. Nevertheless, this is a challenge especially in dynamically varying urban areas due to scale ambiguities, independent motions, and measurement noise. We propose to use probabilistic learning with auxiliar depth cues. Specifically, we developed an expert model that specializes monocular egomotion estimation units on typical scene structures, i.e. statistical variations of scene depth layouts. The framework adaptively selects the best fitting expert. For on-line estimation of egomotion, we adopted a probabilistic subspace flow estimation method. Learning in our framework consists of two components: 1) Partitioning of datasets of video and ground truth odometry data based on unsupervised clustering of dense stereo depth profiles and 2) training a cascade of subspace flow expert models. A probabilistic quality measure from the estimates of the experts provides a selection rule overall leading to improvements of egomotion estimation for long test sequences.

Rich face models already have a large impact on the fields of computer vision, perception research, as well as computer graphics and animation. Attributes such as descriptiveness, semantics, and intuitive control are desirable properties but hard to achieve. Towards the goal of building such high-quality face models, we present a 3D model-based analysis-by-synthesis approach that is able to parameterize 3D facial surfaces, and that can estimate the state of semantically meaningful components, even from noisy depth data such as that produced by Time-of-Flight (ToF) cameras or devices such as Microsoft Kinect. At the core, we present a specialized 3D morphable model (3DMM) for facial expression analysis and synthesis. In contrast to many other models, our model is derived from a large corpus of localized facial deformations that were recorded as 3D scans from multiple identities. This allows us to analyze unstructured dynamic 3D scan data using a modified Iterative Closest Point model fitting process, followed by a constrained Action Unit model regression, resulting in semantically meaningful facial deformation time courses. We demonstrate the generative capabilities of our 3DMMs for facial surface reconstruction on high and low quality surface data from a ToF camera. The analysis of simultaneous recordings of facial motion using passive stereo and noisy Time-of-Flight camera shows good agreement of the recovered facial semantics.

How likely is it that a driver notices a person standing on the side of the road? In this paper we introduce the concept of pedestrian detectability. It is a measure of how probable it is that a human observer perceives pedestrians in an image. We acquire a dataset of pedestrians with their associated detectabilities in a rapid detection experiment using images of street scenes. On this dataset we learn a regression function that allows us to predict human detectabilities from an optimized set of image and contextual features. We exploit this function to infer the optimal focus of attention for pedestrian detection. With this combination of human perception and machine vision we propose a method we deem useful for the optimization of Human-Machine-Interfaces in driver assistance systems.

Affordable 3D vision is just about to enter the mass market for consumer products such as video game consoles or TV sets. Having depth information in this context is beneficial for segmentation as well as gaining robustness against illumination effects, both of which are hard problems when dealing with color camera data in typical living room situations. Several techniques compute 3D (or rather 2.5D) depth information from camera data such as realtime stereo, time-of-flight (TOF), or real-time structured light, but all produce noisy depth data at fairly low resolutions. Not surprisingly, most applications are currently limited to basic gesture recognition using the full body. In particular, TOF cameras are a relatively new and promising technology for compact, simple and fast 2.5D depth measurements. Due to the measurement principle of measuring the flight time of infrared light as it bounces off the subject, these devices have comparatively low image resolution (176 x 144 ... 320 x 240 pixels) with a high level of noise present in the raw data.

We propose a novel set of medial feature interest points based on gradient vector flow (GVF) fields [18]. We exploit the long ranging GVF fields for symmetry estimation by calculating the flux flow on it. We propose interest points that are located on maxima of that flux flow and offer a straight forward way to estimate salient local scales. The features owe their robustness in clutter to the nature of the GVF which accomplishes two goals simultaneously - smoothing of orientation information and its preservation at salient edge boundaries. A learning framework based on them, in contrast to classical edge-based feature detectors, would unlikely be distracted by background clutter and spurious edges, as these new mid-level features are shape-centered. We evaluate our scale-invariant feature coding scheme against standard SIFT keypoints by demonstrating generalization over scale in a patch-based pedestrian detection task.

Experience indicates that the sense of presence in a virtual environment is enhanced when the participants are able to actively move through it. When exploring a virtual world by walking, the size of the model is usually limited by the size of the available tracking space. A promising way to overcome these limitations are motion compression techniques, which decouple the position in the real and virtual world by introducing imperceptible visual-proprioceptive conflicts. Such techniques usually precalculate the redirection factors, greatly reducing their robustness. We propose a novel way to determine the instantaneous rotational gains using a controller based on an optimization problem. We present a psychophysical study that measures the sensitivity of visual-proprioceptive conflicts during walking and use this to calibrate a real-time controller. We show the validity of our approach by allowing users to walk through virtual environments vastly larger than the tracking space.

To reduce the number of traffic accidents and to increase the drivers comfort, the thought of designing driver assistance systems arose in the past years. Fully or partly autonomously guided vehicles, particularly for road traffic, pose high demands on the development of reliable algorithms. Principal problems are caused by having a moving observer in predominantly natural environments. At the Institut fur Neuroinformatik methods for analyzing driving relevant scenes by computer vision are developed in cooperation with several partners from the automobile industry. We present a solution for a driver assistance system. We concentrate on the aspects of video-based scene analysis and organization of behavior.

In previous years, many methods providing the ability to recognize rigid obstacles-sedans and trucks-have been developed. These methods provide the driver with relevant information. They are able to cope reliably with scenarios on motorways. Nevertheless, not much attention has been given to image processing approaches to increase the safety of pedestrians in urban environments. In the paper, a method for the detection, tracking, and final recognition of pedestrians crossing the moving observer's trajectory is suggested. A combination of data- and model-driven approaches is realized. The initial detection process is based on a fusion of texture analysis, model-based grouping of, most likely, the geometric features of pedestrians, and inverse-perspective mapping (binocular vision). Additionally, motion patterns of limb movements are analyzed to determine initial object-hypotheses. The tracking of the quasirigid part of the body is performed by different algorithms that have been successfully employed for the tracking of sedans, trucks, motorbikes, and pedestrians. The final classification is obtained by a temporal analysis of the walking process.