Human-Robot Collaboration (HRC) in dynamic assembly environments requires intuitive communication for seamless interactions. Eye gaze, a nonverbal cue, is critical for interpreting human intent and enhancing robot adaptability. This research introduces a three-tier framework categorizing eye gaze into fixation, scanning, and task-switching behaviors, aiming to improve real-time robot responsiveness. Preliminary evaluations with multiple human subjects demonstrated the potential of this classification to enhance task alignment and detect anomalies in human behavior. Future work will explore automated personalization of the algorithm for individuals and integrate multimodal data to refine adaptability in human-robot collaborative systems.

Extracted driving behavior of human driven vehicles can benefit the development of various applications like trajectory prediction or planning, abnormal driving detection, driving behavior classification, traffic simulation modeling, etc. In this paper, we focus on modeling human driving behavior in order to find simplifications for trajectory planning. Using a time-discrete kinematic bicycle model with the vehicle’s acceleration and steering rate as inputs, we model the human driven trajectories of an urban intersection drone dataset for different input sampling times. While most planning algorithms are using input sampling times below 0.33s, we are able to model 98.2% of the human driven trajectories of the investigated dataset with a sampling time of 0.6s. Using longer input sampling times can result in smoother trajectories and longer planning horizons, and thus more efficient trajectories. In a next step, we analyze the correlations between the input of our model and the current state/last input. Such a priori knowledge could simplify common planning algorithms like model predictive control or tree-search based planners by limiting the action space of the ego-vehicle. We propose nonlinear transformations for steering rate and steering angle to represent correlations between speed, acceleration, steering angle and steering rate. In the transformed space the statistics are very well modeled by multivariate Gaussian distributions. Using a multivariate Gaussian, a fast usable behavior model is extracted which is independent of the environment.

One necessary condition for autonomous driving is an accurate and reliable representation of the environment around the vehicle. Current architectures rely on cameras, radars, and lidars to capture the visual environment and to localize and track other traffic participants. Human drivers can see but also hear and use a lot of auditory information for understanding the environment in addition to visual cues. In this paper, we present a pure sound localization and recognition system to extract an auditory representation of the environment. First, the environmental sound is classified into seven main categories of traffic objects followed by six specific kinds of sirens in an emergency case using a simple neural network layout. Second, each object is localized via a combined time-delay of arrival and amplitude-based localization algorithm. The system is evaluated on real-world data focusing on a robust detection and accurate localization of emergency vehicles.

One research aspect of the growing field of driver monitoring is the estimation of the driver's focus of attention and the detection of seen or missed surrounding objects. For this purpose, wearable head eye trackers, also called eye tracking glasses, are commonly used in research. For new Advanced Driver Assistance Systems, however, only remote gaze tracking systems in combination with series surround sensors (e. g. radar, camera) are a reasonable and suitable configuration. Remote eye trackers, though, do not reach the precision of wearable devices. Moreover, the use of inside-outside looking systems is affected by calibration errors. Both aspects lead to traffic participants classified as missed although they have been perceived by the driver and vice versa. Insight in the performance not only of the eye tracker itself but also on the model performance of an object of fixation detection algorithm, i. e. how a model handles those shortcomings, would be useful. In this work, we propose an approach to evaluate different models. For this, we combine the remote eye tracker with a wearable device, compare different models based on the newly obtained reference, and discuss the encountered problems.

In this paper, we present a method to generate compact geometric track-maps for train-borne localization applications. Therefore, we first give a brief overview on the purpose of track maps in train-positioning applications. It becomes apparent that there are hardly any adequate methods to generate suitable geometric track-maps. This is why we present a novel map generation procedure. It uses an optimization formulation to find the continuous sequence of track geometries that fits the available measurement data best. The optimization is initialized with the results from a localization filter [1] developed in our previous work. The localization filter also provides the required information for shape identification and measurement association. The presented approach will be evaluated on simulated data as well as on real measurements.

In this paper, we present a cooperative visual odometry system based on the detection of mobile markers. To this end, we introduce a simple scheme that realizes visual mobile marker odometry via accurate fixed marker-based camera positioning and we discuss the characteristics of errors inherent to the method compared to classical fixed marker-based navigation and visual odometry. The proposed cooperative scheme has the advantage of not needing any feature or fiducial marker in the environment, which can be used for indoor and underwater applications where it can be harder to extract reliable features. We provide specific configurations of UAV and UGV robots including one that allows continuous movements of the UAV, and a minimal caterpillar-like configuration that works with one UGV alone. Finally, we present a real-world implementation and an evaluation of the proposed configurations.

Automotive visual odometry has become a highly researched topic. The published works have turned cameras into a very precise source of ego motion estimation. However, for automotive application, visual odometry has to be integrated into a sensor fusion to simultaneously obtain global localization, maximum availability as well as highest precision. Since any visual odometry remains sensitive to its environment, good motion estimation cannot always be guaranteed. Consequently, a self-validation scheme is one of the barriers towards application in a sensor fusion system. In order to solve this problem, we first formulate an Ackermann vehicle's motion as a function of its forward speed and yaw rate. Secondly, we present a data-driven model, which achieves to reconstruct the sideward speed as a function of the yaw rate only. As we show, both models reach the quality of the estimated sideward motion from visual odometry. The therewith achieved redundancy can be used for different tasks. Of course, the estimation of the sideward motion can be excluded from the visual odometry scheme to save computation time. This is of special interest for monocular systems, where yet no absolute scale of a translation motion could be directly calculated. Instead, we propose to maintain the estimation of the sideward motion in the visual odometry and to compare the result to the modeled motion. As we show, the emerging deviation is a very good metric for self-validation of the overall visual odometry estimate. Integrating the resulting method into our visual odometry system, we currently achieve the best frame-to-frame result in the KITTI benchmark.

In this paper, we propose an optimization scheme to detect outliers in a visual odometry pipeline that is purely based on the optical flow of one monocular camera without requiring information about depth. First, we review different optical flow error measures and uncover the different sensitivity changes of the endpoint error for optical flow to the directional and the absolute error components. Then, we analyse the dependency of the direction of the flow induced by the translational components of the camera motion assuming the flow induced by the rotational camera motion components to be known. Based on a reliable estimate of the focus of expansion, we solve for a normalized directional flow error that takes the influence of the discretization error induced by the pixel size into account. Here, a circular and a squared error bound on the discretization error is investigated. Finally, we present a monocular outlier detection pipeline including the normalized directional flow error as a suitable constant threshold criterion almost invariant to the camera motion and the scene geometry. Further on, evaluations of the overall monocular outlier detection method called FERO included in a stereo visual odometry pipeline are given to compare the performance of monocular against stereoscopic outlier detection based on Kitti benchmark.

A necessary prerequisite for future driver assistance systems as well as automated driving is a suitable and accurate representation of the environment around the vehicle with a sufficient range. To extend the range of the environment representation, sharing the environment detections of multiple vehicles via vehicle-to-vehicle (V2V) communication is a promising approach. In this paper, we present a method to fuse shared free space detections from multiple vehicles. The detections are represented as Parametric Free Space (PFS) maps, which are especially suitable for real-time radio V2V-transmission due to their compactness. A graph-based algorithm to fuse PFS maps is proposed that solves possible contradictions between the maps and incorporates the maps' uncertainty attributes. By solely operating on the contour, the fusion can be carried out by a simple path search in a fusion graph that is constructed from the maps. This results in an efficient method that finds the fusion result within few iterations. To account for possible errors in the relative poses between the PFS maps, we furthermore present an adapted Iterative Closest Point (ICP) matching to align the maps before the fusion. Therein we employ a modified soft-assign scheme for robust outlier rejection, and incorporate the PFS maps' boundary orientation to improve the matching process. We show the capabilities of our method by presenting results on real test drive data.

In this paper, we present an outlier removal scheme for stereo-based visual odometry which is especially suited for improving high-speed pose change estimations in large-scale depth environments. First we investigate the variance of the reprojection error on the 3D position of a feature given a fixed error in pose change to conclude that a detection of outliers based on a fixed threshold on the reprojection error is inappropriate. Then we propose an optical flow dependent feature-adaptive scaling of the reprojection error to reach almost invariance to the 3D position of each feature. This feature-adaptive scaling is derived from an approximation showing the relation between longitudinal pose change of the camera, absolute value of the optical flow, and distance of the feature. Using this scaling, we develop an iterative alternating scheme to guide the separation of inliers from outliers. It optimizes the tradeoff between finding a good criterion to remove outliers based on a given pose change and improving the pose change hypothesis based on the current set of inliers. Including the new outlier removal scheme into a pure two-frame stereo-based visual odometry pipeline without applying bundle adjustment or SLAM-filtering we are currently ranked amongst the top camera-based algorithms and furthermore outperform camera and laser scanner methods in Kitti benchmark's high-speed scenarios.

Situation recognition is a prerequisite for many advanced driver assistance systems as well as for partially and fully automated vehicles. Current situation recognition approaches focus mainly on estimating maneuvers of single scene entities. However, assessing multiple, possibly interacting, traffic participants simultaneously is crucial in complex traffic scenes and has hardly been investigated. Considering the variability and combinatorics of such scenarios, having specialized situation recognition systems covering each case directly is unrealistic. In this paper, we present a flexible framework for assessing generic traffic scenes with multiple interacting traffic participants. It is able to construct a fully interaction-respecting probabilistic situation assessment, while relying on reusable state-of-the-art single-entity-based maneuver predictions. The benefits and applicability are presented on a real-world data set. The evaluation indicates that the approach is not only able to reconstruct underlying interdependent probability distributions; it outperforms specially designed models, due to the reduced complexities of the single-entity-based recognition models.

We present a novel parametric representation of general dynamic driving environments. It is particularly suitable for near-future Advanced Driver Assistance Systems due to its compactness, inherent consistency between static and dynamic entities, suppression of irrelevant details, as well as its sensor-independent, real-time capable generation. By building upon a common occupancy grid map-based environment representation, cells belonging to dynamic objects are simultaneously extracted, classified, and tracked in an object-based manner via an Interacting-MultipleModel-Unscented-Kalman-Probabilistic-Data-Association (IMM-UK-PDA) filter and cleared from the grid. The remaining static environment grid is subsequently processed by methods from the image analysis domain, followed by an Information Filter-based contour tracking of relevant free space boundaries to create a spatiotemporally smooth compact Parametric Free Space (PFS) map. The PFS map represents the static part of the traffic scene with modest bandwidth requirements. The system runs in real time on an experimental vehicle equipped with an automotive radar and a stereo camera and is evaluated within real traffic environments.

This paper describes an integrated Bayesian approach to maneuver-based trajectory prediction and criticality assessment that is not limited to specific driving situations. First, a distribution of high-level driving maneuvers is inferred for each vehicle in the traffic scene via Bayesian inference. For this purpose, the domain is modeled in a Bayesian network with both causal and diagnostic evidences and an additional trash maneuver class, which allows the detection of irrational driving behavior and the seamless application from highly structured to nonstructured environments. Subsequently, maneuver-based probabilistic trajectory prediction models are employed to predict each vehicle's configuration forward in time. Random elements in the designed models consider the uncertainty within the future driving maneuver execution of human drivers. Finally, the criticality time metric time-to-critical-collision-probability (TTCCP) is introduced and estimated via Monte Carlo simulations. The TTCCP is a generalization of the time-to-collision (TTC) in arbitrary uncertain multiobject driving environments and valid for longer prediction horizons. All uncertain predictions of all maneuvers of every vehicle are taken into account. Additionally, the criticality assessment considers arbitrarily shaped static environments, and it is shown how parametric free space (PFS) maps can advantageously be utilized for this purpose.

One of the main challenges when using multiple robots to explore unknown environments is the allocation of target points or regions to the individual units. To date, commonly used approaches produce uneven assignment of targets in situations where multiple targets have to be assigned to each robot. This imbalance can lead to increased overall exploration time and poses an interesting task for further investigation. Based on insights from Multi-Robot Routing and Traveling Salesman Problem research, we propose a specific algorithm for target point allocation that has advantageous properties in highly dynamic applications like exploration. The presented pairwise optimization procedure is suitable for application in distributed and challenging settings, not requiring central coordination or all to all communication, making our exploration strategy robust and flexible. We provide a theoretical analysis and statistical evaluations. Comparisons with representative approaches from the literature show that our algorithm is competitive with the best performing centralized approach.

Motion is a crucial source of information for a variety of tasks in social interactions. The process of how humans recognize complex articulated movements such as gestures or face expressions remains largely unclear. There is an ongoing discussion if and how explicit low-level motion information, such as optical flow, is involved in the recognition process. Motivated by this discussion, we introduce a computational model that classifies the spatial configuration of gradient and optical flow patterns. The patterns are learned with an unsupervised learning algorithm based on translation-invariant nonnegative sparse coding called VNMF that extracts prototypical optical flow patterns shaped, for example, as moving heads or limb parts. A key element of the proposed system is a lateral inhibition term that suppresses activations of competing patterns in the learning process, leading to a low number of dominant and topological sparse activations. We analyze the classification performance of the gradient and optical flow patterns on three real-world human action recognition and one face expression recognition data set. The results indicate that the recognition of human actions can be achieved by gradient patterns alone, but adding optical flow patterns increases the classification performance. The combined patterns outperform other biological-inspired models and are competitive with current computer vision approaches.

In the context of robotics coverage describes the deployment of mobile sensors or agents in a target area. This paper presents a new solution to coverage in nonconvex environments. We apply a modification to the coverage approach in (Cortes, et.al., 2004), making use of visibility sets and a contraction of the environment, leading to a change in the bounds of integration. The resulting feedback control is fully distributed, has guaranteed, inherent collision avoidance without separate path planning and reduced communication and computation requirements compared to similar approaches. We also formulate an extension to agents with limited sensing range. Our method works in unknown environments, if all regions are of equal interest. We provide simulation results demonstrating the advantageous properties of the approach.

We propose a method capable of acquiring an occupancy grid map-based representation of the local, static driving environment around an intelligent vehicle in the presence of dynamic objects. These corrupt the representation due to violating the underlying static-world assumptions of common grid mapping algorithms and are therefore detected and filtered from the map. For this purpose, a subsequent step is suggested that identifies, clusters and merges dynamic cell hypothesis in a novel way. Thereafter, an Interacting-Multiple-Model-Unscented-Kalman-Probabilistic-Data-Association (IMM-UK-PDA) tracker is used to classify of whether cell movements behave consistently with possible movement characteristics of real dynamic objects or are just generated by noise or newly observed static environment. In opposition to many other approaches, the method explicitly combines information of newly occupied and free areas, completes the shape of only partly visible dynamic objects and uses an advanced object tracking scheme to clean the grid from dynamic object corruptions. The method is evaluated with grids generated by an automotive radar and stereo camera in real traffic environments.

Intersections are the most accident-prone spots in the road network. In order to assist the driver in complex urban intersection situations, an ADAS will be required not only to recognize current but also to anticipate future maneuvers of the involved road users. Current approaches for intention estimation focus mainly on discerning only two intentions based on a vehicle's behavior. We argue that for distinguishing between more than two intentions not just a vehicle's kinematic behavior but also its driving situation needs to be taken into account. In our system we estimate four different intentions by modeling and recognizing driving situations in a Bayesian Network and using the behavior as additional evidence. For the behavior based estimation we present a newly engineered feature, the Anticipated Velocity at Stop line, that turned out to be a very strong indicator for the intention. Our system is evaluated on a real-world data set comprising approaches to seven different intersections on which we can show that our approach is able to estimate a driver's intention with a high accuracy.

Practical intrusion detection in Wireless Multihop Networks (WMNs) is a hard challenge. The distributed nature of the network makes centralized intrusion detection difficult, while resource constraints of the nodes and the characteristics of the wireless medium often render decentralized, node-based approaches impractical. We demonstrate that an active-probing-based network intrusion detection system (AP-NIDS) is practical for WMNs. The key contribution of this paper is to optimize the active probing process: we introduce a general Bayesian model and design a probe selection algorithm that reduces the number of probes while maximizing the insights gathered by the AP-NIDS. We validate our model by means of testbed experimentation. We integrate it to our open source AP-NIDS DogoIDS and run it in an indoor wireless mesh testbed utilizing the IEEE 802.11s protocol. For the example of a selective packet dropping attack, we develop the detection states for our Bayes model, and show its feasibility. We demonstrate that our approach does not need to execute the complete set of probes, yet we obtain good detection rates.

Visual motion is a rich source of information that is directly coupled to the underlying shape of a moving object. One way to describe motion is to use optical flow fields. Due to the aperture problem, dense optical flow estimation is an ill-constraint problem, while sparse optical flow estimation looses the shape information of moving objects. Current estimation algorithms based on regularization or segmentation fail at surface deformations or when the relevant motion is less dominant then its sourrounding movements. Both is e.g. true for face movements, where small movement patterns, so called action units, need to be preserved for further image analysis. We present a novel approach to capture the characteristics of local motion patterns that is based on the brightness constancy equation of optical flow estimation in combination with feature extraction using translation invariant non-negative sparse coding. Our approach simultaneously learns basic motion patterns and estimates the flow field without requiring pretrained motion patterns from ground truth optical flow data. We show on a face expression dataset how this method can preserve weak movements even in the presence of large head movements.

We propose a highly compact, generic representation of the driving environment, so-called Parametric Free Space (PFS) maps, specifically suitable for future Advanced Driver Assistance Systems (ADAS) and bring them into line with existing metric representations known from mobile robotics. PFS maps combine a closed contour description of arbitrarily shaped outer free space boundaries with a representation of inner free space boundaries, respectively objects, by geometric primitives. A real-time capable algorithm is presented that obtains the representation by building upon an intermediate occupancy grid map-based environment representation generated from an automotive radar and a stereo camera. The proposed representation preserves all relevant information contained in a grid map - thus remaining function-independent - in a much more compact way, which is considered particularly important for the transmission via low data-rate automotive communication interfaces. Examples are shown in real traffic scenarios.

DisCoverage is a distributed strategy for frontier-based multi-robot exploration. The robots coordinate by a partition of the environment, and choose their target points by optimizing a locally decomposable objective function. In [9] DisCoverage for convex regions was proposed. In this work, we extend DisCoverage to support arbitrary non-convex real-world environments with obstacles. Therefore, we introduce a transformation of non-convex environments to robot centric star-shaped domains. This results in a general solution with broader applications for exploration and path planning. Simulations as well as experiments with real robots demonstrate the exploration progress.

The main aspect in multi-robot exploration is the efficient coordination of a group of robots. Inspired by previous results on the coverage problem, we propose a novel, frontier-based approach for multi-robot exploration. This approach merges the step of choosing appropriate target points with the step of planning a collision-free path. This is achieved by optimizing an objective function consisting of distance and orientation costs as well as an estimated information gain. The optimization yields motion control laws directly solving the exploration task. Using a Voronoi partition of the environment ensures, that each robot autonomously creates and optimizes the objective function to obtain a collision-free path in a distributed fashion. Simulations demonstrate the effectiveness of our approach.

The pixel movements in an image sequence grabbed by a camera that is mounted on a mobile platform comprise the superposition of several motion components. These motion components are caused by the egomotion of the camera and by the different movements of the objects seen by the camera. Utilizing sensory information from a calibrated stereo rig and egomotion measurements of the mobile platform we develop a probabilistic framework that estimates optical flow relative to the visual flow induced by the egomotion. Despite rapid egomotion changes and a large range of pixel movements the proposed Dynamic Bayesian Network allows to infer the optical flow induced by moving objects. This is used to segregate moving individuals from static background while the stereo rig is moving. We present optical flow and figure-background segmentation results by applying this general framework to image sequences captured by the humanoid robot ASIMO while he is walking and observing moving people. 1

The basic idea of Lucas and Kanade is to constrain the local motion measurement by assuming a constant velocity within a spatial neighborhood. We reformulate this spatial constraint in a probabilistic way assuming Gaussian distributed uncertainty in spatial identification of velocity measurements and extend this idea to scale and time dimensions. Thus, we are able to combine uncertain velocity measurements observed at different image scales and positions over time. We arrive at a new recurrent optical flow filter formulated in a Dynamic Bayesian Network applying suitable factorisation assumptions and approximate inference techniques. The introduction of spatial uncertainty allows for a dynamic and spatially adaptive tuning of the constraining neighborhood. Here, we realize this tuning dependenton the local Structure Tensor of the intensity patterns of the image sequence. We demonstrate that a probabilistic combination of spatiotemporal integration and modulation of a purely local integration area improves the Lucas and Kanade estimation.

For the interaction of a mobile robot with a dynamic environment, the estimation of object motion is desired while the robot is walking and/or turning its head. In this paper, we describe a system which manages this task by combining depth from a stereo camera and computation of the camera movement from robot kinematics in order to stabilize the camera images. Moving objects are detected by applying optical flow to the stabilized images followed by a filtering method, which incorporates both prior knowledge about the accuracy of the measurement and the uncertainties of the measurement process itself. The efficiency of this system is demonstrated in a dynamic real-world scenario with a walking humanoid robot.

We develop an optical flow estimation framework that focuses on motion estimation over time formulated in a dynamic Bayesian network. It realizes a spatiotemporal integration of motion information using a dynamic and robust prior that incorporates spatial and temporal coherence constraints on the flow field. The main contribution is the embedding of these particular assumptions on optical flow evolution into the Bayesian propagation approach that leads to a computationally feasible two-filter inference method and is applicable for on and offline parameter optimization. We analyse the possibility to optimize imposed Student's t-distributed model uncertainties, which are the camera noise and the transition noise. Experiments with synthetic sequences illustrate how the probabilistic framework improves the optical flow estimation because it allows for noisy data, motion ambiguities and motion discontinuities.

In this paper we present an approach for probabilistic motion pattern segmentation. We combine level-set methods for image segmentation with motion estimations based on probability distribution functions (pdf's) calculated at each image position. To this end, we extend a region based level-set framework to exploit the motion pdf's. We then compare segmentation results of the pdf-based with those of optical-flow-based motion segmentation approaches. We found that the straightforward way of characterizing the segmented region by spatially averaging the motion measurement pdf's does not yield satisfactory results. However, describing the spatial characteristics of the motion pdf's with nonparametric density estimates enables to solve complex motion segmentation problems. In particular for situations with demanding motion patterns like partly overlapping objects and transparent motion, we show that the probabilistic approach yields better results. This confirms the idea that for motion processing it is beneficial to consistently retain the uncertainty and ambiguity of the measurement process right up to the final integration stage, instead of directly processing optical flow vectors.

Velocity distributions are an enhanced representation of image velocity containing more velocity information than velocity vectors. In particular, non-Gaussian velocity distributions allow for the representation of ambiguous motion information caused by the aperture problem or multiple motions at motion boundaries. To resolve motion ambiguities, discrete non-Gaussian velocity distributions are suggested, which are integrated over space, time, and scales using a joint Bayesian prediction and refinement approach. This leads to a hierarchical velocity-distribution representation from which robust velocity estimates for both slow and high speeds as well as statistical confidence measures rating the velocity estimates can be computed.

This paper proposes a biologically inspired and technically implemented sound localization system to robustly estimate the position of a sound source in the frontal azimuthal half-plane. For localization, binaural cues are extracted using cochleagrams generated by a cochlear model that serve as input to the system. The basic idea of the model is to separately measure interaural time differences and interaural level differences for a number of frequencies and process these measurements as a whole. This leads to two-dimensional frequency versus time-delay representations of binaural cues, so-called activity maps. A probabilistic evaluation is presented to estimate the position of a sound source over time based on these activity maps. Learned reference maps for different azimuthal positions are integrated into the computation to gain time-dependent discrete conditional probabilities. At every timestep these probabilities are combined over frequencies and binaural cues to estimate the sound source position. In addition, they are propagated over time to improve position estimation. This leads to a system that is able to localize audible signals, for example human speech signals, even in reverberating environments.