In this work we aim to bridge remote sensing scene understanding and mapping on Earth and Mars. Accordingly, we present common challenges to both domains, like an abundance of data, class imbalances, and the scarcity of high resolution ground-truth data. Additionally, we introduce a novel multi-modal semantic segmentation dataset which is based on a geologic map of a rover landing site on Mars and distill the aforementioned challenges into a single dataset. We present initial results and discuss how well a deep neural network can learn to create geologic maps from orbital images of Mars.

Accurately detecting craters in remotely sensed images is an important task when analysing the properties of planetary bodies. Commonly, only large craters in the range of several kilometres are detected. In this work we provide the first example of automatically detecting tiny craters in the range of several meters with the help of a deep neural network by using only a small set of annotated craters. Additionally, we propose a novel way to group overlapping detections and replace the commonly used non-maximum suppression with a probabilistic treatment. As a result, we receive valuable uncertainty estimates of the detections and the aggregated detections are shown to be vastly superior.

Semantic segmentation on radar point clouds is a new challenging task in radar data processing. We demonstrate how this task can be performed and provide results on a large data set of manually labeled radar reflections. In contrast to previous approaches where generated feature vectors from clustered reflections were used as an input for a classifier, now the whole radar point cloud is used as an input and class probabilities are obtained for every single reflection. We thereby eliminate the need for clustering algorithms and manually selected features.

Recent image segmentation methods focus mostly on the topic of semantic segmentation and are trained in a supervised fashion. This work proposes a novel and unsupervised bayesian segmentation method, which includes the edges of an image as part of the model. This reduces typical noise patterns in unsupervised segmentation and increases the overall capability of the segmentation. Two ways are proposed to include edges. One way is a passive edge model which grades the segmentation according to a precomputed edge map, and a second variant where this method is used in conjunction with an active edge movement scheme. Both methods are tested on a publicly available dataset, compared to methods from the literature, and the benefit of including edges is emphasized.

Radar has a long tradition in driver assistance systems. One of its advantages is its robustness against certain light and weather conditions. It is usually used to track surrounding objects and to detect objects which can be utilized for assistance functions, e.g. for emergency braking. However for autonomous driving, a deeper understanding of the vehicle's surroundings is necessary. In this paper we show that semantic knowledge can be obtained from a radar grid by classifying the containing objects on the cell level. This allows for omitting a prior object extraction and results directly in a semantic radar grid. We show on a data set recorded with four radar sensors that this approach produces good results distinguishing cars and other objects.

Linear unmixing is a widely known and frequently applied technique in remote sensing, with the purpose of identifying the combination of materials that compose the multispectral reflection of a measured pixel. In this paper, we apply the same concept to surface reflectance modeling by estimating a reflectance under arbitrary conditions based on a linear combination of 100 basic BRDFs. These so-called end members are taken from the MERL Database of densely sampled BRDF data of different materials (data available at: http://www.merl.com/brdf/). It is shown that the linear unmixing is an effective method to reproduce real reflection behavior accurately. Furthermore, the linearly unmixed BRDF was utilized as a lookup table for Photometric Stereo based surface reconstruction. The reconstruction accuracy is compared with two exemplarily chosen, commonly used parametric BRDF models, the Lambert+Blinn and the Lambert Cook-Torrance model. The linearly mixed BRDF results in accurately reconstructed surfaces and furthermore proves to be a robust method compared to the analytical models.

In this paper, we present an algorithm for the fusion of surface normals estimated based on Shape from Shading with absolute depth data under exploitation of the mutual advantages, regarding non-Lambertian surfaces with non-uniform albedos. While photometric 3D reconstruction methods yield dense surface detail information which is reliable on small scales, active range scanning provides absolute depth data which are typically noisy on small scales but reliable on large scales. The proposed algorithm applies an iterative refinement to the reconstructed surface in order to suppress errors that result from measurement uncertainties in the surface normals and the absolute depth data by simultaneous minimization of a global error functional. The obtained surface is the best fit to the observed image intensities and depth data. We apply our framework to small-scale real-world objects and to regions of the lunar surface.

The recognition and prediction of intersection situations and an accompanying threat assessment are an indispensable skill of future driver assistance systems. This study focuses on the recognition of situations involving two vehicles at intersections. For each vehicle, a set of possible future motion trajectories is estimated and rated based on a motion database for a time interval of 2-4 s ahead. Possible situations involving two vehicles are generated by a pairwise combination of these individual motion trajectories. An interaction model based on the mutual visibility of the vehicles and the assumption that a driver will attempt to avoid a collision is used to rate possible situations. The correspondingly favoured situations are classified with a probabilistic framework. The proposed method is evaluated on a real-world differential GPS data set acquired during a test drive of about 10 km, including three road intersections. Our method is typically able to recognise the situation correctly about 1.5-3 s before the last vehicle has passed its minimum distance to the centre of the intersection.

The recognition and prediction of situations is an indispensable skill of future driver assistance systems. This study focuses on the recognition of situations involving two vehicles at intersections. For each vehicle, a set of possible future motion trajectories is estimated and rated based on a motion database for a time interval of 2-4 seconds ahead. Realistic situations are generated by a pairwise combination of these individual motion trajectories and classified according to nine categories with a polynomial classifier. In the proposed framework, situations are penalised for which the time to collision significantly exceeds the typical human reaction time. The correspondingly favoured situations are combined by a probabilistic framework, resulting in a more reliable situation recognition and collision detection than obtained based on independent motion hypotheses. The proposed method is evaluated on a real-world differential GPS data set acquired during a test drive of 10 km, including three road intersections. Our method is typically able to recognise the situation correctly about 1-2 seconds before the distance to the intersection centre becomes minimal.

The recognition of potentially hazardous situations on road intersections is an indispensable skill of future driver assistance systems. In this context, this study focuses on the task of vehicle tracking in combination with a long-term motion prediction (1-2 s into the future) in a dynamic scenario. A motion-attributed stereo point cloud obtained using computationally efficient feature-based methods represents the scene, relying on images of a stereo camera system mounted on a vehicle. A two-stage mean-shift algorithm is used for detection and tracking of the traffic participants. A hierarchical setup depending on the history of the tracked object is applied for prediction. This includes prediction by optical flow, a standard kinematic prediction, and a particle filter based motion pattern method relying on learned object trajectories. The evaluation shows that the proposed system is able to track the road users in a stable manner and predict their positions at least one order of magnitude more accurately than a standard kinematic prediction method.

In this contribution we introduce the Shape Flow algorithm (SF), a novel method for spatio-temporal 3D pose estimation of a 3D parametric curve. The SF is integrated into a tracking system and its suitability for tracking human body parts in 3D is examined. Based on the example of tracking the human hand-forearm limb it is shown that the use of two SF instances with different initializations leads to an accurate and temporally stable tracking system. In our framework, the temporal pose derivative is available instantaneously, therefore we avoid delays typically encountered when filtering the pose estimation results over time. All necessary information is obtained from the images, only a coarse initialisation of the model parameters is required. Experimental investigations are performed on 5 real-world test sequences showing 3 different test persons in an average distance of 1.2-3.3 m to the camera in front of cluttered background. We achieve typical pose estimation accuracies of 40-100 mm for the mean distance to the ground truth and 4-6 mm for the pose differences between subsequent images.

In this study we propose an integrated approach to the problem of 3D pose estimation. The main difference to the majority of known methods is the usage of complementary image information, including intensity and polarisation state of the light reflected from the object surface, edge information, and absolute depth values obtained based on a depth from defocus approach. Our method is based on the comparison of the input image to synthetic images generated by an OpenGL-based renderer using model information about the object provided by CAD data. This comparison provides an error term which is minimised by an iterative optimisation algorithm. Although all six degrees of freedom are estimated, our method requires only a monocular camera, circumventing disadvantages of multiocular camera systems such as the need for external camera calibration. Our framework is open for the inclusion of independently acquired depth data. We evaluate our method on a toy example as well as in two realistic scenarios in the domain of industrial quality inspection. Our experiments regarding complex real-world objects located at a distance of about 0.5 m to the camera show that the algorithm achieves typical accuracies of better than 1 degree for the rotation angles, 1-2 image pixels for the lateral translations, and several millimetres or about 1 percent for the object distance.

We address the problem of image sequence analysis by classification. As an example we consider the recognition of walking pedestrians in complex traffic scenes. Polynomial support vector machines are applied to complete image sequences, representing extremely high-dimensional input patterns, and to reduced feature sets obtained by standard "global" principal component analysis. These approaches are compared to the adaptable time delay neural network (ATDNN) algorithm based on receptive fields that perform a "local" spatio-temporal processing of the image sequence, generating feature sets that are classified by polynomial support vector-machines in an extended version of the ATDNN algorithm. The computational complexity of the local approaches is up to two and the memory demand up to four orders of magnitude lower than the corresponding values for the global approaches while the recognition performance of the local approaches is even higher than that of the global ones.

In this letter we present an algorithm based on a time-delay neural network with spatio-temporal receptive fields and adaptable time delays for image sequence analysis. Our main result is that tedious manual adaptation of the temporal size of the receptive fields can be avoided by employing a novel method to adapt the corresponding time delay and related network structure parameters during the training process.

In this paper we present an algorithm for recognizing walking pedestrians in sequences of color images taken from a moving camera. The recognition is based on the characteristic motion of the legs of a pedestrian walking parallel to the image plane. Each image is segmented into region-like image parts by clustering pixels in a combined color/position feature space. The proposed clustering technique implies matching of corresponding clusters in consecutive frames and therefore allows clusters to be tracked over a sequence of images. Based on the observation of clusters over time a two-stage classifier extracts those clusters which most likely represent the legs of pedestrians. A fast polynomial classifier performs a rough preselection of clusters by evaluating temporal changes of a shape-dependent clusters feature. The final classification is done by a time delay neural network (TDNN) with spatio-temporal receptive fields.