Abstract Despite ongoing research on the topic of adversarial examples in deep learning for computer vision, some fundamentals of the nature of these attacks remain unclear. As the manifold hypothesis posits, high-dimensional data tends to be part of a low-dimensional manifold. To verify the thesis with adversarial patches–a special form of adversarial attack that can be used to fool object detectors in the physical world–this paper provides an analysis of a set of adversarial patches and investigates the reconstruction abilities of five different dimensionality reduction methods. Quantitatively, the performance of reconstructed patches in an attack setting is measured and the impact of sampled patches from the latent space during adversarial training is investigated. The evaluation is performed on two publicly available datasets for person detection. The results indicate that more sophisticated dimensionality reduction methods offer no advantages over a simple principal component analysis.

This paper proposes a novel method for vision-based metric cross-view geolocalization (CVGL) that matches the camera images captured from a ground-based vehicle with an aerial image to determine the vehicle's geo-pose. Since aerial images are globally available at low cost, they represent a potential compromise between two established paradigms of autonomous driving, i.e. using expensive high-definition prior maps or relying entirely on the sensor data captured at runtime. We present an end-to-end differentiable model that uses the ground and aerial images to predict a probability distribution over possible vehicle poses. We combine multiple vehicle datasets with aerial images from orthophoto providers on which we demonstrate the feasibility of our method. Since the ground truth poses are often inaccurate w.r.t. the aerial images, we implement a pseudo-label approach to produce more accurate ground truth poses and make them publicly available. While previous works require training data from the target region to achieve reasonable localization accuracy (i.e. same-area evaluation), our approach overcomes this limitation and outperforms previous results even in the strictly more challenging cross-area case. We improve the previous state-of-the-art by a large margin even without ground or aerial data from the test region, which highlights the model's potential for global-scale application. We further integrate the uncertainty-aware predictions in a tracking framework to determine the vehicle's trajectory over time resulting in a mean position error on KITTI-360 of 0.78m.

While current methods for interactive Video Object Segmentation (iVOS) rely on scribble-based interactions to generate precise object masks, we propose a Click-based interactive Video Object Segmentation (CiVOS) framework to simplify the required user workload as much as possible. CiVOS builds on de-coupled modules reflecting user interaction and mask propagation. The interaction module converts click-based interactions into an object mask, which is then inferred to the remaining frames by the propagation module. Additional user interactions allow for a refinement of the object mask. The approach is extensively evaluated on the popular interactive DAVIS dataset, but with an inevitable adaptation of scribble-based interactions with click-based counterparts. We consider several strategies for generating clicks during our evaluation to reflect various user inputs and adjust the DAVIS performance metric to perform a hardware-independent comparison. The presented CiVOS pipeline achieves competitive results, although requiring a lower user workload.

Despite their black box nature, deep neural networks have been successfully used in practical applications lately. In areas where the results of these applications can lead to safety hazards or decisions of ethical relevance, the application provider is accountable for the resulting decisions and should therefore be able to explain, how, and why a specific decision was made. For image processing networks, saliency map generators are a possible solution. A saliency map gives a visual hint on what is of special importance for the network’s decision, can reveal possible dataset biases and give a more profound insight in the decision process of the black box.This paper investigates how 2D saliency map generators need to be adapted for 3D input data, and additionally, how the methods behave when applied not only to ordinary video input but rather multispectral 3D input data. This is exemplarily shown on 3D video input data in human action recognition in the infrared and visual spectrum and evaluated by using the insertion and deletion metrics. The dataset used in this work is the Multispectral Action Dataset, where each scene is available in the long-wave infrared as well as the visual spectrum. To be able to draw a more general conclusion, the two investigated networks, 3D-ResNet 18 and Persistent Appearance Network (PAN), follow a different mindset.It could be shown, that the saliency methods can also be applied to 3D input data with remarkable results. The results show that a combined training with both, infrared and RGB 3D input data, lead to more focused saliency maps and outperform a training with only RGB or infrared data.

This paper proposes a novel method for geo-tracking, i.e. continuous metric self-localization in outdoor environments by registering a vehicle's sensor information with aerial imagery of an unseen target region. Geo- tracking methods offer the potential to supplant noisy signals from global navigation satellite systems (GNSS) and expensive and hard to maintain prior maps that are typically used for this purpose. The proposed geo-tracking method aligns data from on-board cameras and lidar sensors with geo-registered orthophotos to continuously localize a vehicle. We train a model in a metric learning setting to extract visual features from ground and aerial images. The ground features are projected into a top-down perspective via the lidar points and are matched with the aerial features to determine the relative pose between vehicle and orthophoto. Our method is the first to utilize on-board cameras in an end-to-end differentiable model for metric self-localization on unseen orthophotos. It exhibits strong generalization, is robust to changes in the environment and requires only geo-poses as ground truth. We evaluate our approach on the KITTI-360 dataset and achieve a mean absolute position error (APE) of 0.94m. We further compare with previous approaches on the KITTI odometry dataset and achieve state-of-the-art results on the geo-tracking task.3

Representations of sequential data are commonly based on the assumption that observed sequences are realizations of an unknown underlying stochastic process, where the learning problem includes determination of the model parameters. In this context, a model must be able to capture the multi-modal nature of the data, without blurring between single modes. This paper proposes probabilistic B'{e}zier curves (

Statistical models of the human body surface are generally learned from thousands of high-quality 3D scans in predefined poses to cover the wide variety of human body shapes and articulations. Acquisition of such data requires expensive equipment, calibration procedures, and is limited to cooperative subjects who can understand and follow instructions, such as adults. We present a method for learning a statistical 3D Skinned Multi-Infant Linear body model (SMIL) from incomplete, low-quality RGB-D sequences of freely moving infants. Quantitative experiments show that SMIL faithfully represents the RGB-D data and properly factorizes the shape and pose of the infants. To demonstrate the applicability of SMIL, we fit the model to RGB-D sequences of freely moving infants and show, with a case study, that our method captures enough motion detail for General Movements Assessment (GMA), a method used in clinical practice for early detection of neurodevelopmental disorders in infants. SMIL provides a new tool for analyzing infant shape and movement and is a step towards an automated system for GMA.

Three-dimensional environment perception is a key element of autonomous driving and driver assistance systems. A common image based approach to determine three-dimensional scene information is stereo matching, which is limited by the stereo camera baseline. In contrast to stereo matching based methods, we present an approach to reconstruct three-dimensional object trajectories combining temporal adjacent views for object point triangulation. We track two-dimensional object shapes on pixel level exploiting instance-aware semantic segmentation techniques and optical flow cues. We apply Structure from Motion (SfM) to object and background images to determine initial camera poses relative to object instances as well as background structures and refine the initial SfM results by integrating stereo camera constraints using factor graphs. We compute object trajectories using stereo sequence constraints of object and background reconstructions. We show qualitative results using publicly available video data of driving sequences. Due to the lack of suitable ground truth, we create a synthetic benchmark dataset of stereo sequences with vehicles in urban environments. Our algorithm achieves an average trajectory error of 0.09 meter using the dataset. The dataset is on our website1publicly available.

The precise determination of correspondences between pairs of images is still a fundamental building block of many computer vision systems. Despite the maturity of modern feature matchers, multispectral methods are still lacking robustness and speed. We focus on the problem of finding point correspondences in a multispectral imaging setup. Most methods aim at invariant feature transforms (e.g. multi-modal descriptors) which come at the cost of reduced discriminance.We model the appearance change by learning an image transformation, which maps one image modality to the respective target image, conditioned on the data of the original spectral band. This approach is coupled with a pipeline of state of the art matching methods with view synthesis of increasing complexity and algorithm run-time.We evaluate the approach on a wide spectrum of multispectral datasets including near-infrared, color-infrared and night and day thermal infrared imagery. The proposed approach provides significant improvements in terms of speed and robustness compared to standard multi-modal registration approaches. In addition, the approach fits very well into existing system approaches by design.Applications are numerous and include multispectral sensor fusion, multispectral odometry systems, multispectral segmentation or multispectral super-resolution methods.

Visual object tracking is a challenging task in computer vision, especially if there are no constraints to the scenario and the objects are arbitrary. The number of tracking algorithms is very large and all have diverse advantages and disadvantages. Normally they show various behaviour and their failures in the tracking process occur at different moments in the sequence. So far, there is no tracker which can solve all scenarios robustly and accurately. One possible approach to this problem is using a whole collection of tracking algorithms and fusing them. There exist various strategies to fuse tracking algorithms. In some of them only the resulting outputs are fused. This means that new algorithms can be integrated with less effort. This fusion can be called “high-level” because the tracking algorithms only interact through the last step in their procedure. Three fusion methods are investigated. They are called Weighted mean fusion, MAD fusion and attraction field fusion. In order to evaluate the three different approaches a collection of thermal image sequences has been investigated. These sequences show maritime scenarios with various objects such as ships and other vessels.

The quadratic system provided by the Time of Arrival technique can be solved analytical or by non-linear least squares minimization. In real environments the measurements are always corrupted by noise. This measurement noise effects the analytical solution more than non-linear optimization algorithms. On the other hand it is also true that local optimization tends to find the local minimum, instead of the global minimum. This article presents an approach how this risk can be significantly reduced in noisy environments. The main idea of our approach is to transform the local minimum to a saddle point, by increasing the number of dimensions. In addition to numerical tests we analytically prove the theorem and the criteria that no other local minima exists for non-trivial constellations.

We present a method to perform online Multiple Object Tracking (MOT) of known object categories in monocular video data. Current Tracking-by-Detection MOT approaches build on top of 2D bounding box detections. In contrast, we exploit state-of-the-art instance aware semantic segmentation techniques to compute 2D shape representations of target objects in each frame. We predict position and shape of segmented instances in subsequent frames by exploiting optical flow cues. We define an affinity matrix between instances of subsequent frames which reflects locality and visual similarity. The instance association is solved by applying the Hungarian method. We evaluate different configurations of our algorithm using the MOT 2D 2015 train dataset. The evaluation shows that our tracking approach is able to track objects with high relative motions. In addition, we provide results of our approach on the MOT 2D 2015 test set for comparison with previous works. We achieve a MOTA score of 32.1.

Online multi-person tracking benefits from using an online learned appearance model to associate detections to tracks and further to close gaps in detections. Since Integral Channel Features (ICF) are popular for fast pedestrian detection, we propose an online appearance model that is using the same features without recalculation. The proposed method uses online Multiple-Instance Learning (MIL) to incrementally train an appearance model for each person discriminating against its surrounding. We show that a low number of discriminatingly selected Integral Channel Features are sufficient to achieve state-of-the-art results on the MOT2015 and MOT2016 benchmark.

We present a method to reconstruct the three-dimensional shape of a moving instance of a known object category in video data. We exploit state-of-the-art semantic segmentation techniques to extract the object's two-dimensional shape in each frame. Therefore, our method is robust to occlusion, handles stationary objects and extends naturally to multiple video sequences. We apply Structure from Motion (SfM) to previously generated object images in order to compute a three-dimensional representation of the object. Our approach allows us to remove outliers in SfM reconstructions and to compute clean object meshes by leveraging previously computed semantic segmentations and virtual camera positions. We evaluate the accuracy of our method using a multi-view dataset of a moving vehicle. A laser scan serves as ground truth. We applied our algorithm on publicly available video data and on 25 sequences from our dataset. The algorithm achieves an average point distance of 3.3 cm evaluated on seven trajectories contained in the dataset.

Appearance-based action recognition can be considered as a natural extension of appearance-based object detection from the spatial to the spatio-temporal domain. Although this step seems natural, most action recognition approaches are evaluated in isolation. Towards this end the contribution of this paper is twofold. First, a view-independent approach to action recognition is proposed and second the sensitivity w.r.t. a combination of person detection and action recognition is evaluated. Action recognition is performed in a hierarchical manner: First, the relative camera orientation in the scene is estimated and second, the action is determined using view-dependent Hough forests. The proposed approach is evaluated on the multi-view i3DPost dataset [1] and its performance is compared to single-step approaches using Hough forests. The results suggest that the recognition rate increases, when using the proposed hierarchical method compared to single-step approaches. Further, the performance rates of hierarchical Hough forests on ground truth data are compared to the results of hierarchical Hough forests in combination with a person detector.

The Local Position Measurement system (LPM) is one of the most precise systems for 3D position estimation. It is able to operate in- and outdoor and updates at a rate up to 1000 measurements per second. Previous scientific publications focused on the time of arrival equation (TOA) provided by the LPM and filtering after the numerical position estimation. This paper investigates the advantages of the TOA over the time difference of arrival equation transformation (TDOA) and the signal smoothing prior to its fitting. The LPM was designed under the general assumption that the position of the base station and position of the reference station are known. The information resulting from this research can prove vital for the system's self-calibration, providing data aiding in locating the relative position of the base station without prior knowledge of the transponder and reference station positions.

Action recognition in surveillance systems has to work 24/7 under all kinds of weather and lighting conditions. Towards this end, most action recognition systems only work in the visible spectrum which limits their general usage to daytime applications. In this work Hough forests are applied to the longwave infrared spectrum which can capture humans both in the dark and in daylight. Further, Integral Channel Features which have shown promising results in the spatial domain are applied to the spatio-temporal domain and are incorporated into the Hough forest approach. This approach is evaluated on a new outdoor dataset containing different violent and non-violent actions recorded in the visible and infrared spectrum. It is further shown that for the visible spectrum the proposed approach achieves state-of-the-art results on the KTH and i3DPost dataset.

In this paper, we approach the task of appearance based person re-identification for scenarios where no biometric features can be used. For that, we build on a person re-identification approach that uses the Implicit Shape Model (ISM) and SIFT features for re-identification. This approach builds identity models of persons during tracking and employs these models for re-identification. We apply this re-identification, which was until now only evaluated in the infrared spectrum, to data acquired in the visible spectrum. Furthermore we evaluate view independence of the re-identification approach and introduce methods that extend view invariance. Specifically, we (i) propose a method for online view-determination of a tracked person, (ii) use the online view-determination to generate view specific identity models of persons which increase model distinctiveness in re-identification, and (iii) introduce a method to convert identity models between views to increase view independence.

In this paper, we address the task of appearance based person reidentification in infrared image sequences. While common approaches for appearance based person reidentification in the visible spectrum acquire color histograms of a person, this technique is not applicable in infrared for obvious reasons. To tackle the more difficult problem of person reidentification in infrared, we introduce an approach that relies on local image features only and thus is completely independent of sensor specific features which might be available only in the visible spectrum. Our approach fits into an Implicit Shape Model (ISM) based person detection and tracking strategy described in previous work. Local features collected during tracking are employed for person reidentification while the generalizing appearance codebook used for person detection serves as structuring element to generate person signatures. By this, we gain an integrated approach that allows for fast online model generation, a compact representation, and fast model matching. Since the model allows for a joined representation of appearance and spatial information, no complex representation models like graph structures are needed. We evaluate our person reidentification approach on a subset of the CASIA infrared dataset.

A fundamental problem in computer vision is the precise determination of correspondences between pairs of images. Many methods have been proposed which work very well for image data from one modality. However, with the wide availability of sensor systems with different spectral sensitivities there is growing demand to automatically fuse the information from multiple sensor types. We focus on the problem of finding point and local region correspondences in an inter-modality imaging setup. We use a Generalized Hough Transform to determine small regions with a similar geometric relationship of local image features to robustly identify correct matches. We additionally optimize region correspondences by a fast non-linear optimization of a self-similarity distance measure. This measure outperforms standard multi-modal registration approaches like mutual information or correlation ratio in case of local image regions. The method is evaluated on Visible/Infrared (IR) and Visible/Light Detection and Ranging (LiDAR) intensity image data pairs and shows very promising results. Potential applications are numerous and include for instance multi-spectral camera calibration, multi-spectral texturing of 3D-models, multi-spectral segmentation or multi-spectral super-resolution.

The automatic detection and tracking of pedestrians in imagery constitute important and challenging problems both in computer vision and driver assistance systems. We address these problems for the case of a forward looking monocular infrared camera under strong vehicle induced camera motion. An integrated detection & tracking strategy is introduced based on a state-of-the-art feature based object detector originally developed for images in the visual spectrum. The proposed pedestrian detection algorithm can be applied to both infrared and visual imagery. We show the difficulties arising from the specifics of infrared data under strong camera motion and how to tackle these problems by replacing common motion models like the Kalman filter by a feature matching approach.