Abstract Despite ongoing research on the topic of adversarial examples in deep learning for computer vision, some fundamentals of the nature of these attacks remain unclear. As the manifold hypothesis posits, high-dimensional data tends to be part of a low-dimensional manifold. To verify the thesis with adversarial patches–a special form of adversarial attack that can be used to fool object detectors in the physical world–this paper provides an analysis of a set of adversarial patches and investigates the reconstruction abilities of five different dimensionality reduction methods. Quantitatively, the performance of reconstructed patches in an attack setting is measured and the impact of sampled patches from the latent space during adversarial training is investigated. The evaluation is performed on two publicly available datasets for person detection. The results indicate that more sophisticated dimensionality reduction methods offer no advantages over a simple principal component analysis.

Despite their black box nature, deep neural networks have been successfully used in practical applications lately. In areas where the results of these applications can lead to safety hazards or decisions of ethical relevance, the application provider is accountable for the resulting decisions and should therefore be able to explain, how, and why a specific decision was made. For image processing networks, saliency map generators are a possible solution. A saliency map gives a visual hint on what is of special importance for the network’s decision, can reveal possible dataset biases and give a more profound insight in the decision process of the black box.This paper investigates how 2D saliency map generators need to be adapted for 3D input data, and additionally, how the methods behave when applied not only to ordinary video input but rather multispectral 3D input data. This is exemplarily shown on 3D video input data in human action recognition in the infrared and visual spectrum and evaluated by using the insertion and deletion metrics. The dataset used in this work is the Multispectral Action Dataset, where each scene is available in the long-wave infrared as well as the visual spectrum. To be able to draw a more general conclusion, the two investigated networks, 3D-ResNet 18 and Persistent Appearance Network (PAN), follow a different mindset.It could be shown, that the saliency methods can also be applied to 3D input data with remarkable results. The results show that a combined training with both, infrared and RGB 3D input data, lead to more focused saliency maps and outperform a training with only RGB or infrared data.

Appearance-based action recognition can be considered as a natural extension of appearance-based object detection from the spatial to the spatio-temporal domain. Although this step seems natural, most action recognition approaches are evaluated in isolation. Towards this end the contribution of this paper is twofold. First, a view-independent approach to action recognition is proposed and second the sensitivity w.r.t. a combination of person detection and action recognition is evaluated. Action recognition is performed in a hierarchical manner: First, the relative camera orientation in the scene is estimated and second, the action is determined using view-dependent Hough forests. The proposed approach is evaluated on the multi-view i3DPost dataset [1] and its performance is compared to single-step approaches using Hough forests. The results suggest that the recognition rate increases, when using the proposed hierarchical method compared to single-step approaches. Further, the performance rates of hierarchical Hough forests on ground truth data are compared to the results of hierarchical Hough forests in combination with a person detector.

Action recognition in surveillance systems has to work 24/7 under all kinds of weather and lighting conditions. Towards this end, most action recognition systems only work in the visible spectrum which limits their general usage to daytime applications. In this work Hough forests are applied to the longwave infrared spectrum which can capture humans both in the dark and in daylight. Further, Integral Channel Features which have shown promising results in the spatial domain are applied to the spatio-temporal domain and are incorporated into the Hough forest approach. This approach is evaluated on a new outdoor dataset containing different violent and non-violent actions recorded in the visible and infrared spectrum. It is further shown that for the visible spectrum the proposed approach achieves state-of-the-art results on the KTH and i3DPost dataset.

Image-based 6-DoF pose estimation of arbitrarily shaped 3D objects based on their shape is a rarely studied problem. Most existing image-based methods for pose estimation either exploit textural information in form of local features or, if shape-based, rely on the extraction of straight line segments or other primitives. Straight-forward extensions of 2D approaches are potentially more general, but in practice assume a limited range of possible view angles. The general problem is that a 3D object can potentially produce completely different 2D projections depending on its relative pose to the observing camera. One way to reduce the solution space is to exploit temporal information, i.e. perform tracking. Again, existing model-based tracking approaches rely on relatively simple object geometries. In this paper, we propose a particle filter based tracking approach that can deal with arbitrary shapes and arbitrary or even no texture, i.e. it offers a general solution to the rigid object tracking problem. As our approach can deal with occlusions, it is in particular of interest in the context of goal-directed imitation learning involving the observation of object manipulations. Results of simulation experiments as well as real-world experiments with different object types prove the practical applicability of our approach.