Vehicle detection in aerial images is a crucial image processing step for many applications such as screening of large areas as used for surveillance, reconnaissance, or rescue tasks. In recent years, several deep learning-based frameworks have been proposed for object detection. However, these detectors were developed for data sets that considerably differ from aerial images. In this paper, we systematically investigate the potential of fast R-CNN and faster R-CNN for aerial images, which achieve top performing results on common detection benchmark data sets. Therefore, the applicability of eight state-of-the-art object proposal methods used to generate a set of candidate regions and of both detectors is examined. Relevant adaptations to account for the characteristics of the aerial images are provided. To overcome the shortcomings of the original approach in the case of handling small instances, we further propose our own networks that clearly outperform state-of-the-art methods for vehicle detection in aerial images. Furthermore, we analyze the impact of the different adaptations with respect to various ground sampling distances to provide a guideline for detecting small objects in aerial images. All experiments are performed on two publicly available data sets to account for differing characteristics such as varying object sizes, number of objects per image, and varying backgrounds.

Accurate detection of objects in aerial images is an important task for many applications such as traffic monitoring, surveillance, reconnaissance and rescue tasks. Recently, deep learning based detection frameworks clearly improved the detection performance on aerial images compared to conventional methods comprised of hand-crafted features and a classifier within a sliding window approach. These deep learning based detection frameworks use the output of the last convolutional layer as feature map for localization and classification. Due to the small size of objects in aerial images, only shallow layers of standard models like VGG-16 or small networks are applicable in order to provide a sufficiently high feature map resolution. However, high-resolution feature maps offer less semantic and contextual information, which results in approaches being more prone to false alarms due to objects with similar shapes especially in case of tiny objects. In this paper, we extend the Faster R-CNN detection framework to cope this issue. Therefore, we apply a deconvolutional module that up-samples low-dimensional feature maps of deep layers and combines the up-sampled features with the features of shallow layers while the feature map resolution is kept sufficiently high to localize tiny objects. Our proposed deconvolutional framework clearly outperforms state-of-the-art methods on two publicly available datasets.

Small drones are a rising threat due to their possible misuse for illegal activities, in particular smuggling and terrorism. The project SafeShore, funded by the European Commission under the Horizon 2020 program, has launched the “drone-vs-bird detection challenge” to address one of the many technical issues arising in this context. The goal is to detect a drone appearing at some point in a video where birds may be also present: the algorithm should raise an alarm and provide a position estimate only when a drone is present, while not issuing alarms on birds. This paper reports on the challenge proposal, evaluation, and results.

Recent progress in the development of unmanned aerial vehicles (UAVs) causes serious safety issues for mass events and safety-sensitive locations like prisons or airports. To address these concerns, robust UAV detection systems are required. In this work, we propose an UAV detection framework based on video images. Depending on whether the video images are recorded by static cameras or moving cameras, we initially detect regions that are likely to contain an object by median background subtraction or a deep learning based object proposal method, respectively. Then, the detected regions are classified into UAV or distractors, such as birds, by applying a convolutional neural network (CNN) classifier. To train this classifier, we use our own dataset comprised of crawled and self-acquired drone images, as well as bird images from a publicly available dataset. We show that, even across a significant domain gap, the resulting classifier can successfully identify UAVs in our target dataset. We evaluate our UAV detection framework on six challenging video sequences that contain UAVs at different distances as well as birds and background motion.

Growing cities and increasing traffic densities result in an increased demand for applications such as traffic monitoring, traffic analysis, and support of rescue work. These applications share the need for accurate detection of relevant vehicles, e.g. in aerial imagery. Recently, the application of deep learning based detection frameworks like Faster R-CNN clearly outperformed conventional detection methods for vehicle detection in aerial images. In this paper, we propose a detection framework that fuses Faster R-CNN and semantic labeling to integrate contextual information. We achieve an improved detection performance by decreasing the number of false positive detections while the number of candidate regions to classify is reduced. To demonstrate the generalization of our approach, we evaluate our detection framework for various ground sampling distances on a publicly available dataset.

With the increasing use of unmanned aerial vehicles (UAVs) by consumers, automatic UAV detection systems have become increasingly important for security services. In such a system, video imagery is a core modality for the detection task, because it can cover large areas and is very cost-effective to acquire. Many detection systems consist of two parts: flying object detection and subsequent object classification. In this work, we investigate the suitability of a number of flying object detection approaches for the task of UAV detection based on video data from static and moving cameras. We compare approaches based on image differencing with object proposal detectors which are learned from data. Finally, we classify each detection by a convolutional neural network (CNN) into the classes UAV or clutter. Our approach is evaluated on six sequences of challenging real world data which contain multiple UAVs, birds, and background motion.

Person re-identification (re-id) is the task of matching multiple occurrences of the same person from different cameras, poses, lighting conditions, and a multitude of other factors which alter the visual appearance. Typically, this is achieved by learning either optimal features or distance metrics which are adapted to specific pairs of camera views dictated by the pairwise labelled training datasets. In this work, we formulate a deep learning based novel approach to automatic prototype-domain discovery for domain perceptive person re-id. The approach scales to new and unseen scenes without requiring new training data. We learn a separate re-id model for each of the discovered prototype-domains and during model deployment, use the person probe image to automatically select the model of the closest prototype-domain. Our approach requires neither supervised nor unsupervised transfer learning, i.e. no data available from target domains. Extensive evaluations are carried out using automatically detected bounding boxes with low-resolution and partial occlusion on two large scale re-id benchmarks, CUHK-SYSU and PRW. Our approach outperforms state-of-the-art unsupervised methods significantly and is competitive against supervised methods which use labelled test domain data.

Wide area motion imagery (WAMI) acquired by an airborne sensor enables continuous monitoring of large urban areas. Reliable vehicle tracking in this imagery remains challenging due to low frame rate and small object size. Many approaches solely rely on motion detections provided by frame differencing or background subtraction. Recent approaches for persistent tracking, i.e. tracking vehicles even if they become stationary, compensate for missing motion detections by combining a detection-based tracker with a second tracker based on appearance or local context. We propose a novel single tracker framework based on multiple hypothesis tracking (MHT) that enables persistent tracking in WAMI data by recovering missing motion detections with a classifier-based detector, thus avoiding the additional complexity introduced by combining two trackers. We adapt the MHT approach to the specific context of WAMI tracking by integrating an appearance-based similarity measure, vehicle-collision tests, and clutter handling. An evaluation on a region of interest in the WPAFB 2009 dataset shows state-of-the-art performance.

Incorporating 3D information has proven to be effective in many computer vision tasks and it is no exception in the context of facial analysis. However, limited application has been witnessed in face hallucination (FH), probably due to the difficulty of fitting 3D models onto low-resolution (LR) images. This paper presents a pure 3D approach to address this problem. By extending the LR image formation process to the 3D domain, the classic Lucas-Kanade algorithm is exploited to improve the precision of the error-prone 3D model fitting on LR images. The established correspondence between the input image and 3D training textures then facilitates reconstruction of high-resolution (HR) patches directly on the mesh, which can be employed to render realistic frontal faces for recognition. Extensive evaluation on several publicly available datasets reveals superior qualitative and quantitative results over state-of-the-art methods in fitting, FH and recognition, which shows the advantage of the proposed 3D framework over its 2D rivals, especially for non-frontal head poses and low image quality.

Vehicle detection in aerial images is a crucial image processing step for many applications like screening of large areas. In recent years, several deep learning based frameworks have been proposed for object detection. However, these detectors were developed for datasets that considerably differ from aerial images. In this paper, we systematically investigate the potential of Fast R-CNN and Faster R-CNN for aerial images, which achieve top performing results on common detection benchmark datasets. Therefore, the applicability of 8 state-of-the-art object proposals methods used to generate a set of candidate regions and of both detectors is examined. Relevant adaptations of the object proposals methods are provided. To overcome shortcomings of the original approach in case of handling small instances, we further propose our own network that clearly outperforms state-of-the-art methods for vehicle detection in aerial images. All experiments are performed on two publicly available datasets to account for differing characteristics such as ground sampling distance, number of objects per image and varying backgrounds.

Super-resolution (SR) offers an effective approach to boost quality and details of low-resolution (LR) images to obtain high-resolution (HR) images. Despite the theoretical and technical advances in the past decades, it still lacks plausible methodology to evaluate and compare different SR algorithms. The main cause to this problem lies in the missing ground truth data for SR. Unlike in many other computer vision tasks, where existing image datasets can be utilized directly, or with a little extra annotation work, evaluating SR requires that the dataset contain both LR and the corresponding HR ground truth images of the same scene captured at the same time. This work presents a novel prototype camera system to address the aforementioned difficulties of acquiring ground truth SR data. Two identical camera sensors equipped with a wide-angle lens and a telephoto lens respectively, share the same optical axis by placing a beam splitter in the optical path. The back-end program can then trigger their shutters simultaneously and precisely register the region of interests (ROIs) of the LR and HR image pairs in an automated manner free of sub-pixel interpolation. Evaluation results demonstrate the special characteristics of the captured ground truth HR-LR face images compared to the simulated ones. The dataset is made freely available for noncommercial research purposes.

Wide Area Motion Imagery (WAMI) enables the surveillance of tens of square kilometers with one airborne sensor Each image can contain thousands of moving objects. Applications such as driver behavior analysis or traffic monitoring require precise multiple object tracking that is dependent on initial detections. However, low object resolution, dense traffic, and imprecise image alignment lead to split, merged, and missing detections. No systematic evaluation of moving object detection exists so far although many approaches have been presented in the literature. This paper provides a detailed overview of existing methods for moving object detection in WAMI data. Also we propose a novel combination of short-term background subtraction and suppression of image alignment errors by pixel neighborhood consideration. In total, eleven methods are systematically evaluated using more than 160,000 ground truth detections of the WPAFB 2009 dataset. Best performance with respect to precision and recall is achieved by the proposed one.

Direct reconstruction of 3D face shape—solely based on a sparse set of 2D feature points localized by a facial landmark detector—offers an automatic, efficient and illumination-invariant alternative to the conventional analysis-by-synthesis 3D Morphable Model (3DMM) fitting. In this paper, we propose a novel algorithm that addresses the inconsistent correspondence of 2D and 3D landmarks at the facial contour due to head pose and localization ambiguity along the edge. To facilitate dynamic correspondence while fitting, a small subset of 3D vertices that serves as the contour candidates is annotated offline. During the fitting process, we employ the Levenberg-Marquardt Iterative Closest Point (LM-ICP) algorithm in combination with Distance Transform (DT) within the constrained domain, which allows for fast convergence and robust estimation of 3D face shape against pose variation. Superior evaluation results reported on ground truth 3D face scans over the state-of-the-art demonstrate the efficacy of the proposed method

This paper presents a fully automatic system that recovers 3D face models from sequences of facial images. Unlike most 3D Morphable Model (3DMM) fitting algorithms that simultaneously reconstruct the shape and texture from a single input image, our approach builds on a more efficient least squares method to directly estimate the 3D shape from sparse 2D landmarks, which are localized by face alignment algorithms. The inconsistency between self-occluded 2D and 3D feature positions caused by head pose is ad-dressed. A novel framework to enhance robustness across multiple frames selected based on their 2D landmarks combined with individual self-occlusion handling is proposed. Evaluation on groundtruth 3D scans shows superior shape and pose estimation over previous work. The whole system is also evaluated on an “in the wild” video dataset [12] and delivers personalized and realistic 3D face shape and texture models under less constrained conditions, which only takes seconds to process each video clip.

Deformable model fitting to high-resolution facial images has been extensively studied for over two decades. However, due to the ill-posed problem caused by low-resolution images, most existing work cannot be applied directly and degrades quickly as the resolution decreases. To address this issue, this paper extends the Constrained Local Model (CLM) to a multi-resolution model consisting of a 4-level patch pyramid, and deploys various feature descriptors for the local patch experts as well. We evaluate the proposed work on the BioID, the MUCT and the Multi-PIE datasets. Superior results are achieved on almost all resolution levels, demonstrating the effectiveness and necessity of our resolution-aware approach for the low-resolution fitting. Improved performance of patch models employing several feature combinations over the single intensity feature under different conditions is also presented.

In this paper an evaluation of state-of-the-art binary keypoint descriptors, namely BRIEF, ORB, BRISK and FREAK, is presented. In contrast to previous evaluations we used the Stanford Mobile Visual Search (SMVS) data set because binary descriptors are mainly used in mobile applications. This large data set does provide a lot of characteristic transformations for mobile devices, but no ground truth data. The often used Oxford data set is used only for validation purposes. We use ratio-test and RANSAC (RANdom SAmple Consensus) for evaluation and present results for accuracy, precision and average number of best matches as performance metrics. The validity of the results is also checked by evaluating these binary keypoint descriptors on Oxford data set. The obtained results show that BRISK is the keypoint descriptor which gives highest percentage of precision and largest number of best matches among all the binary descriptors. Next to BRISK is FREAK, which offers comparably good result.

Adaptive behavior is one of the main challenges in building computerized systems, especially in the case of systems which are delivering information to the end users. Indeed, since the information overload has become the main drawback for the future development of such systems (cf. Big Data challenge), there is a huge movement in the research community to develop concepts for better adaptation of the form and size of information that will be delivered to a user (usually taking different forms of the personalization). However, the main effort has been dedicated to the contextualization of the user's task in order to determine what is the best way to tailor/adapt the presentation of information to the user, neglecting the role of the user's internal context, expressed as the user's (short-term) interest. The same is valid for the AR systems. In this tutorial we present novel results in modeling users' interest in the context of AR systems and demonstrate some practical results in realizing such an approach in a multisensor AR system based on the usage of the see-through AR glasses. Due to the need for continuously adapt the AR content to the user's interest, such models are facing many challenges in sensing the user's behavior (using acoustic-, video-, gesture- and bio-sensors), interpreting it as an interest and deciding in real-time what kind of the adaptation to perform. We argue that this lead to a new class of AR system that we coined as adaptive AR (AR) systems. This work has been partially realized within the scope of the FP7 ICT research project ARtSENSE (www.artsense.eu), that is developing new AR concepts for improving personalized museum's experience. The tutorial will present practical results from applying the approach in three cultural heritage institutions in Europe (Paris, Madrid and Liverpool).

We extend estimation of range flow to handle brightness changes in image data caused by inhomogeneous illumination. Standard range flow computes 3D velocity fields using both range and intensity image sequences. Toward this end, range flow estimation combines a depth change model with a brightness constancy model. However, local brightness is generally not preserved when object surfaces rotate relative to the camera or the light sources, or when surfaces move in inhomogeneous illumination. We describe and investigate different approaches to handle such brightness changes. A straightforward approach is to prefilter the intensity data such that brightness changes are suppressed, for instance, by a highpass or a homomorphic filter. Such prefiltering may, though, reduce the signal-to-noise ratio. An alternative novel approach is to replace the brightness constancy model by 1) a gradient constancy model, or 2) by a combination of gradient and brightness constancy constraints used earlier successfully for optical flow, or 3) by a physics-based brightness change model. In performance tests, the standard version and the novel versions of range flow estimation are investigated using prefiltered or nonprefiltered synthetic data with available ground truth. Furthermore, the influences of additive Gaussian noise and simulated shot noise are investigated. Finally, we compare all range flow estimators on real data.