Geolocating multiple targets of interest on the ground from an aerial platform is an important activity in many applications, such as visual surveillance. However, due to the limited measurement accuracy of commonly used airborne sensors (including altimeters, accelerometer, gyroscopes, and so on) and the small size, complex motion, and a large number of ground targets in aerial images, most of the current unmanned aerial vehicle (UAV)-based ground target geolocation algorithms have difficulty in obtaining accurate geographic location coordinates online, especially at middle and high altitudes. To solve these problems, in this article, a novel online ground multitarget geolocation framework using a UAV platform is proposed, which minimizes the introduction of sensor error sources and uses only monocular aerial image sequences and global positioning system (GPS) data to perform parallel processing of target detection and rapid 3-D sparse geographic map construction and target geographic location estimations, thereby improving the accuracy and speed of ground multitarget online geolocation. In this framework, a detection algorithm based on deep learning is first adopted to improve the accuracy and robustness of small target detection in aerial images by constructing an aerial image dataset. Then, we propose a novel target geolocation algorithm based on 3-D map construction, which combines continuous images and GPS data collected online using a UAV platform to generate a 3-D geographic map and accurately estimates the GPS location of the target center pixel through projection and triangulation of the map points on the image. Finally, we design a data transmission architecture that selects multiple processes to perform image acquisition, target detection, and target geolocation tasks in parallel and utilizes database communication between the processes to achieve accurate online geolocation of ground targets in aerial images, regardless of whether the target is in a static or moving state. To evaluate the effectiveness of the proposed framework, we build an online ground multitarget geolocation system using a quad-rotor UAV and carry out a large number of experiments in simulations and real environments. Qualitative and quantitative experimental results proved that the framework can accurately locate ground targets in various complex environments, such as parks, highways, schools, cities, different flight altitudes (50–2000 m), and different attitude angles, and the average positioning error is approximately 1 m at 2000 m for cities with rich 3-D structures.

Limited by aircraft flight altitude and camera parameters, it is necessary to obtain wide-angle panoramas quickly by stitching aerial images, which is helpful in rapid disaster investigation, recovery after earthquakes, and aerial reconnaissance. However, most existing stitching algorithms do not simultaneously meet practical real-time, robustness, and accuracy requirements, especially in the case of a long-distance multistrip flight. In this paper, we propose a novel image-only real-time UAV image mosaic framework for long-distance multistrip flights that does not require any auxiliary information, such as GPS or GCPs. The framework has a complete structure, mainly consisting of the three tasks of automatic initialization, current frame tracking, and real-time mosaic generation. The stitching plane is determined in the initialization process, the homography transformation of the current image is estimated in the tracking task, and the image is mapped to the stitching plane to generate and update the panorama in the real-time mosaic process. The core idea is that, in the tracking task, we introduce and develop a keyframe insertion strategy to generate a keyframe list and, on this basis, design a homography matrix estimation based on a local optimization strategy to reduce the accumulated error when continuously stitching image sequences collected online by UAVs and to realize real-time, effective UAV image mosaic construction. In addition, this framework has good scalability, which is not limited to a specific algorithm. To evaluate the effectiveness of the proposed framework, we carry out a large number of experiments on the AirSim simulation platform and present an exhaustive evaluation in some sequences from a popular dataset. Qualitative and quantitative experimental results in simulation and real environments demonstrate that our algorithm can obtain an effective and robust mosaic image in real-time. Through strategy comparison experiments, it is proven that the keyframe insertion strategy and the local optimization strategy both improve the stitching performance. Compared with five state-of-art image stitching approaches, the mosaic effect of the proposed method is comparable or better. In terms of algorithm speed, its performance is superior to them. Additionally, experiments of illumination change and feature replacement in the framework verify the good adaptability and scalability of the algorithm.

Tracking multiple people in crowds is a fundamental and essential task in the multimedia field. It is often hindered by difficulties, such as dynamic occlusion between objects, cluttered background, and abrupt illumination changes. To respond to this need, in this paper, we combine deep and depth to build a stereo tracking system for crowds. The core of the system is the fusion of the advantages of deep learning and depth information, which is exploited to achieve object segmentation and improve the multiobject tracking performance in severe occlusion. More specifically, first, to obtain more accurate detection observations in the tracking system, we present a novel object-level segmentation method. This method combines the effective detection results of deep learning with depth information to obtain precise object segmentation results. Then, we integrate the segmentation results and three-dimensional (3-D) information to extract 2-D and 3-D characteristics to represent the target, and design three similarity models to realize a stereo tracking method through data association in crowds. Finally, we build a diverse stereo dataset including various challenging indoor and outdoor scenes. The comprehensive experiments verify the effective and robust tracking performance of our system in various scenarios, and the system has rich output results including segmentation results, target distance, and tracking results. Moreover, the qualitative and quantitative comparison results show that the proposed algorithm not only has good object segmentation performance but also improves the tracking performance of completely and partially occluded objects, which is superior to the tested state-of-the-art tracking approaches.

Human detection in dense crowds poses to be a demanding task owing to complex background and serious occlusion. In this paper, we propose a novel real-time and reliable human detection system. We solve the human detection problem by presenting a novel cube surface model captured by a binocular stereo vision camera. We first propose a cube surface model to estimate the 3D background cubes in the surveillance area. We then develop a shadow-free strategy for cube surface model updating. Thereafter, we present a shadow weighted clustering method to efficiently search for human as well as remove false alarms. Ultimately, we have developed a highly robust human detection system, and we carefully evaluate our system in many real challenge indoor and outdoor scenes. Expensive experiments demonstrate our system achieves real-time performance, higher detection rate and lower face alarms in comparison with state-of-the-art human detection methods.