3D building reconstruction from monocular remote sensing imagery is an important research problem that has been extensively studied for several decades. Although monocular remote sensing imagery is a more economic data source compared with the LiDAR data and multi-view imagery, its limited information results in great challenges and restricts the performance of existing monocular reconstruction methods. Moreover, the expensive cost and the limited quantity of 3D annotations also restrict the application scenes of existing methods, which are mostly based on fully-supervised learning. In our previous work, we have proposed MTBR-Net, a monocular building reconstruction method that consists of a fully-supervised multi-task network and a post-processing module for optimizing the reconstruction results. In this work, we further propose WS-MTBR-Net, a weakly-supervised building reconstruction network that uses fewer 3D annotations and achieves better performance in an end-to-end manner. Specifically, our WS-MTBR-Net fully leverages the relation between different components of a 3D building instance and the property of off-nadir images to improve the footprint segmentation boundary, based on six modified tasks and a new network structure with an improved feature warping module to support weakly-supervised learning. We also design a new training strategy via a hybrid loss function that enables utilizing the training samples with different annotation levels, i.e., complete 3D annotations, 2D footprint annotations, and image-level angle annotations. Results on BONAI Shanghai and Xi’an test datasets demonstrate that our method achieves competitive performance when using 50% fewer 3D-annotated samples, and improves the footprint segmentation F1-score by around 4% compared with current state-of-the-art.

Extracting building footprints from aerial images is essential for precise urban mapping with photogrammetric computer vision technologies. Existing approaches mainly assume that the roof and footprint of a building are well overlapped, which may not hold in off-nadir aerial images as there is often a big offset between them. In this paper, we propose an offset vector learning scheme, which turns the building footprint extraction problem in off-nadir images into an instance-level joint prediction problem of the building roof and its corresponding “roof to footprint” offset vector. Thus the footprint can be estimated by translating the predicted roof mask according to the predicted offset vector. We further propose a simple but effective feature-level offset augmentation module, which can significantly refine the offset vector prediction by introducing little extra cost. Moreover, a new dataset, Buildings in Off-Nadir Aerial Images (BONAI), is created and released in this paper. It contains 268,958 building instances across 3,300 aerial images with fully annotated instance-level roof, footprint, and corresponding offset vector for each building. Experiments on the BONAI dataset demonstrate that our method achieves the state-of-the-art, outperforming other competitors by 3.37 to 7.39 points in F1-score. The codes, datasets, and trained models are available athttps://github.com/jwwangchn/BONAI.git.

Pedestrian detection from UAV images is vital for many fields. Given that visible images are susceptible to bad illumination, thermal images with the ability to characterize the temperature of an object can provide auxiliary information. It is useful to fuse the visible and thermal images to improve the pedestrian detection performance. Unfortunately, studies on pedestrian detection with UAV visible-thermal image pairs are still rare. Therefore, we propose a method for Multi-source Pedestrian Detection based on Feature Fusion (MPDFF). With the registered visible and thermal image pairs as the input, MPDFF can achieve better characterization of pedestrians by concatenating the features from the two images. MPDFF performs much better than the methods that use only single-source images, which demonstrates that visible and thermal images are complementary in pedestrian detection.

3D building reconstruction from monocular remote sensing imagery is an important research problem and an economic solution to large-scale city modeling, compared with reconstruction from LiDAR data and multi-view imagery. However, several challenges such as the partial invisibility of building footprints and facades, the serious shadow effect, and the extreme variance of building height in large-scale areas, have restricted the existing monocular image based building reconstruction studies to certain application scenes, i.e., modeling simple low-rise buildings from near-nadir images. In this study, we propose a novel 3D building reconstruction method for monocular remote sensing images, which tackles the above difficulties, thus providing an appealing solution for more complicated scenarios. We design a multi-task building reconstruction network, named MTBR-Net, to learn the geometric property of oblique images, the key components of a 3D building model and their relations via four semantic-related and three offset-related tasks. The network outputs are further integrated by a prior knowledge based 3D model optimization method to produce the the final 3D building models. Results on a public 3D reconstruction dataset and a novel released dataset demonstrate that our method improves the height estimation performance by over 40% and the segmentation F1-score by 2% - 4% compared with current state-of-the-art.