In the remote sensing community, extracting buildings from remote sensing imagery has triggered great interest. While many studies have been conducted, a comprehensive review of these approaches that are applied to optical and synthetic aperture radar (SAR) imagery is still lacking. Therefore, we provide an in-depth review of both early efforts and recent advances, which are aimed at extracting geometrical structures or semantic attributes of buildings, including building footprint generation, building facade segmentation, roof segment and superstructure segmentation, building height retrieval, building type classification, building change detection, and annotation data correction. Furthermore, a list of corresponding benchmark datasets is given. Finally, challenges and outlooks of existing approaches as well as promising applications are discussed to enhance comprehension within this realm of research.

While the volume of remote sensing data is increasing daily, deep learning in Earth Observation faces lack of accurate annotations for supervised optimization. Crowdsourcing projects such as OpenStreetMap distribute the annotation load to their community. However, such annotation inevitably generates noise due to insufficient control of the label quality, lack of annotators, frequent changes of the Earth’s surface as a result of natural disasters and urban development, among many other factors. We presentAdaptively trIggered Online Object-wise correction (AIO2)to address annotation noise induced by incomplete label sets. AIO2 features anAdaptive Correction Trigger (ACT)module that avoids label correction when the model training under- or overfits, and anOnline Object-wise Correction (O2C)methodology that employs spatial information for automated label modification. AIO2 utilizes a mean teacher model to enhance training robustness with noisy labels to both stabilize the training accuracy curve for fitting in ACT and provide pseudo labels for correction in O2C. Moreover, O2C is implementedonlinewithout the need to store updated labels every training epoch. We validate our approach on two building footprint segmentation datasets with different spatial resolutions. Experimental results with varying degrees of building label noise demonstrate the robustness of AIO2. Source code will be available at https://github.com/zhu-xlab/AIO2.git.

Identifying suitable building roofs for the installation of photovoltaic (PV) systems is important to sustainable energy planning. However, most existing approaches neglect roof superstructures that can obstruct the installation of PV systems. In this research, we propose a novel method, which can help to deal with this issue by detecting roof superstructures from aerial imagery. Considering that semantic information about roof masks is also informative, we propose to first learns roof segmentation maps that are further used to learn roof superstructure maps. Experiments are conducted on Roof Information Dataset (RID). Our method outperforms the state-of-the-art methods both quantitatively and qualitatively.

Building footprint maps are important to urban planning and monitoring. However, most existing approaches that fall back on convolutional neural networks (CNNs), require massive annotated samples for network learning. In this research, we propose a novel semi-supervised network, which can help to deal with this issue by leveraging a large amount of unlabeled data. Considering that rich information is also encoded in feature maps, we propose to integrate the consistency of both features and outputs in the end-to-end network training of unlabeled samples on data perturbation, enabling to impose additional constraints. Experiments are conducted on Inria dataset. Our approach is much superior to the state-of-the-art methods in both quantitative and qualitative results.

Building footprint generation is a vital task in a wide range of applications, including, to name a few, land use management, urban planning and monitoring, and geographical database updating. Most existing approaches addressing this problem fall back on convolutional neural networks (CNNs) to learn semantic masks of buildings. However, one limitation of their results is blurred building boundaries. To address this, we propose to learn attraction field representation for building boundaries, which is capable of providing an enhanced representation power. Our method comprises two elemental modules: an Img2AFM module and an AFM2Mask module. More specifically, the former aims at learning an attraction field representation conditioned on an input image, which is capable of enhancing building boundaries and suppressing the background. The latter module predicts segmentation masks of buildings using the learned attraction field map. The proposed method is evaluated on three datasets with different spatial resolutions: the ISPRS dataset, the INRIA dataset, and the Planet dataset. From experimental results, we find that the proposed framework can well preserve geometric shapes and sharp boundaries of buildings, which brings significant improvements over other competitors. The trained model and code are available at https://github.com/lqycrystal/AFM_building.

Accurate and reliable building footprint maps are vital to urban planning and monitoring, and most existing approaches fall back on convolutional neural networks (CNNs) for building footprint generation. However, one limitation of these methods is that they require strong supervisory information from massive annotated samples for network learning. State-of-the-art semi-supervised semantic segmentation networks with consistency training can help to deal with this issue by leveraging a large amount of unlabeled data, which encourages the consistency of model output on data perturbation. Considering that rich information is also encoded in feature maps, we propose to integrate the consistency of both features and outputs in the end-to-end network training of unlabeled samples, enabling to impose additional constraints. Prior semi-supervised semantic segmentation networks have established the cluster assumption, in which the decision boundary should lie in the vicinity of low sample density. In this work, we observe that for building footprint generation, the low-density regions are more apparent at the intermediate feature representations within the encoder than the encoder’s input or output. Therefore, we propose an instruction to assign the perturbation to the intermediate feature representations within the encoder, which considers the spatial resolution of input remote sensing imagery and the mean size of individual buildings in the study area. The proposed method is evaluated on three datasets with different resolutions: Planet dataset (3 m/pixel), Massachusetts dataset (1 m/pixel), and Inria dataset (0.3 m/pixel). Experimental results show that the proposed approach can well extract more complete building structures and alleviate omission errors.

3D building reconstruction from monocular remote sensing imagery is a promising and economical way to generate 3D city models at a large scale, yet the task is rarely touched. The paper tackles the problem via an end-to-end network. The goal is achieved by a modified network, named Mask-Height R-CNN, based on Mask R-CNN, with an additional height prediction head in the Region Proposal Network (RPN). Unlike most deep learning based methods, the height estimation is done on the instance level instead of pixel level, which does not require the assembly of the height maps and building masks. The proposed network gains good performances on ISPRS datasets, with 3D F1 scores of over 0.8.

Building footprint generation is a vital task of satellite imagery interpretation. However, the segmentation masks of buildings obtained by existing semantic segmentation networks often have blurred boundaries and irregular shapes. In this research, we propose a new boundary regularization network for building footprint generation in satellite images. More specifically, we consider semantic segmentation and boundary regularization in an end-to-end generative adversarial network (GAN). The learned building footprints are regularized by the interplay between the generator and discriminator. By doing so, the straight boundaries and geometric details of the building could be preserved. Experiments are conducted on a collected dataset of Planetscope satellite imagery (spatial resolution: 4.77 m/pixel). Our approach is much superior to the state-of-the-art methods in both quantitative and qualitative results.

Building segmentation is of great importance in the task of remote sensing imagery interpretation. However, the existing semantic segmentation and instance segmentation methods often lead to segmentation masks with blurred boundaries. In this paper, we propose a novel instance segmentation network for building segmentation in high-resolution remote sensing images. More specifically, we consider segmenting an individual building as detecting several keypoints. The detected keypoints are subsequently reformulated as a closed polygon, which is the semantic boundary of the building. By doing so, the sharp boundary of the building could be preserved. Experiments are conducted on selected Aerial Imagery for Roof Segmentation (AIRS) dataset, and our method achieves better performance in both quantitative and qualitative results with comparison to the state-of-the-art methods. Our network is a bottom-up instance segmentation method that could well preserve geometric details.

Building footprint maps are vital to many remote sensing (RS) applications, such as 3-D building modeling, urban planning, and disaster management. Due to the complexity of buildings, the accurate and reliable generation of the building footprint from RS imagery is still a challenging task. In this article, an end-to-end building footprint generation approach that integrates convolution neural network (CNN) and graph model is proposed. CNN serves as the feature extractor, while the graph model can take spatial correlation into consideration. Moreover, we propose to implement the feature pairwise conditional random field (FPCRF) as a graph model to preserve sharp boundaries and fine-grained segmentation. Experiments are conducted on four different data sets: 1) Planetscope satellite imagery of the cities of Munich, Paris, Rome, and Zurich; 2) ISPRS Benchmark data from the city of Potsdam; 3) Dstl Kaggle data set; and 4) Inria Aerial Image Labeling data of Austin, Chicago, Kitsap County, Western Tyrol, and Vienna. It is found that the proposed end-to-end building footprint generation framework with the FPCRF as the graph model can further improve the accuracy of building footprint generation by using only CNN, which is the current state of the art.

Building footprint information is an essential ingredient for 3-D reconstruction of urban models. The automatic generation of building footprints from satellite images presents a considerable challenge due to the complexity of building shapes. Recent developments in deep convolutional neural networks (DCNNs) have enabled accurate pixel-level labeling tasks. One central issue remains, which is the precise delineation of boundaries. Deep architectures generally fail to produce fine-grained segmentation with accurate boundaries due to progressive downsampling. In this work, we have proposed a end-to-end framework to overcome this issue, which uses the graph convolutional network (GCN) for building footprint extraction task. Our proposed framework outperforms state-of-the-art methods.

Building footprint information is an essential ingredient for 3-D reconstruction of urban models. The automatic generation of building footprints from satellite images presents a considerable challenge due to the complexity of building shapes. In this letter, we have proposed improved generative adversarial networks (GANs) for the automatic generation of building footprints from satellite images. We used a conditional GAN (CGAN) with a cost function derived from the Wasserstein distance and added a gradient penalty term. The achieved results indicated that the proposed method can significantly improve the quality of building footprint generation compared to CGANs, the U-Net, and other networks. In addition, our method nearly removes all hyperparameters tuning.

For a long time, the available global products of the urban area extent were limited to a coarse spatial resolution, e.g., Moderate Resolution Imaging Spectroradiometer (MODIS) global land cover (GLC) 500-m data and European Space Agency GlobCover 300-m data. This limitation was broken by the GlobeLand30 data, which is the world's first 30-m resolution GLC data set. However, detection accuracies of urban areas for the GlobeLand30 data (i.e., artificial surfaces) are not satisfactory. Therefore, in order to refine the detection accuracy of urban areas on the basis of the GlobeLand30 data, we propose a novel framework for urban area delineation by combining a set of remote sensing images and a geographical information system database, including the GlobeLand30 data, the National Land Cover Database (NLCD), the Land Use Interpretation Map (LUIM) of China, and Landsat images. First, the GlobeLand30 and land use/land cover products (e.g., NLCD or LUIM) are overlapped, and the study area is then separated into reliable and unreliable areas with a majority voting rule. Finally, the unreliable areas are confirmed by use of the Landsat data with a multiclassifier system. Experiments were conducted over two study areas that, respectively, represent typical patterns of American and Chinese urban areas: 1) the states of Utah, Mississippi, and Pennsylvania in the U.S. and 2) the provinces of Ningxia, Fujian, and Jilin in China. The results show that the accuracy of the GlobeLand30 data for urban area delineation can be significantly improved by integrating the multisource data and using the multiclassifier system.