The spatial resolution of Earth Observation (EO) images plays a key role in building footprint extraction. For the spatial resolution enhancement, deep learning-based image super-resolution methods have been widely used due to their remarkable performance. Transformer-based networks are effective and has drawn much attention in computer vision but underutilized in remote sensing, especially for super-resolving building datasets. Therefore, in this paper, we developed a novel transformer-based Single-Image Super-Resolution (SISR) method, named Pyramid Vision Transformer-Residual Feature Aggregation Network (PVT_RFANet), to improve the spatial resolution of building datasets. Specifically, the PVT v2 network was embedded into our Momentum Spatial-Channel Attention Residual Feature Aggregation Network (MSCA-RFANet). Moreover we conducted a comparative study to compare our method with Bicubic interpolation (BI), Super-resolution Convolutional Neural Network (SRCNN), Deep Recursive Residual Network (DRRN), SRResNet, and MSCA-RFANet. Using Peak Signal-Noise Ratio (PSNR) and Similarity Structure Index Measurement (SSIM) as the evaluation metrics, our method showed highest performance with the PSNR of 22.01 dB and the SSIM of 0.50 on the WHU Building Dataset, which demonstrated the superior performance of the proposed method.

With the development of oil/gas production, its adverse impact has drawn much attention. Therefore, automated detecting oil/gas well sites become important. Current research has been focused on detection using sole source RGB images, which was not the common case in remote sensing. In this study, we explored the use of a combination of the Residual Channel Attention Network (RCAN) and the state-of-the-art object detection method, You Only Look Once (YOLO) v4, to detect oil/gas well sites from multi-sensor images. To testify the feasibility of the combination, we selected 18 RapidEye and 15 WorldView images which cover the oil sands area in Alberta, Canada. We applied a pre-trained RCAN to unify the spatial resolution of different images to 2m/pixel. To maximize the feature space, we preserved 5 bands which were available in both images. YOLO v4 was applied on cropped image patches to detect oil/gas well sites. The experiment results showed that using the framework proposed in this study, oil/gas well sites can be localized accurately although the bounding boxes of the sites may not perfectly align with the objects.

This paper explores the effect of COVID-19 outbreaks on human activity through nighttime light images of Greater Toronto Area (GTA), Canada. The methods used in this paper include image preprocessing, image classification, and spatial analysis. By using the nighttime light radiance data from VIIRS/NPP data products and COVID-19 cases and comparing this data from the pre-pandemic year, the impact of COVID-19 was analyzed. The result shows that during the pandemic year the monthly average nighttime light radiance has decreased about 4.3-5.0% compared to the pre-pandemic year. The classification results shows that the average percentage of changes in residential areas, public facilities, and commercial areas are 0.3%, −0.7%, and −1.2%, respectively of each corresponding month. Meanwhile, the spatial analysis results show population distribution patterns in GTA during the pandemic year. Overall, the nighttime lights (NTL) images can be used for a preliminary understanding of how COVID-19 affected human activities and is corroborated with other forms data collection used for the pandemic analysis.

Robust road boundary extraction and completion play an important role in providing guidance to all road users and supporting high-definition (HD) maps. The significant challenges remain in remarkable and accurate road boundary recovery from poor road boundary conditions. This paper presents a novel deep learning framework, named BoundaryNet, to extract and complete road boundaries by using both mobile laser scanning (MLS) point clouds and high-resolution satellite imagery. First, road boundaries are extracted by conducting a curb-based extraction method. Such extracted 3D road boundary lines are used as inputs to feed into a U-shaped network for erroneous boundary denoising. Then, a convolutional neural network (CNN) model is proposed to complete the road boundaries. Next, to achieve more complete and accurate road boundaries, a conditional deep convolutional generative adversarial network (c-DCGAN) with the assistance of road centerlines extracted from satellite images is developed. Finally, according to the completed road boundaries, the inherent road geometries are calculated. The proposed methods were evaluated using satellite imagery and four MLS point cloud datasets with varying densities and road conditions in urban environments. The quality evaluation metrics of 82.88%, 82.43%, 88.86%, and 84.89% were achieved for four data sets. The experimental results indicate that the BoundaryNet model can provide a promising solution for road boundary completion and road geometry estimation.

This article presents a semi-automated method to extract the lane features along the curved roads from mobile laser scanning (MLS) point clouds. The proposed method consists of four steps. After data pre-processing, a road edge detection algorithm is performed to distinguish road curbs and extract road surfaces. Then, textual and directional road markings such as arrows, symbols, and words, to inform drivers in necessary cases, are detected by intensity thresholding and conditional Euclidean clustering algorithms. Furthermore, lane markings are extracted by local intensity analysis and distance thresholding methods according to road design standards, because they are more regular along the road. Finally, centerline points on lanes are estimated based on the coordinates of extracted lane markings. Our method shows strong feasibility and robustness when creating high-definition (HD) maps from MLS data, by increasing the number of blocks in the curve and the distance threshold control in curved lane centerline extraction. Quantitative evaluations show that the average recall, precision, and F1-score obtained from four datasets for road marking extraction are 93.87%, 93.76%, and 93.73%, respectively. The generated lane centerlines are evaluated by overlaying them on manually labeled reference buffers from 4 cm resolution orthoimagery. The comparative study indicates that the proposed methods can achieve higher accuracy and robustness than most state-of-the-art methods.

Accurate road marking extraction and classification play a significant role in the development of autonomous vehicles (AVs) and high-definition (HD) maps. Due to point density and intensity variations from mobile laser scanning (MLS) systems, most of the existing thresholding-based extraction methods and rule-based classification methods cannot deliver high efficiency and remarkable robustness. To address this, we propose a capsule-based deep learning framework for road marking extraction and classification from massive and unordered MLS point clouds. This framework mainly contains three modules. Module I is first implemented to segment road surfaces from 3D MLS point clouds, followed by an inverse distance weighting (IDW) interpolation method for 2D georeferenced image generation. Then, in Module II, a U-shaped capsule-based network is constructed to extract road markings based on the convolutional and deconvolutional capsule operations. Finally, a hybrid capsule-based network is developed to classify different types of road markings by using a revised dynamic routing algorithm and large-margin Softmax loss function. A road marking dataset containing both 3D point clouds and manually labeled reference data is built from three types of road scenes, including urban roads, highways, and underground garages. The proposed networks were accordingly evaluated by estimating robustness and efficiency using this dataset. Quantitative evaluations indicate the proposed extraction method can deliver 94.11% in precision, 90.52% in recall, and 92.43% in F1-score, respectively, while the classification network achieves an average of 3.42% misclassification rate in different road scenes.

Although significant improvement has been achieved in fully autonomous driving and semantic high-definition map (HD) domains, most of the existing 3D point cloud segmentation methods cannot provide high representativeness and remarkable robustness. The principally increasing challenges remain in completely and efficiently extracting high-level 3D point cloud features, specifically in large-scale road environments. This paper provides an end-to-end feature extraction framework for 3D point cloud segmentation by using dynamic point-wise convolutional operations in multiple scales. Compared to existing point cloud segmentation methods that are commonly based on traditional convolutional neural networks (CNNs), our proposed method is less sensitive to data distribution and computational powers. This framework mainly includes four modules. Module I is first designed to construct a revised 3D point-wise convolutional operation. Then, a U-shaped downsampling-upsampling architecture is proposed to leverage both global and local features in multiple scales in Module II. Next, in Module III, high-level local edge features in 3D point neighborhoods are further extracted by using an adaptive graph convolutional neural network based on the K-Nearest Neighbor (KNN) algorithm. Finally, in Module IV, a conditional random field (CRF) algorithm is developed for postprocessing and segmentation result refinement. The proposed method was evaluated on three large-scale LiDAR point cloud datasets in both urban and indoor environments. The experimental results acquired by using different point cloud scenarios indicate our method can achieve state-of-the-art semantic segmentation performance in feature representativeness, segmentation accuracy, and technical robustness.

Recently, the advancement of deep learning (DL) in discriminative feature learning from 3-D LiDAR data has led to rapid development in the field of autonomous driving. However, automated processing uneven, unstructured, noisy, and massive 3-D point clouds are a challenging and tedious task. In this article, we provide a systematic review of existing compelling DL architectures applied in LiDAR point clouds, detailing for specific tasks in autonomous driving, such as segmentation, detection, and classification. Although several published research articles focus on specific topics in computer vision for autonomous vehicles, to date, no general survey on DL applied in LiDAR point clouds for autonomous vehicles exists. Thus, the goal of this article is to narrow the gap in this topic. More than 140 key contributions in the recent five years are summarized in this survey, including the milestone 3-D deep architectures, the remarkable DL applications in 3-D semantic segmentation, object detection, and classification; specific data sets, evaluation metrics, and the state-of-the-art performance. Finally, we conclude the remaining challenges and future researches.

This paper recognizes the research gaps and difficulties in generating transition lines (the paths that pass through a road intersection) in road intersections from mobile laser scanning (MLS) point clouds. The proposed method contains three modules: road surface detection, lane marking extraction, and transition line generation. First, the points covering the road surface are extracted using the voxel-based upward growing and the improved region growing. Then, lane markings are extracted and identified according to the multi-thresholding and the geometric filtering. Finally, transition lines are generated through a combination of the lane node structure generation algorithm and the cubic Catmull-Rom spline algorithm. The experimental results demonstrate that transition lines can be successfully generated for both T- and cross-intersections with promising accuracy. In the validation of lane marking extraction using the manually interpreted lane marking points, the method can achieve average precision, recall, and F1-score of 90.80%, 92.07%, and 91.43%, respectively. The success rate of transition line generation is 96.5%. Furthermore, the buffer-overlay-statistics (BOS) method validates that the proposed method can generate lane centerlines and transition lines within 20-cm-level localization accuracy from the MLS point clouds.

In this paper, we designed an end-to-end spectral-spatial residual network (SSRN) that takes raw 3-D cubes as input data without feature engineering for hyperspectral image classification. In this network, the spectral and spatial residual blocks consecutively learn discriminative features from abundant spectral signatures and spatial contexts in hyperspectral imagery (HSI). The proposed SSRN is a supervised deep learning framework that alleviates the declining-accuracy phenomenon of other deep learning models. Specifically, the residual blocks connect every other 3-D convolutional layer through identity mapping, which facilitates the backpropagation of gradients. Furthermore, we impose batch normalization on every convolutional layer to regularize the learning process and improve the classification performance of trained models. Quantitative and qualitative results demonstrate that the SSRN achieved the state-of-the-art HSI classification accuracy in agricultural, rural-urban, and urban data sets: Indian Pines, Kennedy Space Center, and University of Pavia.

The assessment of pavement cracks is one of the essential tasks for road maintenance. This paper presents a novel framework, called ITVCrack, for automated crack extraction based on iterative tensor voting (ITV), from high-density point clouds collected by a mobile laser scanning system. The proposed ITVCrack comprises the following: 1) the preprocessing involving the separation of road points from nonroad points using vehicle trajectory data; 2) the generation of the georeferenced feature (GRF) image from the road points; and 3) the ITV-based crack extraction from the noisy GRF image, followed by an accurate delineation of the curvilinear cracks. Qualitatively, the method is applicable for pavement cracks with low contrast, low signal-to-noise ratio, and bad continuity. Besides the application to GRF images, the proposed framework demonstrates much better crack extraction performance when quantitatively compared to existing methods on synthetic data and pavement images.

This paper presents a survey of literature about road feature extraction, giving a detailed description of a Mobile Laser Scanning (MLS) system (RIEGL VMX-450) for transportation-related applications. This paper describes the development of automated algorithms for extracting road features (road surfaces, road markings, and pavement cracks) from MLS point cloud data. The proposed road surface extraction algorithm detects road curbs from a set of profiles that are sliced along vehicle trajectory data. Based on segmented road surface points, we create Geo-Referenced Feature (GRF) images and develop two algorithms, respectively, for extracting the following: 1) road markings with high retroreflectivity and 2) cracks containing low contrast with their surroundings, low signal-to-noise ratio, and poor continuity. A comprehensive comparison illustrates satisfactory performance of the proposed algorithms and concludes that MLS is a reliable and cost-effective alternative for rapid road inspection.

This paper presents a region-based approach to segmentation of the satellite synthetic aperture radar (SAR) intensity imagery. The approach is based on a Voronoi tessellation, the Bayesian inference, and the reversible jump Markov chain Monte Carlo (RJMCMC) algorithm. By Voronoi tessellation, the approach partitions a SAR image into a set of polygons corresponding to the components of the segmented homogenous regions. Each polygon is assigned a label to indicate a homogeneous region. The labels for all the polygons form a label field, which is characterized by an improved Potts model. The intensities of pixels in each polygon are assumed to satisfy identical and independent gamma distributions in terms of their label. Following the Bayesian paradigm, the posterior distribution that characterizes the SAR image segmentation can be obtained up to the integration constant. Then, a RJMCMC scheme is designed to simulate the posterior distribution and estimate its parameters. Finally, an optimal segmentation is obtained by the maximum a posteriori algorithm. The results obtained on both real Radarsat-1/2 and simulated SAR intensity images show that our approach works well and is very promising.