Unmanned aerial vehicles equipped with modern vision algorithms are crucial for missions such as reconstruction and target acquisition. However, when deployed in the field, undulating terrain can cause significant fluctuations in image scale and degrade the performance of vision algorithms. Instead of developing specialized image processing schemes with limited adaptability, this paper presents a novel 3D visual coverage algorithm that is compatible with existing generic vision algorithms and maintains a uniform image scale for ground targets. In detail, photogrammetric constraints are initially introduced to generate aerial waypoints, and then the negative effects of valley clustering are addressed. Elastic Photogrammetric Constraints (EPC) are further proposed to eliminate valley clustering effects induced by saddle terrain. The experimental results demonstrate that EPC reduces the traversal path length by up to 37.38 % compared to the previous work, but with a minor trade-off in scale variations.

Localization of robots using subsurface features observed by ground-penetrating radar (GPR) enhances and adds robustness to common sensor modalities, as subsurface features are less affected by weather, seasons, and surface changes. We introduce an innovative multimodal odometry approach using inputs from GPR, an inertial measurement unit (IMU), and a wheel encoder. To efficiently address GPR signal noise, we introduce an advanced feature representation called the subsurface feature matrix (SFM). The SFM leverages frequency domain data and identifies peaks within radar scans. Additionally, we propose a novel feature matching method that estimates GPR displacement by aligning SFMs. The integrations from these three input sources are consolidated using a factor graph approach to achieve multimodal robot odometry. Our method has been developed and evaluated with the CMU-GPR public dataset, demonstrating improvements in accuracy and robustness with real-time performance in robotic odometry tasks.

Optimizing the generalization performance of road surface crack models in practical applications represents a challenging task. Especially for thin and irregular cracks with random and expansive topologies, the loss functions used in current deep learning-based crack segmentation models are sensitive to single pixels, which tends to cause the model to overfit the training data, diminishing its generalization ability in real-world scenarios. Therefore, we take the loss function as a starting point and explore the introduction of proactive fault tolerance mechanisms into the training process of the crack segmentation model, which is called Proactive Fault Tolerance Loss (PFT Loss), to enhance the generalization capability of model in actual applications. Specifically, the PFT Loss function establishes correlations between the segmentation prediction pixels and the corresponding labeled pixels within the neighborhood window using Markov Random Fields (MRFs). The correlation is used as a reference for predicting relative shifts in segmented pixels. Proactive Fault Tolerance is performed on the loss between labeling and prediction to achieve a more natural and adaptive training method for crack segmentation. Full experiments are conducted on five public crack datasets and one self-constructed dataset. The experimental results indicate that the model trained with PFT Loss has better segmentation performance compared to other loss functions.

Robot localization using subsurface features captured by Ground-Penetrating Radar (GPR) complements and improves robustness over existing common sensor modalities, as subsurface features are less sensitive to weather, season and surface scene changes. Here, we propose a novel subsurface feature-based localization method that uses only GPR measurements with a known subsurface map. An efficient feature descriptor, the dominant energy curve (DEC), is designed to identify different locations in cluttered conditions. Specifically, image processing techniques that involve background segmentation, energy point detection, and energy curve refinement are designed to extract DEC features from a 2D radargram. With DECs features obtained, a metric subsurface feature map is constructed. Finally, we perform robot localization by feature matching under a particle swarm optimization framework. We have implemented our method and tested it with the public CMU-GPR dataset. The results show that our algorithm improves accuracy and robustness with real-time performance for robot localization tasks. Specifically, the mean localization error is 0.50 m for all cases.

The accurate perception of subsurface objects and defects is vital in airport routine maintenance. For the complex subsurface environment of the airport runway, to obtain the high-performance segmentation of typical subsurface targets, a multistream attention segmentation network is proposed. The network takes the ground penetrating radar (GPR) raw data and the preprocessed B-scan image as multistream input. It carries out sufficient feature fusion on multiple modals, scales, and levels to get robust feature representation. Furthermore, we proposed two attention mechanisms suitable for multistream feature fusion, which can learn more effective features. We validate our method on an actual airport runway dataset. Experimental results show that our method can obtain an F1-measure of 82.08%, 89.12%, and 82.54% for three typical subsurface targets: void, pipe, and steel mesh, respectively.

Ground penetrating radar (GPR) is widely used for detecting airport runway subsurface defects. The trailing interference in the GPR data disguises subsurface defect responses, which seriously affects the accuracy of subsurface defect detection. To tackle the challenge of subsurface defects detection under trailing interference in real scenarios, a new multi-modal fusion network referred to as M2FNet is proposed. Based on the premise that the trailing signal is highly similar across all adjacent A-scans, the model employs a transformer encoder to extract global features of the signal with long-distance correlation. In contrast, the subsurface defects only show echo characteristics in a few adjacent B-scans. The phase of the trailing and the target signal is opposite, which is easy to be discovered from the Top-scan view. Thus, a hybrid convolutional neural network structure is used to extract local features from GPR images of different views. This dual network structure extremely enhance the representation learning of GPR data. In order to investigate various subsurface defects and trailing interference collected by multiple GPR systems under different conditions, the first large-scale hybrid dataset called ASD-GPR is created. Transfer learning is employed to enhance the model’s ability to detect rare defects by fine-tuning it for real-world situations, differing from synthetic training data scenarios. The results of the experiments reveal that M2FNet outperforms state-of-the-art object detection methods in various real-world scenarios, demonstrating superior performance in detecting subsurface defects.

In this article, we report system and algorithmic developments for a sensing suite comprising a camera and a ground penetrating radar (GPR) with a wheel encoder designed for both surface and subsurface infrastructure inspection, which is a multimodal mapping task. To fuse different sensor modalities properly, we solve a novel GPR-camera calibration problem and a synchronization-challenged sensor fusion problem. First, we design a calibration rig, model the GPR imaging process, introduce a mirror to obtain the joint coverage between the camera and the GPR, and employ the maximum-likelihood estimator to estimate the relative pose between the camera and the GPR with error analysis. Second, we propose a data collection scheme using the customized artificial landmarks to synchronize camera images (temporally evenly spaced) and GPR/encoder data (spatially evenly spaced). We also employ pose graph optimization with location discrepancy as penalty functions to perform data fusion for 3-D reconstruction. We have tested our system in physical experiments. The results show that our system successfully fuses encoder-camera-GPR sensory data and accomplishes a metric 3-D reconstruction. Moreover, our sensor fusion approach reduces the end-to-end distance error from 6.4 to 0.7 cm in a real bridge inspection experiment if comparing to the counterpart that only uses encoder measurements.

We propose a novel subsurface pipeline mapping and 3D reconstruction method by fusing ground-penetrating radar (GPR) scans and camera images. To facilitate the simultaneous detection of multiple pipelines, we model the GPR sensing process and prove hyperbola response for general scanning with nonperpendicular angles. Furthermore, we fuse visual simultaneous localization and mapping outputs, encoder readings with GPR scans to classify hyperbolas into different pipeline groups. We extensively apply the J-linkage method and maximum likelihood estimation with error analysis to improve algorithm robustness and accuracy. As a result, we optimally estimate the radii and locations of all pipelines. We have implemented our method and tested it in physical experiments with representative pipeline configurations. Two different kinds of 3-m-long pipes are used, with radii being 4.62 and 3.02 cm, respectively. The results show that our method successfully reconstructs all subsurface pipes. Moreover, the average estimation errors for two orientation angles of pipelines are 1.73° and 0.73°, respectively. The average localization error is 4.47 cm. Note to Practitioners-Automatic and accurate underground pipeline mapping technology is very important in civil construction projects. Lack of 3D utility pipeline maps may lead to accidental damage in civil construction and maintenance. Although ground-penetrating radar (GPR-based pipeline mapping methods have been studied for several years, these methods require the perpendicular scanning with respect to the pipe, which is impossible to guarantee in practice since the orientations of pipelines are unknown. Furthermore, these traditional methods can only estimate one pipeline at a time in a survey area and require prior knowledge of pipe diameter. We propose a robotic subsurface pipeline mapping method with a GPR and a camera to handle difficult factors such as multiple pipes, unknown pipeline orientation, and unknown pipeline diameters. Hence, we can perform GPR scanning along any generic linear trajectories. Our method has been tested in physical experiments with representative pipeline configurations. The results are sufficiently accurate, and it proves that our method can be an effective technology to reconstruct the underground pipelines.

Pavement crack detection from images is a challenging problem due to intensity inhomogeneity, topology complexity, low contrast, and noisy texture background. Traditional learning-based approaches have difficulties in obtaining representative training samples. We propose a new unsupervised multi-scale fusion crack detection (MFCD) algorithm that does not require training data. First, we develop a windowed minimal intensity path-based method to extract the candidate cracks in the image at each scale. Second, we find the crack correspondences across different scales. Finally, we develop a crack evaluation model based on a multivariate statistical hypothesis test. Our approach successfully combines strengths from both the large-scale detection (robust but poor in localization) and the small-scale detection (detail-preserving but sensitive to clutter). We analyze and experimentally test the computational complexity of our MFCD algorithm. We have implemented the algorithm and have it extensively tested on three public data sets, including two public pavement data sets and an airport runway data set. Compared with six existing methods, experimental results show that our method outperforms all counterparts. Specifically, it increases the precision, recall, and F1-measure over the state-of-the-art by 22%, 12%, and 19%, respectively, on one public data set.

We report system and algorithmic development for a sensing suite comprising multiple sensors for both surface and subsurface transportation infrastructure inspection focusing on multi-modal mapping for inspection. The sensing suite contains a camera, a ground penetrating radar (GPR), and a wheel encoder. We design the sensing suite and propose a data collection scheme using customized artificial landmarks (ALs). We use ALs to synchronize two types data streams: camera images that are temporally evenly-spaced and GPR/encoder data that are spatially evenly-spaced. We also employ pose graph optimization with synchronization as penalty functions to further refine synchronization and perform data fusion for 3D reconstruction. We have implemented the system and tested it in physical experiments. The results show that our system successfully fuses three sensory data and product metric 3D reconstruction. The sensor fusion approach reduces the end-to-end distance error from 7.45cm to 3.10cm.

We propose a novel subsurface pipeline mapping method by fusing Ground Penetrating Radar (GPR) scans and camera images. To facilitate the simultaneous detection of multiple pipelines, we model the GPR sensing process and prove hyperbola response for general scanning with non-perpendicular angles. Furthermore, we fuse visual simultaneous localization and mapping outputs, encoder readings with GPR scans to classify hyperbolas into different pipeline groups. We extensively apply the J-Linkage method and maximum likelihood estimation to improve algorithm robustness and accuracy. As the result, we optimally estimate the radii and locations of all pipelines. We have implemented our method and tested it in physical experiments with representative pipeline configurations. The results show that our method successfully reconstructs all subsurface pipes. Moreover, the average localization error is 4.69cm.

Measuring the quality of roads and ensuring they are ready for autonomous driving is important for future transportation systems. Here we focus on developing metrics and algorithms to assess lane marking (LM)qualities from an egocentric view of an inspection vehicle equipped with a global positioning system (GPS)receiver, a frontal-view camera, and a light detection and ranging (LIDAR)system. We propose three quality metrics for LMs: correctness, shape, and visibility. The correctness metric measures the divergence between the expected LMs based on prior map inputs and the actual sensor inputs. The shape metric evaluates smoothness in road curvature and width range. The visibility metric evaluates the contrast between LMs and background road surfaces. We propose a dual-modal algorithm to compute these metrics. We have implemented the algorithms and tested them under KITTI dataset. The results show that our metrics can successfully detect LM anomalies in all testing scenarios.

To facilitate scene understanding and robot navigation in a modern urban area, we design a multilayer feature graph (MFG) based on two views from an on-board camera. The nodes of an MFG are features such as scale invariant feature transformation (SIFT) feature points, line segments, lines, and planes while edges of the MFG represent different geometric relationships such as adjacency, parallelism, collinearity, and coplanarity. MFG also connects the features in two views and the corresponding 3D coordinate system. Building on SIFT feature points and line segments, MFG is constructed using feature fusion which incrementally, iteratively, and extensively verifies the aforementioned geometric relationships using random sample consensus (RANSAC) framework. Physical experiments show that MFG can be successfully constructed in urban area and the construction method is demonstrated to be very robust in identifying feature correspondence.