Multispectral Point Clouds (MPCs) acquired from Unmanned Aerial Vehicles (UAVs), leveraging LiDAR’s canopy-penetrating capacity, provide distinct advantages for detecting occluded targets beneath vegetation canopy. However, limited samples, missing target spatial morphology, and unstructured data format have led to low occluded detection accuracy. Given these constraints, an unsupervised stereo detection method for occluded targets based on the spherical shell model with MPCs from UAVs has been proposed for the first time. The method exploits the spatial-spectral differences between targets and the background, treating targets as anomalies within the background, and enables occluded target detection without requiring training. The spherical shell model has been constructed in MPCs to avoid contaminating the global background with targets, leveraging its local separation characteristics for unsupervised detection of occluded targets. An adaptive radius and multi-scale spatial feature extraction have been designed to enhance the method’s robustness. The collaborative representation model has been utilized to achieve unsupervised detection by suppressing expressible points in local background features. To bridge the gap between algorithmic metrics and practical requirements, we have further proposed a target-level evaluation method that overcomes the limitations of conventional methods, which are susceptible to density-induced false alarm distortions in unstructured MPCs. Experiments on three real-world MPCs and a public airborne hyperspectral and LiDAR dataset show that our method achieves higher detection accuracy than existing spectral-based detectors and performs better on exposed targets.

Airborne Laser Scanning (ALS) offers significant potential for three-dimensional (3D) object detection due to its ability to penetrate the canopy and acquire high-precision 3D spatial information. However, complex terrain distribution and backgrounds similar to objects hinder effective object detection in airborne scenes. To address these challenges, we propose TG-ADet, the first 3D object detection network explicitly designed for ALS point clouds. Our approach introduces three key components and integrates them into a unified framework. A multi-stage terrain guidance module predicts the terrain distribution and guides multiple detection stages based on prediction results, focusing on objects under various terrain conditions. A sparse feature enhancement module that aggregates voxel features and leverages auxiliary tasks to improve the backbone’s feature representation and suppress background interference. Additionally, an integrated data augmentation method generates training samples that align with ALS data distributions during network training, while increasing terrain complexity during testing. Experiments on two ALS point cloud datasets demonstrate that TG-ADet significantly outperforms state-of-the-art methods and achieves robust detection performance in challenging scenarios.

LiDAR and photogrammetry are active and passive remote sensing techniques for point cloud acquisition, respectively, offering complementary advantages and heterogeneous. Due to the fundamental differences in sensing mechanisms, spatial distributions, and coordinate systems, their point clouds exhibit significant discrepancies in density, precision, noise, and overlap. Coupled with the lack of ground truth for large-scale scenes, integrating the heterogeneous point clouds is a highly challenging task. This article proposes a self-supervised registration network based on a masked autoencoder, focusing on heterogeneous LiDAR and photogrammetric point clouds. At its core, the method introduces a multiscale masked training strategy to extract robust features from heterogeneous point clouds under self-supervision. To further enhance registration performance, a rotation-translation embedding module is designed to effectively capture the key features essential for accurate rigid transformations. Building upon robust representations, a transformer-based architecture seamlessly integrates local and global features, fostering precise alignment across diverse point cloud datasets. The proposed method demonstrates strong feature extraction capabilities for both LiDAR and photogrammetric point clouds, addressing the challenges of acquiring ground truth at the scene level. Experiments conducted on two real-world datasets validate the effectiveness of the proposed method in solving heterogeneous point cloud registration problems.

Bidirectional Reflectance Distribution Function (BRDF) plays a vital role in quantitative remote sensing. Recently, UAV has gradually emerged as the leading choice for BRDF acquirement. However, challenges remain in extracting usable BRDF data from UAV multispectral images (MSIs), including labor-intensive and limited accuracy. To tackle these challenges, an automated workflow for pixel-level BRDF extraction is proposed in this paper, comprising two main stages: 3D reconstruction and back projection. Pixel-level accuracy is achieved through 3D reconstruction, enhanced by the integration of commercial software for streamlined automation. Back projection is critical for precise location. Experiments were carried out for both selected ROIs and all pixel areas, validating the efficacy of our proposed approach.

Small-footprint waveform LiDAR requires waveform decomposition for accurate target structure characterization. To improve the ability for identifying close targets, this paper first introduces the Landweber (LW) deconvolution method to decompose the small-footprint LiDAR waveforms. Our study emphasizes the advantages of the deconvolution methods in capturing more targets of waveforms. Generally, the LW approach introduced with optimization excels in detecting more targets after false target removal. Experiments were conducted on datasets collected under various conditions using small-footprint waveform LiDAR system. The findings highlight an average target distance error of 0.083m, showcasing superior performance compared to direct decomposition methods. When compared with the GOLD and RL methods, the decomposition accuracy is nearly indistinguishable, but the success rates are higher. Our research establishes the LW method as a viable waveform decomposition method, contributing to the diversity of choices for waveform data processing.

Bidirectional Reflectance Distribution Function (BRDF) is an important quantity in remote sensing, describing the variations of reflectance factors with viewing geometries. Current empirical or semi-empirical BRDF models are often constrained to limited types of landcovers due to the assumptions regarding surface cavity distribution. In this paper, a universal approach to remote sensing BRDF approximation based on theHemiSphere Harmonics (HSH)basis function is proposed. We derived the HSH, which match the BRDF definition domain, as basis functions of an infinite series to achieve high-precision BRDF representation. Besides, the proposed approach is universal across various landcovers since HSH basis functions are complete and orthogonal, enabling to approximate arbitrary BRDF data. To our knowledge, it is the first time to utilize the basis function for the BRDF approximation in remote sensing. The proposed approach was validated on both the satellite dataset with 16 landcovers and UAV dataset with 6 landcovers. The results demonstrate that the proposed approach outperforms current BRDF models, and show universality across different landcovers, spectral bands and platforms.

Small-footprint waveform LiDAR necessitates waveform decomposition for accurate target structure characterization. However, the limited range resolution and heavy computations lead to this being hindered. Addressing these challenges, we propose a high-resolution and efficient waveform decomposition method on fundamentals of the LiDAR physics model. To enhance LiDAR ranging resolution, we introduce a novel technique that separates the transmitted pulses and received multi-target waveforms to simulate narrow transmitted pulse conditions. Then, the separated pulses and waveforms are input into a deconvolution algorithm, which incorporates an automatic stopping criterion for iteration to ensure accurate results. For efficient processing of waveforms, we design a lightweight classification method that categorizes waveforms into single-target and multi-target waveforms before waveform decomposition, with only the latter undergoing complex downstream processing. Indoor and airborne experiments are conducted on datasets collected using small-footprint full waveform LiDAR. The indoor results demonstrate that the average target distance error is reduced to 0.064mwith a significant improvement in efficiency, surpassing mainstream methods. The airborne results reveal that our method is able to decompose faster to get more points for better structural characterization.

Airborne LiDAR point cloud segmentation (PCS) is often employed as a preprocessing step for the subsequent object recognition for scene interpretation. Current segmentation methods often aim at single-wavelength LiDAR data by fully exploiting the spatial information, which makes them unsuitable for multispectral point cloud (MPC) data due to ignoring the use of spectral signatures. In this article, a normalized spatial–spectral supervoxel segmentation method is proposed for MPC data. Specifically, a normalized spectral–spatial metric is developed to construct the${k}$-dimensional tree (KD tree) for MPC data clustering. Considering the uneven density distribution of MPC, an adaptive energy minimization principle based on the sum of the distance is devised to accurately select the seed points of voxels, solving the problem of undersegmentation. To reduce the cross-boundary points, the normalized spectral–spatial metric with the concave–convex judgment is extended to further optimize the edges between adjacent voxels. An important asset of our method is to segment MPC without the need for any manual annotation. Experiments on two MPC datasets show that the proposed method yields better performance compared to several comparative methods.

Spectral reconstruction based on satellite multispectral (MS) images can produce high spatial resolution hyperspectral (HS) images at a reasonable cost, significantly expanding the application of satellite-based HS remote sensing. As a challenging ill-posed problem, existing methods have difficulty making full use of local and global information of space and spectra to guide the reconstruction, resulting in limited accuracy in large-scale scenes with complex ground features and severe spectral mixing. In this article, we propose a novel convolution and Transformer joint network (CTJN) to address the challenge of high-accuracy spectral reconstruction in complex scenes. The CTJN is cascaded with shallow feature extraction modules (SFEMs) and deep feature extraction modules (DFEMs), which can explore local spatial features and global spectral features. Besides, a high-frequency Transformer block (HF-TB) is designed to highlight the detailed features of the images to prevent significant high-frequency information loss, which could improve the reconstruction results in regions with drastic feature changes. Moreover, a spatial–spectral recalibration block (SSRB) is proposed to perform explicit constraints on the reconstructed points by exploiting the correlation among neighboring pixels and adjacent spectra. Extensive experimental results on four HS–MS datasets and one MS dataset demonstrate that the proposed CTJN outperforms the state-of-the-art methods in large-scale and small-scale scenes.

LiDAR strip adjustment is a key prerequisite for subsequent applications based on point cloud data since it inevitably suffers from spatial discrepancies caused by laser ranging errors, mounting errors, etc. Most current LiDAR strip adjustment methods rely on the extraction of structural features which are often unsuitable for non-urban scenes. Alternative strip adjustment methods based on correspondence distance minimalization ignore spatial alignment. To overcome these limitations, this paper presents an accurate spatial alignment method for UAV LiDAR strip adjustment in non-urban scenes. Firstly, we construct a novel point cloud feature descriptor called Spherical Shell Point Feature (SSPF) to extract multi-dimensional non-structural features that are robust to non-urban point clouds. The constructed SSPF is then combined with point coordinates to generate embedded features, which simultaneously consider the point coordinates and spatial alignment. Finally, the embedded features are utilized by a two-stage matching method to match pair-wise points of two adjacent strips. The proposed method is validated on two non-urban datasets collected by two types of LiDARs, which reduces the digital surface model discrepancies by 0.252mand 0.221m, respectively, and proves its superiority compared to mainstream strip adjustment methods as well.

Multispectral point cloud is a novel type of data rich in spectral and spatial information. 3D reconstruction is a low-cost solution for acquiring multispectral point cloud. However, most of the existing methods have been developed for RGB images, which are inapplicable to multispectral images due to the special structure of multispectral sensors and the nonlinear intensity differences. In this paper, a robust 3D reconstruction method for multispectral images is proposed to generate multispectral point cloud by harnessing their spatial and spectral information. Considering the characteristics of multispectral image acquisition, reflectance correction and band alignment steps are introduced into the proposed method, aiming to reduce the impact of band differences and spatial errors on 3D reconstruction. Subsequently, a fused multispectral feature extraction is employed to provide more potential reconstruction feature points. To reduce the mismatched feature points induced by the spectra of vegetation regions, an NDVI-guided feature matching algorithm is proposed that provides accurate correspondence calculation for multispectral images reconstruction. The experiments compared with several well-known methods and a commercial software on two datasets have shown superior reconstruction performance.

Image denoising and classification are typically conducted separately and sequentially according to their respective objectives. In such a setup, where the two tasks are decoupled, the denoising operation does not optimally serve the classification task and sometimes even deteriorates it. We introduce here a unified deep learning framework for joint denoising and classification of high-dimensional images, and we particularly apply it in the framework of hyperspectral imaging. Earlier works on joint image denoising and classification are very scarce, and to the best of our knowledge, no deep learning models were proposed or studied yet for this type of multitask image processing. A key component in our joint learning model is a compound loss function, designed in such a way that the denoising and classification operations benefit each other iteratively during the learning process. Hyperspectral images (HSIs) are particularly challenging for both denoising and classification due to their high dimensionality and varying noise statistics across the bands. We argue that a well-designed end-to-end deep learning framework for joint denoising and classification is superior to current deep learning approaches for processing HSI data, and we substantiate this by results on real HSI images in remote sensing. We experimentally show that the proposed joint learning framework substantially improves the classification performance compared to the common deep learning approaches in HSI processing, and as a by-product, the denoising results are enhanced as well, especially in terms of the semantic content, benefiting from the classification.

Unmanned aerial vehicles (UAVs) equipped with hyperspectral scanners and LiDARs can flexibly acquire rich spectral and geometric information about the observation scene. To combine the complementary advantages of multi-source data, the stereo registration of hyperspectral images and LiDAR data has become one of the hot topics in remote sensing community. However, existing research works are more focused on exploiting intensity information from multi-source data, which is applicable to data acquired on manned vehicle platforms or satellite platforms. For UAV platforms with poor stability and limited load, the low signal-to-noise ratio of LiDAR data and the complex distortion of push-broom images bring great challenges to stereo registration. Under this circumstance, an intensity-independent stereo registration method is proposed in this paper, which is based on the physical model of the integrated system and the sensor detection principles. Specifically, the proposed method utilizes the position and orientation system (POS) to reduce the impact of UAV platform motion on hyperspectral imaging, and projection errors are eliminated by the ray tracing model with aid of LiDAR data. Finally, a virtual ray decomposition model based on geometric features is constructed to realize the stereo registration of hyperspectral images and LiDAR data. Compared with an advanced solution and professional processing software, the proposed method has shown better registration performance on two data of different scenarios.

Hyperspectral intrinsic image decomposition (HyperIID) has been proven to be a very useful approach to reduce the spectral uncertainty in the remote sensing imaging process and improve the classification. In this article, a new HyperIID with enhanced spatial information, called ESI-IID, is proposed to overcome the deficiency of low spatial resolution in the existing HyperIID methods. With the aid of high-resolution (HR) panchromatic (PAN) image, the proposed method embeds the HR spatial information into the intrinsic decomposition model and enhances spatial details of the intrinsic component. The proposed ESI-IID introduces three constraints: 1) we make the constraint on spectral information to protect it from distortion during the spatial resolution enhancement process; 2) we add the constraint on spatial information to make sure that the details of edges will be well kept; and 3) based on the assumption that the reflectance component has a strong correlation in the local neighborhood, we add the self-constraint on reflectance component, in which the similarity matrix consists of two parts extracted from hyperspectral images and PAN image, respectively. Finally, we build a matrix energy function according to the aforementioned constraints and solve it by finding the minimum Frobenius norm iteratively. Both visual and quantitative experiments on simulated and real datasets demonstrate that the proposed method outperforms other alternative methods with high reliability.

Convolutional neural network (CNN) has been widely applied in hyperspectral image (HSI) classification exhibiting excellent performance. Weak generalization of CNN models to different datasets is a common issue in this domain largely because of limited amount of labeled training samples. In this article, we propose afullygroup convolutional neural network (FGCNN) method that integrates cascades of shuffled group convolutions tailored to different network stages. To our knowledge, this is the first reported full-group CNN model in general, and we design it in particular for robust spectral–spatial classification of HSI. In the primary feature extraction stage, we develop an original multiscale spectral feature extraction approach based on a novel concept of multikernel depthwise convolution that we define in terms of shuffled and importance-weighted group convolution. In the subsequent stage, we introduce a discriminative spectral–spatial feature extraction method with a novel group competition block to capture informative features with relatively few parameters. The final feature fusion stage is defined as a novel lightweight group feature fusion method that sharply reduces fusion weights compared to traditional methods with fully connected layers. Experimental results on three datasets show that the proposed FGCNN yields robust classification accuracy under the same hyperparameter settings compared to the current state-of-the-art.

Recent progress in spectral classification is largely attributed to the use of convolutional neural networks (CNNs). While a variety of successful architectures have been proposed, they all extract spectral features from various portions of adjacent spectral bands. In this article, we take a different approach and develop a deep spectral feature fusion method, which extracts both local and interlocal spectral features, capturing thus also the correlations among nonadjacent bands. To our knowledge, this is the first reported deep spectral feature fusion method. Our model is a two-stream architecture, where an intergroup and a groupwise spectral classifier operate in parallel. The interlocal spectral correlation feature extraction is achieved elegantly, by reshaping the input spectral vectors to form the so-called nonadjacent spectral matrices. We introduce the concept of groupwise band convolution to enable the efficient extraction of discriminative local features with multiple kernels adopting the local spectral content. Another important contribution of this work is a novel dual-channel attention mechanism to identify the most informative spectral features. The model is trained in an end-to-end fashion with a joint loss. Experimental results on real datasets demonstrate excellent performance compared with the current state of the art.

Recent progress in spectral classification is dominated by the use of deep learning models. While various learning architectures have been developed, they all extract spectral features from a single view input. In this paper, we investigate a different perspective and develop a unified multiview spectral feature learning framework, which extracts discriminative spectral features from multiple views of inputs. To our knowledge, this is the first reported multiview spectral feature learning method based on deep learning. In this framework, we introduce a multiview spectrum construction method by transforming the input spectral vector into multiple 3D image patches with different sizes, termed as multiview spectrum. This multiview spectrum is fed to a well-designed triple-stream architecture, where a global and two local spectral feature learning networks operate in parallel, capturing thus both global and local spectral contextual features simultaneously. Another important contribution of this work is a novel interactive attention mechanism to identify the most informative spectral contextual features. The model is trained in an end-to-end fashion from scratch with a joint loss. Experimental results on four data sets demonstrate excellent performance compared to the current state-of-the-art.

Hyperspectral satellite data has been widely applied in many fields due to its numerous bands. Along with the advantages of high spectral resolution, hyperspectral satellite data are still limited by some disadvantages of high acquisition cost, low revisiting capability, and low spatial resolution. Compared with hyperspectral satellites, multispectral satellites have a large number, large width, strong coverage and high spatial resolution. Therefore, multispectral data can be used as the input to the spectral reconstruction to obtain hyperspectral data with high temporal resolution. Better hyperspectral data can be obtained by spectral reconstructing with these continuous multi-temporal data than with single-temporal data. A multi-temporal spectral reconstruction network (MTSRN) is proposed in this paper, which is used to reconstruct hyperspectral images from multi-temporal multispectral images. The proposed MTSRN comprises multiple single-temporal spectral reconstruction networks (STSRN) for extracting temporal features and a multi-temporal fusion network (MTFN). The parallel component alternative (PA) post-processing method enhances the physical plausibility of reconstructed hyperspectral data. To demonstrate performance of the proposed method in aspects of multi-temporal reconstruction, experiments are conducted on four multi-temporal hyperspectral and multispectral satellite datasets. The experimental results prove that the proposed MTSRN obtains better spectral reconstruction results compared with the spectral reconstruction method based on single-temporal information.

The multispectral imaging of Unmanned Aerial Vehicle (UAV) is often affected by the variation of illumination, resulting in serious spectral radiation distortion. Precise illumination estimation and compensation is a key step to carry out the radiometric correction on UAV multispectral images (MSIs), especially for the case without irradiance sensors. To accurately estimate the illumination for the radiometric correction, a physics-based illumination estimation and compensation method is proposed in this paper. In the proposed method, an illumination estimation model is built based on the intra-image hypothesis on illumination consistency and the inter-image hypothesis on reflectance consistency. This model is used to obtain the illumination irradiance of each one from numerous MSIs simultaneously. Then the influence of varying illumination can be alleviated with the estimated irradiance based on the physical imaging principle. To validate the effectiveness of the proposed method, numerical experiments are conducted on three UAV datasets acquired under cloudy weather. The experimental results demonstrate that the proposed method outperforms current methods and the Normalized Root Mean Square Error (NRMSE) on the three datasets are noticeably reduced.

The representation power of convolutional neural network (CNN) models for hyperspectral image (HSI) analysis is in practice limited by the available amount of the labeled samples, which is often insufficient to sustain deep networks with many parameters. We propose a novel approach to boost the network representation power with a two-stream 2-D CNN architecture. The proposed method extracts simultaneously, the spectral features and local spatial and global spatial features, with two 2-D CNN networks and makes use of channel correlations to identify the most informative features. Moreover, we propose a layer-specific regularization and a smooth normalization fusion scheme to adaptively learn the fusion weights for the spectral-spatial features from the two parallel streams. An important asset of our model is the simultaneous training of the feature extraction, fusion, and classification processes with the same cost function. Experimental results on several hyperspectral data sets demonstrate the efficacy of the proposed method compared with the state-of-the-art methods in the field.

Convolutional Neural Network (CNN) has been widely applied in hyperspectral image (HSI) classification exhibiting excellent performance. The CNN model overfitting is a common issue in this domain due to limited amount of labelled training samples. In addition, making the full use of spectral information is still considered an open problem. In this paper, we propose a novel group 2D-CNN model for spectral-spatial classification. Specifically, we propose an original multi-scale spectral feature extraction approach based on a novel concept of multi-kernel depthwise convolution. Furthermore, we exploit for the first time shuffle operation on the group convolutions in HSI spectral-spatial feature extraction to effectively limit the amount of learning parameters. As a result, we design a small and efficient network for HSI classification. Experimental results on real data demonstrate favourable performance compared to the current state-of-the-art.