Advanced interpretation of hyperspectral remote sensing images benefits many precise Earth observation tasks. Recently, visual foundation models have promoted the remote sensing interpretation but concentrating on RGB and multi-spectral images. Due to the varied hyperspectral channels, existing foundation models would face image-by-image tuning situation, imposing great pressure on hardware and time resources. In this paper, we propose a tuning-free hyper-spectral foundation model called HyperFree, by adapting the existing visual prompt engineering. To process varied channel numbers, we design a learned weight dictionary covering full-spectrum from 0.4 ∼ 2.5 μm, supporting to build the embedding layer dynamically. To make the prompt design more tractable, HyperFree can generate multiple semantic-aware masks for one prompt by treating feature distance as semantic-similarity. After pre-training HyperFree on constructed large-scale high-resolution hyperspectral images, HyperFree (1 prompt) has shown comparable results with specialized models (5 shots) on 5 tasks and 11 datasets. Code and dataset are accessible at https://rsidea.whu.edu.cn/hyperfree.htm.

Air pollution poses a significant threat to human health. Effective and efficient assessing individual exposure intensity will provide a crucial reference to mitigate potential air pollution risk. Previous proximity models in scenario simulation method neglect the variations among individual absorption efficiency. In this study, we proposed a novel demographic optimization proximity model (DOPM) to quantified sulfur dioxide (SO2) exposure risk under eight groups’ respiratory rates. In assessing exposure risk in Wuhan, one of the megacities in central China, we compared the capacity of DOPM and a classic proximity model on simulating exposure intensity, utilizing the near-Gaussian bi-square diffusion function and the filter optimal bandwidth. Subsequently, we utilized ordinary kriging (OK) interpolation to visualize the SO2 exposure risk map based on the simulation results of DOPM. We found that 9.5 km was the optimal bandwidth when near-Gaussian bi-square diffusion function utilized in proximity model. We also observed DOPM has stronger ability to simulate individual exposure intensity with a R-square of 0.44 when distinguished respiratory rate among receptors. These insights will aid public health researchers in assessing exposure risk across absorption efficiency and help local government officials devise more effective air pollution control strategies.

Night object tracking (NOT) is aimed at tracking objects under low-illumination conditions at night. Existing works concentrate on thermal infrared modality, while some RGB and thermal infrared (RGB-T) data also contain night scenes. However, night scenes in these datasets are mostly well-lit, making it challenging to fully cover low-illumination scenarios. In this article, we focus on the NOT task and build up a novel low-light visible and thermal infrared (LOL-T) multimodal benchmark dataset for NOT-156. To achieve night vision, we design a common-aperture LOL-T camera by integrating a highly dynamic low-light visible imaging sensor with a thermal infrared sensor in a common aperture optical system. The proposed dataset consists of 156 video sequences and a total of 170k annotated frames, including various low-illumination night scenes such as dark rooms, streets, corridors, and so on. Compared with existing datasets, NOT-156 has more comprehensive and distinctive attributes (thermal variation, noise, high illumination overexposure, etc.). Comprehensive experiments are carried out to evaluate the performance of the advanced visible, infrared, and visible-thermal trackers on the proposed NOT-156 dataset. The authors believe that NOT-156 has great potential in the application and development of night vision. The dataset will be made available athttp://rsidea.whu.edu.cn/NOT156_dataset.htm.

Crop mapping is an important task for agriculture-related activities and economic development. Most present studies focus on the crop mapping of staple crops, i.e., soybean, maize, and wheat, few concentrate on the multi-level fine-grained crop classification, which requires knowing finer crop type, i.e., barley or rye, winter wheat or spring wheat for different application. Deep learning methods with the strong ability to extract features automatically have great potential in fine-grained crop mapping. However, the classification of multi-level finer crops is challenged by the extremely similar phenological characters. In this paper, a multi-level fine-grained crop classification method based on multi-expert knowledge distill is proposed to learn the phenological features with different distinction degrees. Specifically, it uses three expert models to distinguish crop types with obvious, similar, and confusing phenological features. Then through a self-paced learning module, the student model first learns the knowledge from three expert models in the early stage and then learns to excavate the phenological features actively during the learning process. The experiment was carried on in Nordrhein Westfalen, Germany based on the EUROCROPS and time-series Sentinel-2 dataset and achieved great performance compared with popular deep learning methods.

Remote sensing (RS) change detection (CD) refers to the difference of RS images at different times in the same geographic area. The method of using high-resolution RS images based on deep learning (DL) has the problem of poor domain adaptation (DA) under geographical isolation conditions, especially in the cropland scene with different phenology and planting type. Based on the universal Siamese network architecture, a multi-view domain confusion (MVDC) module is designed. The module performs discriminant operations on the generated source domain features and target, and guides the generator to generate more domain invariant features. The time consistency information is taken into account while the difference between geographical regions is reduced. The module can theoretically be plugged into any universal CD network. The experimental results show that the adaptive ability of multiple CD methods is improved after the addition of MVDC.

Road extraction from high-resolution remote sensing imagery is important and efficacious due to deep learning. However, most methods face challenges in capturing narrow roads, i.e. rural roads. In this paper, we propose a saliency-aware deep network (SAN) for narrow road extraction. Specifically, a multi-scale context module is employed to extract contextual features in different scales, and a global context module is followed to aggregate the above multi-scale context features, to improve the connectivity of narrow roads. In addition, motivated by visual saliency, a saliency-aware module is proposed to highlight roads during skip connections, to further separate narrow roads from complex backgrounds. In the experiments, SAN is compared with some state-of-the-art methods by using the DeepGlobe road dataset and achieves better performances, especially for narrow roads, which proves its superiority.

Real hyperspectral images (HSIs) are ineluctably contaminated by diverse types of noise, which severely limits the image usability. Recently, transfer learning has been introduced in hyperspectral denoising networks to improve model generalizability. However, the current frameworks often rely on image priors and struggle to retain the fidelity of background information. In this article, an unsupervised adaptation learning (UAL)-based hyperspectral denoising network (UALHDN) is proposed to address these issues. The core idea is first learning a general image prior for most HSIs, and then adapting it to a real HSI by learning the deep priors and maintaining background consistency, without introducing hand-crafted priors. Following this notion, a spatial-spectral residual denoiser, a global modeling discriminator, and a hyperspectral discrete representation learning scheme are introduced in the UALHDN framework, and are employed across two learning stages. First, the denoiser and the discriminator are pretrained using synthetic noisy-clean ground-based HSI pairs. Subsequently, the denoiser is further fine-tuned on the real multiplatform HSI according to a spatial-spectral consistency constraint and a background consistency loss in an unsupervised manner. A hyperspectral discrete representation learning scheme is also designed in the fine-tuning stage to extract semantic features and estimate noise-free components, exploring the deep priors specific for real HSIs. The applicability and generalizability of the proposed UALHDN framework were verified through the experiments on real HSIs from various platforms and sensors, including unmanned aerial vehicle-borne, airborne, spaceborne, and Martian datasets. The UAL denoising scheme shows a superior denoising ability when compared with the state-of-the-art hyperspectral denoisers.

Hyperspectral object tracking (HOT) aims at tracking targets using the rich spectral information from hyperspectral video. Recently, dual Siamese network (DSN) has been proposed for HOT with advanced performances, via integrating a RGB Siamese branch with a hyperspectral Siamese branch to solve small sample challenge of hyperspectral modality. However, there are still challenges of DSN that reduce its practicality: a single DSN model is difficult to process hyperspectral videos with varied channels; the spatial features extracted by the pre-trained RGB branch plays a dominant role, while the hyperspectral features are not fully explored. To address the challenges, we propose a Channel AdapTive dual Siamese network, termed SiamCAT, for HOT with varied channels. Specifically, treating each frame of hyperspectral video as a grayscale image sequence varied with wavelengths, a channel adaptive module is introduced to encode the grayscale image sequence of different lengths into a uniform length, and so that SiamCAT can process hyperspectral video with varied channels. Meanwhile, a guided learning attention module is proposed to progressively learn spectral features of the tracked target highlighted by the spatial attention of the pre-trained RGB branch. Note that, to force spectral features play a leading role, instead of traditional features fusion, the spectral features extracted by the hyperspectral branch are utilized for confirming the target position. In the experiments, SiamCAT were verified by using the HOT competition dataset (i.e., 16-channel, 25-channel, and 15-channel hyperspectral videos with different wavelength ranges) and the WHU-Hi-H3dataset (25-channel hyperspectral videos), and achieved advanced performances.

Crop mapping in a cloudy area is always a challenge due to the lack of time-series clear optical satellite imagery. Making use of time-series synthetic aperture radar (SAR) imagery that is immune to cloud contamination is essential and promising for seamless and large-area crop mapping. However, existing deep learning (DL)-based crop classification methods give the extracted phenological features equal weights, without considering the different contributions of phenological features of the different crop growth periods. In this article, a phenology-based crop mapping network (PhenoCropNet) is proposed to extract the discriminative features from the two levels, including the key phenological dates in the phenological periods and key phenological periods in the whole growth stages. PhenoCropNet includes a phenological calendar information injection (PAI) module that divides the satellite imagery time series (SITS) into multiple sequences according to the phenological calendar information, and a hierarchical attention network structure that uses the two-level bidirectional gated recurrent unit-based self-attention (BiGRUA) modules to automatically extract the features containing the most important phenological information of key phenological dates and key phenological periods. The proposed PhenoCropNet was verified in Hubei province in China, around 185 933 km2, a typical cloudy area in China, for rapid winter crop mapping based on temporal Sentinel-1 SAR imagery. The mapping result shows that the$F1$-score of PhenoCropNet for winter crop mapping could achieve 0.90, showing great potential in large-scale and seamless crop mapping. The code is available on request:https://github.com/LL0912/PhenoCropNet.

Hyperspectral anomaly detection (HAD) aims to find small targets deviating from surroundings in an unsupervised manner. Recently, various deep models have been applied to HAD, such as autoencoder series and generative adversarial networks (GAN) series, which mainly use a proxy task, i.e., iteratively reconstructing low-frequency components (backgrounds) to separate anomalies (two-step paradigm). However, in such an unsupervised manner, most deep HAD model is trained and tested on the same image. Since the learned low-frequency background varies from image to image and the trained model cannot be directly transferred to unseen images. In this paper, the one-step detection paradigm is first proposed, where the model is optimized directly for the HAD task and can be transferred to unseen datasets. The one-step paradigm is optimized to identify the spectral deviation relationship according to the anomaly definition. Compared to learning the specific background distribution in the two-step paradigm, the spectral deviation relationship is universal for different images and guarantees transferability. Further, we instantiated the one-step paradigm as an unsupervised transferred direct detection (TDD) model. To train the TDD model in an unsupervised manner, an anomaly sample simulation strategy is proposed to generate numerous pairs of anomaly samples. A global self-attention module and a local self-attention module are designed to help the model focus on the “spectrally deviating” relationship. The TDD model was validated on six public datasets. The results show that TDD is superior to the recent two-step methods in detection and transferability aspects.

Earth anomalies can locate valuable targets in an unsupervised manner for many defense and surveillance applications. Most models assign a continuous score at the pixel-level, resulting in object-agnostic results with higher false alarms than the instance level results. However, since the anomaly objects contain a variety of categories and have a large intraclass variance, the current state-of-the-art (SOTA) query-based models designed for certain categories perform unsatisfactorily when applied to the anomaly instances. The larger intraclass variance of anomalies makes the learning of general representation more difficult. To bridge this gap, we propose general adaptations guided by the pixel-level anomaly map for any query-based model, which adapts the model from learning certain category representation to learning anomaly-aware representation in different categories. The proposed adaptation first builds a separate branch to output the pixel-level anomaly map, where anomaly information is then extracted to guide the pixel embeddings and queries to focus on a variety of anomaly categories. Especially, the anomaly rank embeddings are devised to make the pixel embeddings aware of the anomaly rank order. The queries are dynamically selected from the anomaly candidates after aligning the anomaly map and pixel embeddings for better locating different anomalies. Finally, the selected queries dot-product the anomaly-aware pixel embeddings to output the anomaly instances. The proposed adaptations are simple, general, and additive, which bring the average improvements of +4.9 box AP and +5.1 mask AP in infrared, synthetic aperture radar (SAR), and hyperspectral modalities.

High spatial, high spectral, and high temporal ($\text {H}^{3}$) information of the objects of interest can be provided by hyperspectral video, which makes it possible to track objects in complex scenarios. However, during motion, changes in the target’s appearance, background, and spectral information can degrade the performance of existing hyperspectral trackers due to insufficient training data. Consequently, this results in weak generalization of these trackers. In this article, to solve the above problems, a deep temporal-spectral–spatial anchor-free Siamese tracking network for hyperspectral video object tracking, namely HA-Net, is proposed. In HA-Net, a Siamese spectral enhancement tracker module based on an RGB tracker (pseudo-color tracker) is designed, which uses the powerful feature expression capabilities of the deep network to learn more discriminative deep spectral features for identifying objects in complex scenarios. The pseudo-color tracker is introduced to solve the problem of model performance limitation due to insufficient training data. By introducing the temporal-spectral–spatial online discrimination learning module, the temporal-spectral–spatial information of the target can be dynamically modeled to adapt to new targets and the dynamic changes of targets. Benefiting from the double Siamese network architecture, the model can be effectively trained from scratch with less than 20 000 training samples. Online learning of temporal-spectral–spatial information for the target, particularly in cases of insufficient training data, can alleviate the issue of model degradation. This approach enhances the model’s robustness when tracking the target in complex scenes. In the 2021 IEEE WHISPERS Hyperspectral Object Tracking (HOT) Challenge, HA-Net obtained the best performance, with a distance precision (DP) score of 0.948 and an area under the curve (AUC) score of 0.688. The running speed is also 14 frames/s, which is superior to the existing hyperspectral object trackers for hyperspectral video. The source code is available athttps://github.com/zhenliuzhenqi/HOT.

Positive and unlabeled (PU) learning is aimed at building a binary classifier to distinguish the target from the background using only the known positive samples, which is an advanced solution for the hyperspectral target detection (HTD) task. However, when PU learning (PUL) meets complex hyperspectral scenarios, there are two main challenges: 1) How to estimate the class prior accurately? The class prior, i.e., the target proportion, is an important prior for PUL to learn the discriminant boundary, but it is difficult to estimate in hyperspectral imagery, due to the interclass spectral similarity and 2) How to remove redundancy and improve the discriminative features of the target? The diagnostic spectral feature extraction is important for the weakly supervised PUL models as it can help with separating the target from the background. In this article, to tackle these challenges, a robust PUL framework with key band selection (PU-KBS) is proposed, which is modeled as an end-to-end and class prior free PUL framework, where the accurate class prior and the most discriminative key band subset are jointly initialized and iteratively updated until reaching the optimal result by evolutionary search. Meanwhile, a deep PUL detector is introduced for guiding the subsequent search direction and discriminative deep feature extraction. The proposed PU-KBS framework was verified using different hyperspectral datasets, where accurate class prior estimation, diagnostic spectral characteristics, and robust detection results could be obtained simultaneously by the PU-KBS framework. Furthermore, the improvement in band selection interpretability and detection performance was proven experimentally.

Agricultural parcel vectorization is important for precision agriculture when analyzing crops at the parcel scale. However, it is a challenging task to vectorize parcels from satellite imagery, where under-segmentation and the discontinuous boundaries of parcels are common problems, due to the varied shapes and sizes of parcels caused by the terrain and the farming mode, and the blurred boundaries caused by the limited resolution and the shadows. In this paper, a region-boundary-parcel decoupled multi-task learning (RBP-MTL) framework is proposed for agricultural parcel vectorization, where the local spatial constraints between the regions, boundaries, and the objects of each parcel are jointly modeled via multi-task learning, to promote the object separability of agricultural parcels and the boundary connectivity. In addition, simple and efficient boundary-object interaction vectorization is introduced to further ensure that each parcel is independent with a complete and separable boundary. In the experiments, the proposed framework was verified using three datasets with different spatial resolutions, i.e., the public parcel extraction dataset from the iFLYTEK Challenge 2021 (~1 m), in addition to a single-temporal Gaofen-1 dataset from Hubei province in China (2 m) and a multi-temporal Sentinel-2 dataset from the Netherlands (10 m), which were annotated and built by the authors. The proposed RBP-MTL framework achieved a state-of-the-art accuracy, with effective extraction of complete parcel boundaries in the different scenes.

An inevitable trend of hyperspectral remote sensing has been toward hyperspectral with high spatial resolution (H2) images. However, the higher resolution also brings higher spatial/spectral heterogeneity of surface features, which increases the difficulty of fine classification. Fully using global spatial–spectral features and contextual information is an effective method to alleviate spatial/spectral heterogeneity. Recently, to extract global spatial–spectral features with long-range dependencies, the self-attention mechanism has been widely used in H2 image classification and has achieved excellent results. As is well known, the simultaneous use of spatial and spectral information has always been a key aspect of hyperspectral image (HSI) processing; however, the current spatial and spectral attention modules only focus on the spatial and spectral features separately. This prevents further improvement in network performance, especially when the sample size is small. Therefore, a spatial–spectral attention-in-attention network (S2AiANet) is proposed, which solves the problem of the current spatial–spectral attention maps only focusing on single features through the spatial–spectral attention-in-attention (S2AiA) module. In addition, a multiscale attention (MSA) module is proposed to enhance the network’s adaptability to various complex scenarios. The experiments on two H2 datasets and one classic HSI dataset demonstrate that S2AiANet can achieve a significant performance improvement compared with the state-of-the-art hyperspectral classifiers.

Deep learning has emerged as the predominant approach for multispectral and hyperspectral image fusion. However, most fusion networks are typically trained and validated on hyperspectral and multispectral image pairs generated from the same hyperspectral images, with degradation simulations inconsistent with real situations and relatively limited volumes of images. When transferring a pretrained multispectral and hyperspectral image fusion model from ground or airborne images to spaceborne images, it encounters a larger dataset and more complex spatial-spectral degradation, leading to spectral distortions and spatial artifacts in the fused images. In this article, the challenges associated with the transfer are addressed through the introduction of a self-supervised multispectral and hyperspectral image fusion unrolling network for spaceborne imagery, termed as MH-FUNet. MH-FUNet adopts a self-supervised paradigm to learn a robust mapping from spaceborne data. It utilizes a deep unrolling network to iteratively refine fusion results from coarse to fine. To account for spatial scale differences between the self-supervised training and test datasets, a multiscale fusion strategy is introduced. This strategy is combined with spectral and spatial attention mechanisms to restore spatial and spectral details. Additionally, a gradient constraint unit is proposed to maintain spatial consistency when up-scaling low-resolution hyperspectral imagery. Performance evaluation of the proposed method is conducted against state-of-the-art fusion techniques on both simulated Chikusei dataset and the proposed real WHU-MHF dataset, which consists of simultaneously observed hyperspectral and multispectral image pairs. MH-FUNet outperforms existing methods across all datasets, demonstrating superior performance in spaceborne multispectral and hyperspectral image fusion experiments.

Remote sensing anomaly detector can find the objects deviating from the background as potential targets for Earth monitoring. Given the diversity in earth anomaly types, designing a transferring model with cross-modality detection ability should be cost-effective and flexible to new earth observation sources and anomaly types. However, the current anomaly detectors aim to learn the certain background distribution, the trained model cannot be transferred to unseen images. Inspired by the fact that the deviation metric for score ranking is consistent and independent from the image distribution, this study exploits the learning target conversion from the varying background distribution to the consistent deviation metric. We theoretically prove that the large-margin condition in labeled samples ensures the transferring ability of learned deviation metric. To satisfy this condition, two large margin losses for pixel-level and feature-level deviation ranking are proposed respectively. Since the real anomalies are difficult to acquire, anomaly simulation strategies are designed to compute the model loss. With the large-margin learning for deviation metric, the trained model achieves cross-modality detection ability in five modalities-hyperspectral, visible light, synthetic aperture radar (SAR), infrared and low-light-in zero-shot manner.

Anomaly segmentation in high spatial resolution (HSR) remote sensing imagery is aimed at segmenting anomaly patterns of the earth deviating from normal patterns, which plays an important role in various Earth vision applications. However, it is a challenging task due to the complex distribution and the irregular shapes of objects, and the lack of abnormal samples. To tackle these problems, an anomaly segmentation model based on pixel descriptors (ASD) is proposed for anomaly segmentation in HSR imagery. Specifically, deep one-class classification is introduced for anomaly segmentation in the feature space with discriminative pixel descriptors. The ASD model incorporates the data argument for generating virtual abnormal samples, which can force the pixel descriptors to be compact for normal data and meanwhile to be diverse to avoid the model collapse problems when only positive samples participated in the training. In addition, the ASD introduced a multi-level and multi-scale feature extraction strategy for learning the low-level and semantic information to make the pixel descriptors feature-rich. The proposed ASD model was validated using four HSR datasets and compared with the recent state-of-the-art models, showing its potential value in Earth vision applications.

Positive-unlabeled learning (PU learning) in hyperspectral remote sensing imagery (HSI) is aimed at learning a binary classifier from positive and unlabeled data, which has broad prospects in various earth vision applications. However, when PU learning meets limited labeled HSI, the unlabeled data may dominate the optimization process, which makes the neural networks overfit the unlabeled data. In this paper, a Taylor variational loss is proposed for HSI PU learning, which reduces the weight of the gradient of the unlabeled data by Taylor series expansion to enable the network to find a balance between overfitting and underfitting. In addition, the self-calibrated optimization strategy is designed to stabilize the training process. Experiments on 7 benchmark datasets (21 tasks in total) validate the effectiveness of the proposed method. Code is at: https://github.com/Hengwei-Zhao96/T-HOneCls.

Deep neural networks have shown huge potential in hyperspectral unmixing (HU). However, the large function space increases the difficulty of obtaining the optimal solution with limited unmixing data. The autoencoder-based blind unmixing methods are sensitive to the hyperparameters, and the optimal solution can be difficult to obtain. Algorithm unrolling, which integrates deep learning and iterative algorithms, can shrink the search space and improve the efficiency of obtaining optimal results. Based on this, a model-driven deep neural network named the group sparsity regularized unmixing unrolling (GSUU) network, which unrolls a regularized matrix factorization objective function for blind HU, is proposed in this paper. Based on the nonnegative matrix factorization (NMF) optimization rules, the GSUU network contains two sub-networks—the A-Block and the S-Block—for alternately and iteratively estimating the optimal endmember spectra and abundance maps. The GSUU method incorporates the spatial group sparsity prior of the abundances, i.e., the fact that spatially adjacent mixed pixels share similar sparse abundances, into a deep unrolling network. The experimental results obtained with both synthetic and real hyperspectral data illustrate that the proposed algorithm can obtain a superior accuracy, compared to the other state-of-the-art unmixing algorithms.

Mixed pixels that contain more than one material type are common in mid/low spatial resolution remote sensing imagery. Hyperspectral unmixing is aimed at decomposing the mixed pixels into endmembers and abundances. However, there are few datasets that are suitable for quantitatively evaluating unmixing accuracies, and the ground-truth abundances of the existing datasets are often generated in an approximate way. To address the lack of real unmixing datasets for quantitative evaluation, we built the realistic mixing miniature scenes (RMMS) dataset, which can be used to quantitatively evaluate the unmixing accuracy of different algorithms. The RMMS dataset consists of a simple mixture scene with homogeneous flat materials and a complex mixture scene with 3-D structural features. The features of the RMMS dataset also take point, line, and polygon characteristics into consideration, and the spectral similarity of the materials increases the challenge of the spectral unmixing. In the RMMS dataset, due to the multiscale observation characteristics of the spatiotemporal scanning modality, it can avoid the registration error between RGB and hyperspectral data, and it can ensure that the endmembers are pure pixels. Most of the autonomous hyperspectral unmixing algorithms focus on solving some of the unmixing problems and have difficulty achieving fully autonomous hyperspectral unmixing (FAHU). In this article, to overcome this shortcoming, a fully autonomous hyperspectral unmixing method called FAHU is proposed to take advantage of the spatial information. Some of the state-of-the-art autonomous hyperspectral unmixing algorithms are used to evaluate the performance with the RMMS dataset, and the experimental results show the advantages and disadvantages of the different autonomous unmixing algorithms.

Hyperspectral object tracking is aimed at tracking targets by using both the spatial information and abundant spectral information, overcoming the drawbacks of traditional RGB tracking in complex scenarios, such as the low resolution or background clutter. However, the current hyperspectral object tracking methods usually have a high computational complexity, due to the huge data volume, making them difficult to apply to real-time applications on edge devices (e.g., robots, unmanned aerial vehicles, and satellites) with limited computational resources. In this paper, a lightweight dual Siamese network for onboard hyperspectral object tracking—termed SiamOHOT—is proposed for real-time and onboard tracking. Specifically, a joint spatial-spectral knowledge distillation method is proposed to teach a lightweight dual Siamese tracker to learn from a deep tracker— SiamHYPER—so that the number of parameters can be compressed to improve the computational efficiency. In addition, a deep learning inference optimizer is introduced to fuse the layers with similar functions and quantify the parameters of the network, to further promote the processing speed when deployed on an embedded platform. The proposed lightweight model was verified using the 2021 WHISPERS Hyperspectral Object Tracking Challenge dataset, and achieved a superior efficiency and accuracy. In addition, a prototype system was built integrating a snapshot hyperspectral imager, the SiamOHOT tracking algorithm, and an artificial intelligence edge device (NVIDIA Jetson Xavier NX), to realize real-time imaging and tracking. The inference speed of the optimized SiamOHOT network is nearly doubled when compared to the teacher model on the prototype system.

Hyperspectral imagery (HSI) one-class classification is aimed at identifying a single target class from the HSI by using only knowing positive data, which can significantly reduce the requirements for annotation. However, when one-class classification meets HSI, it is difficult for classifiers to find a balance between the overfitting and underfitting of positive data due to the problems of distribution overlap and distribution imbalance. Although deep learning-based methods are currently the mainstream to overcome distribution overlap in HSI multi-classificaiton, few researches focus on deep learning-based HSI one-class classification. In this paper, a weakly supervised deep HSI one-class classifier, namelyHOneClsis proposed, where a risk estimator—theOne-Class Risk Estimator—is particularly introduced to make the full convolutional neural network (FCN) with the ability of one class classification in the case of distribution imbalance. Extensive experiments (20 tasks in total) were conducted to demonstrate the superiority of the proposed classifier.

In deep learning (DL)-based hyperspectral imagery classification, “spatial patching” is primarily used as a preprocessing for incorporating local spatial information. This operation can help to promote classification accuracy but it is facing new challenges in the unmanned aerial vehicle (UAV)-borne hyperspectral imagery with high spatial and spectral resolutions (H2imagery). The ground objects’ various spatial scales result in it being challenging to determine the optimal size for the spatial patches. In addition, due to the severe spectral variability and spatial heterogeneity of the H2imagery, “spatial patching” only exploits the local spatial information and results in serious salt-and-pepper (SP) noise and isolated areas in the classification maps. In this article, to address these issues, a novel spectral patching network (SPNet) with an end-to-end DL architecture is proposed for UAV-borne H2imagery classification. The “spectral patching” approach is proposed to preserve the global spatial information and almost all the spectral information of the original hyperspectral imagery. An end-to-end deep encoder–decoder network is then constructed based on the spectral patching mechanism, which introduces the deep residual network (ResNet) and atrous spatial pyramid pooling (ASPP) modules to extract multiscale high-level semantic information for the H2imagery classification. The experimental results obtained with the Wuhan UAV-borne H2imagery (WHU-Hi) UAV-borne hyperspectral data set demonstrate that SPNet can achieve state-of-the-art accuracy and visualization performance in the classification of H2imagery.

Target tracking has received increased attention in the past few decades. However, most of the target tracking algorithms are based on RGB video data, and few are based on hyperspectral video data. With the development of the new “snapshot” hyperspectral sensors, hyperspectral videos can now be easily obtained. However, hyperspectral video target tracking datasets are still rare. In this article, a high spectral-spatial-temporal resolution hyperspectral video target tracking algorithm framework (H3Net) based on deep learning is proposed. The proposed framework consists of two main parts: 1) an unsupervised deep learning-based target tracking training framework for hyperspectral video; and 2) a dual-branch network structure based on a Siamese network. Using the dual-branch network, the H3Net framework can utilize both the spatial and spectral information. The combination of deep learning and a discriminative correlation filter (DCF) makes the features extracted by deep learning more suitable for the DCF. Compared with hyperspectral images, hyperspectral video data require more manpower to annotate, so we propose an unsupervised approach to train H3Net, without any annotation. To solve the problem of the lack of hyperspectral video datasets, we built a 25-band hyperspectral video dataset (the high spectral-spatial-temporal resolution hyperspectral video dataset: the WHU-Hi-H3dataset) for target tracking. The experimental results obtained with the WHU-Hi-H3dataset confirm the potential of unsupervised deep learning in hyperspectral video target tracking.

Hyperspectral images (HSIs) are inevitably corrupted with various types of noise, which seriously degrades the data quality and usability. Denoising is an essential preprocessing task of HSI processing. Recently, benefiting from the great learning ability of deep learning, convolutional neural network (CNN) denoisers have obtained state-of-the-art performances for Gaussian noise removal. However, one central problem remains largely unsolved: how to deal with the complicated noise in the real-world HSIs, especially when a paired training data set is unavailable. In this article, a self-supervised hyperspectral image denoising network (SHDN) is proposed, which consists of a noise estimator and a CNN denoiser. Rather than defining a complex noise model to generate training pairs on the clean HSIs, a self-supervised training scheme is first proposed by considering the noisy HSI itself as the training data. Through the noise estimator, the realistic noise samples can be extracted and combined with the clean bands to make up the training pairs. In addition, to jointly restore the target noisy band and to maintain the spectral consistency, a flexible multi-to-single band convolutional network is designed, where the noisy band and the neighboring bands are jointly aggregated via multiscale contextualized dilated blocks and the spectral–spatial convolutional unit. Experiments on HSIs from spaceborne, airborne, unmanned aerial vehicle (UAV)-borne, and ground-based data sets demonstrate the applicability and the generalization of SHDN in the real scenarios. Additionally, the usability of the noisy bands and the suitability of the SHDN framework in the subsequent applications are verified in the land-cover mapping experiments.

Hyperspectral anomaly detection is aimed at detecting observations that differ from their surroundings, and is an active area of research in hyperspectral image processing. Recently, autoencoders (AEs) have been applied in hyperspectral anomaly detection; however, the existing AE-based methods are complicated and involve manual parameter setting and preprocessing and/or postprocessing procedures. In this article, an autonomous hyperspectral anomaly detection network (Auto-AD) is proposed, in which the background is reconstructed by the network and the anomalies appear as reconstruction errors. Specifically, through a fully convolutional AE with skip connections, the background can be reconstructed while the anomalies are difficult to reconstruct, since the anomalies are relatively small compared to the background and have a low probability of occurring in the image. To further suppress the anomaly reconstruction, an adaptive-weighted loss function is designed, where the weights of potential anomalous pixels with large reconstruction errors are reduced during training. As a result, the anomalies have a higher contrast with the background in the map of reconstruction errors. The experimental results obtained on a public airborne data set and two unmanned aerial vehicle-borne hyperspectral data sets confirm the effectiveness of the proposed Auto-AD method.

Hyperspectral anomaly detection is aimed at detecting observations that differ from their surroundings. To achieve this goal, low-rank models and autoencoders (AEs) have attracted a lot of attention. Although the low-rank model is self-explainable, a low-rank prior may not completely match real data. In contrast, AEs can automatically learn the discriminative features between anomalies and background, whereas AEs are not self-explainable. In this article, a deep low-rank prior-based method (DeepLR) is proposed, which combines a model-driven low-rank prior and a data-driven AE. To be specific, the low-rank prior and a fully convolutional AE architecture are incorporated through modeling an energy minimization problem solved by an iterative optimization framework, in which low-rank background estimation and network training serve as two subproblems. The low-rank background is input into the network to calculate a low-rank regularized loss, constraining the training of the network. Finally, the background can be approximately reconstructed, while the anomalies are reconstructed with significant reconstruction errors; thus, the reconstruction errors indicate the anomalous degree. The experimental results obtained on several public datasets and two large unmanned aerial vehicle (UAV)-borne datasets confirm the merit and viability of the proposed method.

Hyperspectral videos can provide the spatial, spectral, and motion information of targets, which makes it possible to track camouflaged targets that are similar to the background. However, hyperspectral object tracking is a challenging task, due to the huge hyperspectral video data dimension and the "data hungry" problem for the model training. Insufficient training data can seriously interfere with the accuracy and generalization of the tracking models. In this paper, a dual deep Siamese network framework for hyperspectral object tracking (SiamHYPER) is proposed for learning a hyperspectral tracker from a pretrained RGB tracker in the case of the "data hungry" problem. Specifically, in addition to a pretrained RGB-based Siamese tracker, a hyperspectral target-aware module is designed to mine the spectral information during the target prediction, and a spatial-spectral cross-attention module is introduced to further fuse the deep spatial and spectral features extracted from the RGB tracker and the hyperspectral target-aware module. Benefiting from the guidance training of the RGB tracker, a robust hyperspectral object tracker can be trained effectively with only a small number of hyperspectral video samples, to overcome the "data hungry" problem. In the experiments conducted in this study, the SiamHYPER framework was verified using SiamBAN and SiamRPN++, with 13 000 frames of hyperspectral videos for training, and achieved the best performance on the publicly available hyperspectral dataset released as part of the WHISPERS Hyperspectral Object Tracking Challenge. The area under the curve (AUC) of SiamHYPER was increased by nearly 8.9% and 7.2%, respectively, when compared with the current state-of-the-art RGB-based and hyperspectral trackers. In addition, the processing speed of SiamHYPER was 19 FPS, which is much higher than that of the current state-of-the-art hyperspectral trackers. The source code is available at zhenliuzhenqi/HOT: Hyperspectral object tracking (github.com).

Large scale crop mapping is an important task in agricultural resource monitoring. To obtain a distribution map of crops, traditional methods usually require the well-designed manufacture features and the ground-truth labels of all land-cover types for training a multi-class classifier. However, the redundant labeling for each land-cover type is time-consuming and labor-intensive, and the feature design for different remote sensing data is complex and limited to human prior knowledge. In this paper, a deep one-class crop extraction framework is proposed to solve the problems mentioned above. Specifically, it uses the deep one-class crop extraction module to extract the feature automatically for any remote sensing imagery and the one-class crop extraction loss to address the lack of negative class in the deep one-class classification model. In addition, the proposed framework can be applied to multi-modal remote sensing data, i.e. hyperspectral, multispectral, and SAR images, which is verified in the experiments and the proposed framework can achieve the highest accuracy on each multi-modal data.

Subpixel mapping (SPM) is an effective way to solve the mixed pixel problem, which is a ubiquitous phenomenon in remotely sensed imagery, by characterizing subpixel distribution within the mixed pixels. In fact, the majority of the classical and state-of-the-art SPM algorithms can be viewed as a convolution process, but these methods rely heavily on fixed and handcrafted kernels that are insufficient in characterizing a geographically realistic distribution image. In addition, the traditional SPM approach is based on the prerequisite of abundance images derived from spectral unmixing (SU), during which process uncertainty inherently exists and is propagated to the SPM. In this article, a kernel-learnable convolutional neural network (CNN) framework for subpixel mapping (SPMCNN-F) is proposed. In SPMCNN-F, the kernel is learnable during the training stage based on the given training sample pairs of low- and high-resolution patches for learning a geographically realistic prior, instead of fixed priors. The end-to-end mapping structure enables direct subpixel information extraction from the original coarse image, avoiding the uncertainty propagation from the SU. In the experiments undertaken in this study, two state-of-the-art super-resolution networks were selected as application demonstrations of the proposed SPMCNN-F method. In experiment part, three hyperspectral image data sets were adopted, two in a synthetic coarse image approach and one in a real coarse image approach, for the validation. Additionally, a new data set with pairs of Moderate-resolution Imaging Spectroradiometer (MODIS) and Landsat images were adopted in a real coarse image approach, for further validation of SPMCNN-F in large-scale area. The restored fine distribution images obtained in all the experiments showed a perceptually better reconstruction quality, both qualitatively and quantitatively, confirming the superiority of the proposed SPM framework.

Determining the optimal number of endmember sources, which is also called “virtual dimensionality” (VD), is a priority for hyperspectral unmixing (HU). Although the VD estimation directly affects the HU results, it is usually solved independently of the HU process. In this article, a saliency-based autonomous endmember detection (SAED) algorithm is proposed to jointly estimate the VD in the process of endmember extraction (EE). In SAED, we first demonstrate that the abundance anomaly (AA) value is an important feature of undetected endmembers since pure pixels have larger AA values than “distractors” (i.e., mixed pixels and pure pixels of detected endmembers). Then, motivated by the fact that endmembers usually gather in certain local regions (superpixels) in the scene, due to spatial correlation, a superpixel prior is introduced in SAED to distinguish endmembers from noise. Specifically, the undetected endmembers are defined as visual stimuli in the AA subspace, the EE is formulated as a salient region detection problem, and the VD is automatically determined when there are no salient objects in the AA subspace. Since the spatial-contextual information of the endmembers is exploited during the saliency analysis, the proposed method is more robust than the spectral-only methods, which was verified using both real and synthetic hyperspectral images.

The Deep Learning (DL) methods can automatically extract features without artificial prior, provides an effective solution for multi-temporal crop classification. Convolutional Neural Networks (CNNs) have superb spatial-spectral feature extraction capabilities, but often lack consideration of the temporal relationship of multi-temporal images, while Recurrent Neural Networks (RNNs) can better learn the sequential variation pattern. This paper designed a deep spatial-temporal-spectral feature learning network (CropNet) combining the advantages of a deep spatial-spectral feature learning module and a deep temporal-spectral feature learning module for better feature extraction in crop classification from time-series remote sensing images. From the results, the proposed method has better crop classification effects from time-series multispectral images in our experimental areas compared with some common traditional machine learning approaches and common DL methods.

Anomaly detection is an active area of research in hyperspectral information processing. Recently, low-rank representation has been applied in hyperspectral anomaly detection. However, the existing low-rank-based methods either involve a complicated dictionary construction process or the anomaly-background separation which is not sufficient. In this article, to solve these problems, a novel hyperspectral anomaly detection method based on a locally enhanced low-rank prior (LELRP-AD) is proposed. This article is inspired by the observation that, in local homogeneous regions, the background signals hold an enhanced low-rank property while the anomalies exhibit spatial sparsity. Based on this observation, the background pixels can be low-rank reconstructed by a set of basis background signals, whereas anomalies can be represented as sparse residuals. First, image segmentation is performed to enhance the homogeneity of the background, in which a Potts-based image segmentation algorithm is adopted with postprocessing, thus avoiding the need for a complicated spectral dictionary for the representation of the background. Furthermore, the original hyperspectral data matrix is augmented with extracted background endmembers for the low-rank and sparse matrix decomposition, to further achieve anomaly-background separation. The experimental results obtained on four real hyperspectral data sets demonstrate the merit and viability of the proposed method compared with the current state-of-the-art methods.

Deep learning (DL)-based hyperspectral classification primarily use "spatial patching" as preprocessing for incorporating local spatial information. This operation can help to promote classification accuracy but faces the following problems. First, it is difficult to determine the optimal size of spatial patches for different hyperspectral images (HSIs). Second, this operation only exploits spatial features locally but not globally. In this paper, we propose a novel spectral patching network (SPNet) with an end-to-end deep learning architecture for HSI classification. SPNet uses "spectral patching" and Atrous Spatial Pyramid Pooling (ASPP) module to fully preserve the local and global spatial contextual information of original HSIs. The experimental results with UAV-borne hyperspectral dataset demonstrate that the SPNet achieved state-of-the-art accuracy and visualization performance in.

This paper focuses on the blind unmixing technique for analyzing hyperspectral images (HSIs). A joint deconvolution and blind hyperspectral unmixing (DBHU) algorithm is proposed, which is aimed at eliminating the impact of the adjacency effect (AE) on unmixing. In remote sensing imagery, the AE occurs in the presence of atmospheric scattering over a heterogeneous surface. The AE leads to blurring and additional mixing of HSIs and makes it difficult to estimate endmembers and abundances accurately. In this paper, we first model the blurred HSIs by the use of a bilinear mixing model (BMM), where a blurring kernel is used to model the mixing caused by the AE. Based on the BMM, the DBHU problem is formulated as a constrained and biconvex optimization problem. Specifically, the minimum-volume simplex (MVS) is incorporated to deal with the additional mixing caused by the AE, and 3-D total variation (TV) priors are adopted to model the spectral-spatial correlation of the data. In DBHU, the biconvex problem is efficiently solved by a nonstandard application of the alternating direction method of multipliers (ADMM) algorithm, where a block coordinate descent scheme is applied by splitting the original problem into two saddle point subproblems, and then minimizing the subproblems alternately via the ADMM until convergence. The experimental results obtained with both simulated and real data confirm the viability of the proposed algorithm, and DBHU works well, even where both blurring and noise are present in the scene.

Blind hyperspectral unmixing (HU) technique aims at identifying pure materials in a hyperspectral image (HSI), called endmembers, and quantifying the corresponding proportions, called abundances, with little prior knowledge. In this paper, the degradation mechanism during data collection - adjacent effect (AE), is considered in the process of blind HU. Since the AE leads to blurring (the loss of sharpness, contrast and apparent resolution) in scene, it blocks the quantitative analysis of HSI in sub-pixel level and makes the estimated endmembers and abundances inaccurate. To solve this problem, a bilinear mixing model is developed to simulate the AE, and a novel algorithm, termed joint deconvolution and blind HU (DBHU) is proposed. In DBHU, the bi-convex optimization problem is efficiently solved by a nonstandard application of the alternating direction method of multipliers (ADMM) algorithm, where a block coordinate descent scheme is applied by splitting the original problem into two saddle-point subproblems and then minimizing the subproblems alternatively via ADMM until convergence. The experimental results on both simulated and real HSI illustrate the viability of the proposed algorithm.

This paper focuses on the endmember extraction (EE) technique for analyzing hyperspectral images. We first prove that the reconstruction errors (REs) and abundance anomalies (AAs) (abundances that fail to satisfy the abundance constraints) are effective in extracting undetected endmembers. Then, according to the spatial continuity of the endmember objects and differing from noise or outliers with a sparse distribution, the endmembers are assumed to be located at some salient areas in the RE and AA maps. A novel EE algorithm termed saliency-based endmember detection (SED) is proposed, where the visual saliency model is introduced to explore and analyze the spatial information that is contained in the AA and RE maps. Specifically, the AA and RE maps are regarded as the visual inputs, whereas the endmembers are treated as the visual stimuli. In SED, we assume that the pure pixel assumption holds. Based on the characteristics of the human visual system, the proposed method can not only extract endmembers in homogenous areas, but it can also highlight the small targets whose abundances may be spatially varied. In addition, since the spatial information is exploited in the reconstruction, the capability of the endmembers to represent the hyperspectral scene is automatically considered in the process of EE, and the detected endmembers are both accurate and reliable. The experimental results obtained on both simulated and real hyperspectral data confirm the merits and viability of the proposed algorithm.

This paper focuses on the spectral unmixing technique for analyzing hyperspectral image (HSI). In this paper, we first prove that the reconstruction errors and the abundance anomalies (AAs, abundances that are negative or greater than one) are effective in measuring the purity of pixels. Then, due to the continuity of the objects in the space, the endmembers are assumed to be located at some noticeable areas in residual and AA maps. A saliency-based endmember detection (SED) algorithm which aims at iteratively extracting endmembers from the residual and AA maps is proposed, where the visual attention mechanism is developed to understand and analyze the spatial pattern of endmembers. In addition, when searching for new endmembers, the spectral properties are also utilized to promote the robustness of the proposed method. The experimental results on both simulated data and real hyperspectral data illustrate the merits and viability of the proposed algorithm.

In recent years, the science of hyperspectral remote sensing has huge development in virtue of the integration of low-cost lightweight hyperspectral sensors and unmanned aerial vehicles (UAVs). As an alternative of manned aircraft, UAV has some unique advantages enabling the researchers acquire the hyperspectral images of their interest area flexibly and promptly. This review focuses on the recent developments of UAV borne hyperspectral remote sensing system, and gives an overview of the corresponding platforms, sensors, data acquisition, processing and current applications. Future challenges and research directions for UAV borne hyperspectral data are also addressed.

In recent years, blind source separation (BSS) has received much attention in the hyperspectral unmixing field due to the fact that it allows the simultaneous estimation of both endmembers and fractional abundances. Although great performances can be obtained by the BSS-based unmixing methods, the decomposition results are still unstable and sensitive to noise. Motivated by the first law of geography, some recent studies have revealed that spatial information can lead to an improvement in the decomposition stability. In this paper, the group-structured prior information of hyperspectral images is incorporated into the nonnegative matrix factorization optimization, where the data are organized into spatial groups. Pixels within a local spatial group are expected to share the same sparse structure in the low-rank matrix (abundance). To fully exploit the group structure, image segmentation is introduced to generate the spatial groups. Instead of a predefined group with a regular shape (e.g., a cross or a square window), the spatial groups are adaptively represented by superpixels. Moreover, the spatial group structure and sparsity of the abundance are integrated as a modified mixed-norm regularization to exploit the shared sparse pattern, and to avoid the loss of spatial details within a spatial group. The experimental results obtained with both simulated and real hyperspectral data confirm the high efficiency and precision of the proposed algorithm.

The leaked heat pipeline can be detected as temperature anomalies from the air-borne thermal image. Existing methods of thermal anomaly detection are prone to generate a large quantity of false alarms. Although supervised classification can reduce the false positive rate, it requires years of accumulated training data. In this paper, we use human visual system to improve the detection capabilities of thermal anomaly in district heating system. Leakage candidates are selected from the saliency map created by the thermal image, then buffer analysis with pipeline GIS layer is used to reject false detections. Experimental results show that the proposed method has better performance in detection rate when prior knowledge is scarce, and it is more adaptable to actual circumstances.