Knowledge distillation provides an effective paradigm for developing lightweight remote sensing object detectors. However, existing methods predominantly concentrate on design of foreground feature masks while neglecting the suppression of small-scale targets by dominant large-scale counterparts in cluttered backgrounds during the distillation, which severely degrades detection robustness. To address this, we propose scale-conscious knowledge distillation. Multi-scale feature distillation in this framework decouples the coupled feature in conventional methods into hierarchical multi-scale representation spaces, enabling the student model to capture cross-granularity features from micro-details to macro-structures through differentiated receptive fields. Scale-adaptive output distillation innovatively introduces a dynamic weighting mechanism based on the normalized object area, effectively alleviating the gradient vanishing issue for small objects. Experimental results show that our method consistently improves the performance of student models with different architectures on both the DOTA and DIOR datasets. Source codes are available at https://github.com/RQ-W/SCD.git.

Weakly-supervised object detection (WSOD) learns detectors with only image-level classification annotations. Without precise instance-level labels, most previous WSOD methods in remote sensing images (RSIs) select the highest-scoring proposals as the final detection results, which are confronted by two major challenges: (1) instances with small scale or rare poses are easily neglected; (2) optimizing network by the top-scoring region inevitably overlooks many valuable candidate proposals. To mitigate the above-mentioned challenges, we propose a data-driven bidirectional spatial-adaptive network (BSANet). It contains a forward-reverse spatial dropout (FRSD) module to reduce instance ambiguity induced from extreme scales and poses, as well as crowded scene, and to better excavate the entire instances. From attention learning perspective, the proposed FRSD is conceptually similar to a data-driven hard attention mechanism, which adaptively samples and reconstructs the spatially related regions for mining more latent feature responses. Meanwhile, our FRSD effectively alleviates the inherent problem that non-parametric hard attention learning fashion cannot adapt to different datasets. In addition, we build a soft attention branch to simultaneously model soft pixel-level and hard region-level attention information for exploring the complementary benefit between soft and hard attention learning. We evaluate our BSANet on the challenging NWPU VHR-10.v2 and DIOR datasets. Experimental results demonstrate that our method sets a new state-of-the-art.

The integration of Energy Harvesting (EH) technology into Mobile Edge Computing (MEC) presents a promising solution to the energy limitations faced by end devices (EDs) when performing computation-intensive tasks, ultimately enhancing the EDs’ sustainability. However, the intermittent and unpredictable nature of harvested energy introduces significant uncertainty in energy availability, complicating dynamic task execution and resource management. In EH-MEC systems, managing task scheduling and offloading dynamically is crucial for optimizing application latency while ensuring long-term battery energy stability and task reliability. Existing approaches inadequately address the time-coupling between task decisions caused by long-term battery energy stability constraints, and often neglect task reliability considerations. To overcome these limitations, we propose decomposing the original problem into 1) a decoupling problem that transforms the optimization with long-term battery energy constraints into a series of deterministic optimizations within individual time slots, 2) a task scheduling problem that determines task-to-ES assignment and computing resource allocation of ESs to offloaded tasks, and 3) a task offloading problem that determines the optimal offloading decision to achieve minimal latency while meeting energy stability and server reliability constraints. To tackle these subproblems, we design a Lyapunov-based optimization method, a reliabilityaware task scheduling algorithm, and an efficient heuristic-based task offloading algorithm. Extensive simulations and experiments based on empirical data from a physical MEC testbed validate the efficacy of our approach.

Hyperspectral image (HSI) restoration tasks including super-resolution, denoising, and inpainting, present significant challenges due to intrinsic spectral-spatial coupling and limited training data availability. Recent advances in RGB image restoration demonstrate that models pretrained on large-scale datasets acquire exceptional generalization capabilities, suggesting potential cross-modal knowledge transfer solutions for HSI recovery. However, existing approaches exhibit two critical limitations: 1) prohibitive computational costs from mandatory fine-tuning procedures, and 2) inadequate cross-modal adaptation causing spectral distortions. To address these challenges, we propose a Two-Stage Cross-Modal Decoupling Network (CMDN) achieves spectral-faithful HSI restoration without fine-tuning the pretrained RGB prior; instead, we perform unsupervised test-time learning only on a lightweight spectral rectifier for sample-specific spectral calibration. Our methodology introduces two fundamental innovations: First, we develop a theoretically grounded framework using Singular Value Decomposition (SVD) to decouple HSIs into orthogonal spatial coefficients and spectral bases. This decomposition enables strategic reconfiguration of spatial coefficients into pseudo-RGB formats through band reorganization, facilitating direct deployment of frozen RGB-pretrained models for spatial textures recovery while preserving spectral integrity. Second, we propose a Physics Motivated Spectral Rectifier (PMSR) that dynamically adjusts spectral reconstruction weights using spatial gradient priors, correcting spectral deviations through physics-consistent optimization rather than explicit error modeling, thereby achieving superior spectral fidelity. Comprehensive experiments confirm our method's superiority in both spatial reconstruction accuracy and spectral consistency over state-of-the-art techniques. Code is available at: https://github.com/QYo-Liu/CMDN.

Hyperspectral and multispectral image fusion (HMF) enhances spatial-spectral quality by fusing low-resolution hyperspectral images (LR-HSI) with high-resolution multispectral images (HR-MSI). Although recent fusion methods have shown promise in preserving the multi-mode structure of high-dimensional data, existing fusion methods still face some challenges. For tensor-based approaches, conventional mode-wise decomposition, such as order-3 CP or Tucker decomposition, may disrupt intrinsic spatial consistency. Furthermore, although deep learning exhibits powerful feature representation ability, existing deep fusion methods either rely on 'data-driven' deep fusion networks remain insufficiently interpretability with large training data. To address these issues, a novel Self-Expressive High-Order Tensor Unrolling Network (SHOTUN) is proposed for unsupervised HSI-MSI fusion. Within the sparse core tensor decomposition framework, we introduce the intrinsic self-expressive relationships among overlapping image patches as a form of high-order mode representation to preserve spatial structure of the fusion model. During optimization, we adopt an alternative optimizing strategy and design dedicated modules for each sub-problem, yielding an interpretable end-to-end training pipeline. Furthermore, to improve generalization across different sensors, we introduce a pre-training strategy into the unsupervised training for the more accurate estimation of unknown degraded parameters. Extensive experimental results on simulated and real datasets demonstrate the effectiveness of our proposed method. The source code is publicly available at https://github.com/Shawn-H-Wang/SHOTUN.

Hyperspectral images (HSIs) and multispectral images (MSIs) fusion is a hot topic in the remote sensing society. A high-resolution HSI (HR-HSI) can be obtained by fusing a low-resolution HSI (LR-HSI) and a high-resolution MSI (HR-MSI) or RGB image. However, most deep learning-based methods require a large amount of HR-HSIs for supervised training, which is very rare in practice. In this paper, we propose a coupled diffusion posterior sampling (CDPS) method for HSI and MSI fusion in which the HR-HSIs are no longer required in the training process. Because the LR-HSI contains the spectral information and HR-MSI contains the spatial information of the captured scene, we design an unsupervised strategy that learns the required diffusion priors directly and solely from the input test image pair (the LR-HSI and HR-MSI themselves). Then, a coupled diffusion posterior sampling method is proposed to introduce the two priors in the diffusion posterior sampling which leverages the observed LR-HSI and HR-MSI as fidelity terms. Experimental results demonstrate that the proposed method outperforms other state-of-the-art unsupervised HSI and MSI fusion methods. Additionally, this method utilizes smaller networks that are simpler and easier to train without other data.

Hyperspectral and multispectral image (HSI-MSI) fusion involves combining a low-resolution hyperspectral image (LR-HSI) with a high-resolution multispectral image (HR-MSI) to generate a high-resolution hyperspectral image (HR-HSI). Most deep learning-based methods for HSIMSI fusion rely on large amounts of hyperspectral data for supervised training, which is often scarce in practical applications. In this paper, we propose a self-learning Adaptive Residual Guided Subspace Diffusion Model (ARGS-Diff), which only utilizes the observed images without any extra training data. Specifically, as the LR-HSI contains spectral information and the HR-MSI contains spatial information, we design two lightweight spectral and spatial diffusion models to separately learn the spectral and spatial distributions from them. Then, we use these two models to reconstruct HR-HSI from two low-dimensional components, i.e, the spectral basis and the reduced coefficient, during the reverse diffusion process. Furthermore, we introduce an Adaptive Residual Guided Module (ARGM), which refines the two components through a residual guided function at each sampling step, thereby stabilizing the sampling process. Extensive experimental results demonstrate that ARGS-Diff outperforms existing state-of-the-art methods in terms of both performance and computational efficiency in the field of HSI-MSI fusion.

The widespread adoption of Deep Neural Networks (DNNs) across diverse applications has intensified the demand for efficient execution strategies on GPU-CPU heterogeneous devices. While existing device-oriented DNN partitioning methods optimize either energy consumption or execution latency in isolation, they often lack the capability to dynamically adapt partitioning strategies under real-world constraints. To address these limitations, we first formulate the DNN partitioning problem as an optimal execution path search on a directed acyclic graph (DAG), then propose an imitation learning (IL)-based DNN partitioning framework that dynamically allocates DNN layers across GPU-CPU processors to minimize energy consumption while satisfying time constraints. Specifically, we first design aProfiler, which analyzes the layer-wise characteristics such as the execution time and energy consumption of layers in DNNs executed on the GPU-CPU heterogeneous device. Then, we present anOracle Generatorthat generates high-quality partitioning solutions based on the Dijkstra algorithm to form a training datasetOracle.Oraclefrom the proposed generator is input to a proposed Trainer, which iteratively learns from theOracledata to obtain a regression model namedPredictor.Predictorcan predict the optimal partitioning strategy for DNNs in real time. Finally, we design aData Aggregator, which enhancesOracledata through continuous runtime feedback, thereby improving the prediction accuracy of the regression model. We evaluate the proposed IL-based DNN partitioning method on 3 NVIDIA Jetson platforms and 1 Huawei NPU platform. Experimental results demonstrate that compared to 3 benchmarking methods, our method reduces energy consumption by up to 73.15%.

Small target detection in remote sensing images is a significiant reserach focus within the remote sensing domain. Recently, various YOLO algorithms have demonstrated remarkable achievements in the detection of small targets in remote sensing. However, YOLO-based detection algorithms still face challenges in this context, including limited feature expression capacity, difficulties in mitigating aliasing effects and inadequate adaptability to complex-shaped targets. To address these issues, this paper proposes a novel object detection network SOD-YOLOv8n. First, we propose a novel multi-path feature fusion module (MFFM), which enhances feature extraction through diverse dimensional feature processing strategies (global, local, channel, and spatial). It also fuses complementary information across channels via channel shuffling, thereby augmenting feature representation capabilities, and boosting the detection accuracy of small targets in remote sensing. Secondly, we design an anti-aliasing module (AAM) that employs wavelet pooling technology for frequency decomposition to mitigate the aliasing effect generated during model downsampling, thereby better retaining the key high-frequency information of small targets in remote sensing. Finally, we introduce the Shape-IoU loss function, which emphasizes the edge features of the target shape (such as contours, curvature, etc.) by calculating the similarity between the true target and the target shape in the predicted box, so as to better match targets with complex shapes.We conduct extensive experiments on the AI-TOD and USOD remote sensing small target datasets, and the results show that SOD-YOLOv8n outperforms several established state-of-the-art detection models.

The joint classification of hyperspectral image (HSI) and light detection and ranging (LiDAR) data seeks to provide a more comprehensive characterization of target objects. Multi-modal data possess distinct semantic structures in both spectral and spatial dimensions, making efficient feature complementarity and redundancy elimination crucial. To this end, we propose a self-Distillation-based multimodal Feature Alignment Network (DFANet), which employs two branches to capture spectral and spatial similarities respectively, and integrates structural discriminative information from LiDAR at two stages for more effective multimodal data integration. The network comprises three main components: a Feature Alignment Fusion Module (FAFM), an Offset Attention Module (OAM), and a self-distillation mechanism. Specifically, the FAFM guides feature alignment through channel-assimilative mapping of multimodal data. The OAM addresses boundary patch classification challenges by learning offset weights of reference points. The self-distillation mechanism filters out irrelevant information during feature alignment by enhancing the coordination between high-level and low-level features. Adequate experiments indicate that our method achieves better results compared to the most recent hyperspectral classification methods on three public datasets.

Remote sensing (RS) images have rich ground information, diverse object types, and large-scale differences, and these characteristics make difficulties in achieving precise segmentation. Recently, the state-space model improved by Mamba offers global modeling capability while maintaining linear computational complexity. However, it still faces issues with insufficient extraction of global information specific to the spatial and channel dimensions, which is crucial for achieving accurate segmentation, along with lacking sensitivity to local details. Based on this, we propose a hybrid Mamba-attention fusion network (HMAFNet) for RS image semantic segmentation, based on the encoder-decoder architecture. Specifically, the encoder incorporates the spatial-channel Mamba (SCMamba) module, which uses the Mamba to efficiently capture global feature representations across both spatial and channel dimensions. Meanwhile, local information essential for the encoding phase is supplemented by a parallel convolutional branch. In the decoding phase, we propose the information-guided cross fusion (IGCF) module, which generates corresponding features via convolution-based and Mamba-based information-guided branches. The cross-attention mechanism facilitates the interaction and fusion between the features, thereby preserving elaborated details and further eliminating semantic differences. Extensive comparison experiments and ablation experiments on both the Vaihingen and Potsdam datasets show that our proposed HAMFNet can achieve better segmentation results.

With the advancement of object detection technology, few-shot object detection (FSOD) has become a research hotspot. Existing methods face two major challenges: base models have limited generalization to unseen categories, especially with limited few-shot data, where the shared feature representation fails to meet the distinct needs of classification and regression tasks; FSOD is susceptible to overfitting during training. To address these issues, this paper proposes a Multi-Task Decoupled Method (MTDM), which enhances the model’s generalization to new categories by separating the feature extraction processes for different tasks. Additionally, a dynamic adjustment strategy is adopted, which adaptively modifies the IOU threshold and loss function parameters based on variations in the training data, reducing the risk of overfitting and maximizing the utilization of limited data resources. Experimental results show that the proposed hybrid model performs well on multiple few-shot datasets, effectively overcoming the challenges posed by limited annotated data.

This letter focuses on integrated CPU-GPU edge systems with renewable energy sources and studies the resource management problem to minimize the energy consumption of real-time tasks while ensuring temperature and reliability constraints. We propose an imitation learning (IL)-based resource management scheme, ILRM, implemented in two phases: 1) offline Oracle generation and 2) online IL. In the offline phase, we design a fast-converging heuristic to generate near-optimal solutions (i.e., Oracles) for training an online prediction model. In the online phase, we realize IL using the trained model that predicts the resource configuration policies for the incoming task sets to be scheduled. A data aggregation method is also developed to enhance the robustness of the prediction model. We validate ILRM through extensive experiments on both simulated and real integrated CPU-GPU edge platforms.

Hyperspectral unmixing decomposes hyperspectral images (HSIs) into pure spectral signatures (endmembers) and their proportions (abundances). Existing deep learning methods for this task predominantly focus on either linear or nonlinear relationships, often struggling to achieve an optimal equilibrium between the robustness of linear estimations and the precision of nonlinear models. Furthermore, these methods often neglect the integration of spatial information and the interpretability of the results. This article proposes a biobjective model-driven autoencoder network for blind hyperspectral unmixing that simultaneously addresses both linear and nonlinear relationships. By combining linear and nonlinear kernel models within a model-driven deep learning framework, we aim to enhance the interpretability of the results. To effectively capture spatial information, we introduce an Adaptive Composite Kernel that integrates traditional and spatial-spectral kernels, using a dynamic convolution-like mechanism to optimize their respective weights. Our method establishes linear and nonlinear reconstruction losses, enabling the simultaneous estimation of endmembers from both types of relationships, thus capturing the complex and high-dimensional data structures inherent in HSIs. Experiments on real and synthetic datasets demonstrate the superiority of the proposed method.

In the field of salient object detection (SOD), optical remote sensing images (ORSI) differ significantly from natural sensing images (NSI). Existing research in ORSI-based SOD is constrained by the limitations of convolutional neural networks (CNNs) in feature extraction and by the underutilization of feature information in Transformer-based approaches. To address these challenges, this paper presents a Transformer-based Local-Global Information Perception Network (LGIPNet) for ORSI, which enhances encoder-generated features at multiple levels to highlight salient targets through three specialized feature enhancement modules. The Edge Adaptive Enhancement Module (EAEM) focuses on extracting local edge features to guide precise edge generation. The Multi-scale Grouped Weighted Attention Module (MGWAM) extracts local information from low-level features, scales features, and uses multi-scale channel and learnable weighted spatial attention to locate salient targets. For high-level features, the Dual-Domain Attention Module (DDAM) integrates a Channel Enhancement Attention Block (CEA) and an Adaptive Spatial Attention Block (ASA) to refine both local and global information. Specifically, the EAEM sharpens the edges of salient objects, thereby ensuring the precision of boundary detection. The MGWAM, on the other hand, enriches the feature representation across multiple scales, enhancing the network’s capability to encapsulate both fine-grained details and broader contextual information. The DDAM further strengthens the balance between local and global information, preserving feature integrity across levels. Finally, multi-scale features are cascaded to produce the final saliency map. This holistic strategy empowers LGIPNet to accurately identify and emphasize salient objects in ORSI. Experiments on three datasets demonstrate that LGIPNet outperforms existing state-of-the-art methods, establishing its effectiveness and robustness in ORSI-based SOD. The source code is available at https://github.com/sCauliflower/LGIPNet.git.

Knowledge distillation, which intends to transfer the expertise from a complex teacher model to a concise student model, has achieved impressive success in object detection. However, many existing distillation methods are designed for object detection tasks on natural images and perform poorly on more challenging remote sensing images. In this study, we first attribute the deficiencies of knowledge distillation in remote sensing object detection to two core reasons: 1) lack of relation distillation among different instances and pixels and 2) significant differences in feature magnitude between the teacher-student pair. Then, we propose a dual prediction-guided knowledge distillation framework, which includes relation distillation and output distillation to address two issues, respectively. Prediction-guided relation distillation (PGRD) is proposed to capture the knowledge of global relation at both instance-wise and pixel-wise, then allowing students to better understand and depict the features distribution across different categories. Prediction-guided output distillation (PGOD) is proposed to mitigate the impact of feature magnitude inconsistencies on distillation with classification and location knowledge, then allowing students to directly capture task-relevant information. Finally, experimental results have indicated the consistent effectiveness of our method across anchor-based two-stage, one-stage, and anchor-free detectors with 11 comparison knowledge distillation methods on three remote sensing detection datasets. Source codes are available athttps://github.com/RQ-W/DPGD.git.

Hyperspectral target detection (HTD) can provide detailed information about the objects and materials within a scene and holds significant importance in remote sensing image analysis. Traditional HTD methods often suffer from low detection accuracy when applied to hyperspectral images (HSIs) with low spatial resolution, where the target only occupies a few pixels. This can be addressed by exploiting the higher spatial resolution of multispectral images (MSIs). In addition, many cloud-based HTD methods, which rely on the distributed processing capability of cloud computing to cope with large-scale datasets, may result in long transmission delays that cannot meet real-time requirements. This article suggests a cloud-edge collaborative HTD approach based on the fusion of remotely sensed HSIs and MSIs. We first introduce an HTD algorithm that employs low-rank matrix decomposition and hierarchical constraint energy minimization (hCEM) to fuse a low-resolution HSI (LR-HIS) and a high-resolution MSI (HR-MSI). Aiming at a continuous shooting scenario, we further present a cloud-edge implementation of the HTD algorithm through the collaboration of a cloud cluster and edge servers deployed close to data acquisition devices. The overall processing flow of remotely sensed data fusion in the cloud-edge environment is formulated as a flowshop scheduling-like optimization problem. We develop a co-optimization scheduling algorithm to explore the best resource allocation solutions to the formulated problem. Experimental results on both general-purpose and real-world datasets show that the newly proposed HTD algorithm leads to significant improvements in detection accuracy over traditional methods, and the cloud-edge collaborative approach further enhances computational efficiency.

Hyperspectral image (HSI) target detection is an advanced technology within the remote sensing society, widely applied in both civilian and military domains. However, subpixel target detection poses a significantly challenging problem in HSI processing. Existing methods for HSI target detection mostly rely on constructing detectors based on linear mixing model (LMM). Nevertheless, due to multiple scattering, LMM fails to capture the nonlinear features and extensive interactions among various materials commonly present in real HSIs, potentially resulting in poor performance. To address this issue, this work proposes a novel method for HTD based on bilinear mixing model (BMM) in HSI. Specifically, we designed a spectral decomposition network to separate the background and target. To further extract and combine spectral and spatial information of the image for more accurate learning of target and background distributions, a deep network model based on a multiscale feature extraction module is proposed. Finally, to enhance the discrimination between targets and backgrounds, a customized CEM loss is formulated by minimizing the response of the background and maximizing the response of the target. Experimental comparisons on six real HSI datasets validate the superiority of the proposed detector.

Low-rank representation is a popularly used method in remote sensing and image processing systems. Existing low-rank representation-based algorithms often neglect the varying sensitivities of different pixels to the dictionary, which may lead to inaccurate detection. Also, the dense iterative computations involved in these algorithms could induce high computation overheads. This article proposes an efficient low-rank representation method for hyperspectral anomaly detection (HAD) on a central processing unit (CPU)-field programmable gate array (FPGA) hybrid computing platform. The proposed method starts with segmenting the hyperspectral image into superpixels based on the selected number of dictionary categories. The image is divided into center and edge parts using a sliding window along the segmentation boundaries. Then, we use a low-rank and sparse representation (LRASR) to process center pixels that are well-adapted to the dictionary, and a weighted low-rank and collaborative representation model to process edge pixels that are less adaptive to the dictionary. The aforementioned two representation models are integrated to obtain the final reconstructed image. Moreover, we propose the Nesterov acceleration method by incorporating an adaptive step size into the linearized alternating direction method with adaptive penalty (LADMAP). At the hardware level, the acceleration method is deployed on the CPU-FPGA platform to speed up the proposed HAD flow. Experimental results demonstrate that the proposed method outperforms most existing methods in detection accuracy and achieves significant improvement in computational efficiency.

Multi-class change detection from hyperspectral image (HSI) leverages the rich spectral information of HSIs to detect and classify subtle changes of interest in an imaged scene. However, challenges arise due to limited samples in small categories, which hinder the accurate differentiation of changes. This study proposes a spatial-spectral feature-enhanced Mamba and SAM-guided hyperspectral multi-class change detection (SFMS) method. To address the challenges, a tri-plane gated Mamba is designed to complement spatial information by utilizing the abundant spectral information in HSIs. Additionally, frequency domain features are combined with state space models, enabling the detection of more accurate semantic and texture changes using integrated information from frequency domains. This approach effectively mitigates the problem of inaccurate detection in small-sample categories. Furthermore, the segment anything model (SAM) is adapted, with the features of change areas being enhanced through prior knowledge obtained from segmentation, thereby improving the multi-class change detection accuracy. The experimental results demonstrate that the proposed SFMS method outperforms state-of-the-art techniques, achieving superior multi-class change detection while overcoming the challenges associated with detecting small-sample categories.

Vision Transformer (ViT), known for capturing non-local features, is an effective tool for hyperspectral image classification (HSIC). However, ViT's multi-head self-attention (MHSA) mechanism often struggles to balance local details and long-range relationships for complex high-dimensional data, leading to a loss in spectral-spatial information representation. To address this issue, we propose a deformable convolution-enhanced hierarchical Transformer with spectral-spatial cluster attention (SClusterFormer) for HSIC. The model incorporates a unique cluster attention mechanism that utilizes spectral angle similarity and Euclidean distance metrics to enhance the representation of fine-grained homogenous local details and improve discrimination of non-local structures in 3-D HSI and 2-D morphological data, respectively. Additionally, a dual-branch multiscale deformable convolution framework augmented with frequency-based spectral attention is designed to capture both the discrepancy patterns in high-frequency and overall trend of the spectral profile in low-frequency. Finally, we utilize a cross-feature pixel-level fusion module for collaborative cross-learning and fusion of the results from the dual-branch framework. Comprehensive experiments conducted on multiple HSIC datasets validate the superiority of our proposed SClusterFormer model, which outperforms existing methods. The source code of SClusterFormer is available at https://github.com/Fang666666/HSIC SClusterFormer.

Most remote sensing datasets are annotated with horizontal bounding boxes (HBBs), which conflicts with mainstream oriented object detection methods that require oriented bounding boxes (OBBs). Horizontal box supervised oriented object detection has emerged as a promising solution, but existing methods suffer from two key limitations. First, they apply image-level geometric transformations for consistency learning, which binds object orientation to the global image and limits the model's ability to learn instance-specific orientation features. Second, they rely on data augmentation for orientation awareness while still using conventional horizontal convolutional neural networks (CNNs) for regression, failing to extract orientation-sensitive features effectively. To address these issues, we propose the Instance-Level Orientation Information Enhanced Detector (ILOEDet), which integrates the Instance-Aware Rotated Convolution Module (IARCM) and an Instance-Level Flip Consistency (IFC) mechanism to improve orientation sensitivity. Specifically, IARCM leverages classification and center-ness scores to select high-quality instances and their predicted angles, guiding a rotated convolution operation to embed instance-level orientation information into the feature maps. Meanwhile, IFC introduces a self-supervised branch that flips individual object instances to decouple their orientation from the image background, enforcing instance-level consistency constraints for more robust orientation learning. Experiments on the DOTA, HRSC2016, and DIOR-R datasets demonstrate the effectiveness of our approach.

Fine-grained object detection (FGOD) fundamentally comprises two primary tasks: object detection and fine-grained classification. In natural scenes, most FGOD methods benefit from higher instance resolution and fewer environmental variation, attributing more commonly associated with the latter task. In this paper, we propose a unified paradigm named Detector with Classifier2 (DC2), which provides a holistic paradigm by explicitly considering the end-to-end integration of object detection and fine-grained classification tasks, rather than prioritizing one aspect. Initially, our detection sub-network is restricted to only determining whether the proposal is a coarse-category and does not delve into the specific sub-categories. Moreover, in order to reduce redundant pixel-level calculation, we propose an instance-level feature enhancement (IFE) module to model the semantic similarities among proposals, which poses great potential for locating more instances in remote sensing images (RSIs). After obtaining the coarse detection predictions, we further construct a classification sub-network, which is built on top of the former branch to determine the specific sub-categories of the aforementioned predictions. Importantly, the detection network is performed on the complete image, while the classification network conducts secondary modeling for the detected regions. These operations can be denoted as the global contextual information and local intrinsic cues extractions for each instance. Therefore, we propose a multi-stream feature aggregation (MSFA) module to integrate global-stream semantic information and local-stream discriminative cues. Our whole DC2 network follows an end-to-end learning fashion, which effectively excavates the internal correlation between detection and fine-grained classification networks. We evaluate the performance of our DC2 network on two benchmarks SAT-MTB and HRSC2016 datasets. Importantly, our method achieves the new state-of-the-art results compared with recent works (approximately 7% mAP gains on SAT-MTB) and improves baseline by a significant margin (43.2% $v.s.~36.7$ %) without any complicated post-processing strategies. Source codes of the proposed methods are available at https://github.com/zhengshangdong/DC2.

Hyperspectral image (HSI) and multispectral image (MSI) fusion aims to generate high spectral and spatial resolution hyperspectral image (HR-HSI) by fusing high-resolution multispectral image (HR-MSI) and low-resolution hyperspectral image (LR-HSI). However, existing fusion methods encounter challenges such as unknown degradation parameters, and incomplete exploitation of the correlation between high-dimensional structures and deep image features. To overcome these issues, in this article, an unsupervised blind fusion method for LR-HSI and HR-MSI based on Tucker decomposition and spatial-spectral manifold learning (DTDNML) is proposed. We design a novel deep Tucker decomposition network that maps LR-HSI and HR-MSI into a consistent feature space, achieving reconstruction through decoders with shared parameters. To better exploit and fuse spatial-spectral features in the data, we design a core tensor fusion network (CTFN) that incorporates a spatial-spectral attention mechanism for aligning and fusing features at different scales. Furthermore, to enhance the capacity to capture global information, a Laplacian-based spatial-spectral manifold constraint is introduced in shared-decoders. Sufficient experiments have validated that this method enhances the accuracy and efficiency of hyperspectral and multispectral fusion on different remote sensing datasets. The source code is available at https://github.com/Shawn-H-Wang/DTDNML.

The detection of oriented object detection in remote sensing images remains a daunting challenge due to their complex backgrounds, various sizes, and especially arbitrary orientations. However, most existing methods only model the structural features of images in the spatial domain, while the horizontal convolution kernels limit the model’s ability to perceive object direction information. Furthermore, frequency features contain rich information about scale, texture and angle, which can be a good complement to the spatial features. Inspired by this, we propose a multiscale spatial-frequency domain network (MSFN) to utilize spatial-frequency information for oriented object detection, which can be integrated into any CNN architectures seamlessly and perform end-toend training easily. Besides, we design a channel alignment feature fusion module (CA-FFM) to address the problem of fusing low-level texture and high-level semantic features with a significant difference in channel dimension. Experimental results on HRSC2016 and SSDD dataset demonstrate the effectiveness of the proposed method.

In recent years, using Convolutional Neural Network (CNN) for feature extraction and classification of hyperspectral image(HSI) has become a classic method. However, CNN suffer from fixed receptive fields in feature extraction, alongside the loss of shallow-level feature maps within the network. To address this issue, we propose a novel HSI classification method based on dual-layer multi-level architecture and selective kernel attention mechanism(DLMLSK). Firstly, we employ a new dual-layer network to expand the network’s receptive field, enabling the extraction of more abstract features. Secondly, a multi-level fusion module is utilized to integrate features extracted at different levels in the network, facilitating the extraction of global information. Lastly,a selective kernel(SK) attention module is introduced to perform fusion-weighted operations on the fused features, thereby enhancing the model’s selectivity towards features. Experimental results on two datasets demonstrate the effectiveness of our proposed method when compared to several standard HSI classification methods.

Despite the wide application of deep learning in hyperspectral classification, variations in data collection conditions can lead to domain shift between the training and testing datasets. Traditional hyperspectral classification methods are adversely affected by these distribution differences, resulting in poor generalization performance on the testing set. To overcome this challenge, we present an optimized unsupervised domain adaptation approach based on causal invariance. Our method assumes a causal relationship to reflect the effects of changes in class information and domain information on samples. Based on this causal relationship, we construct a network to separate class-related and domain-related features. To further reduce the negative transfer caused by distribution differences, our model introduces intra-class feature consistency. As a result, our method improves the performance of the model on the target domain. Experimental results on two public hyperspectral datasets demonstrate the superior effectiveness of our method.

High-resolution remote sensing object detection is a research field with significant application value and challenges. However, existing methods cannot directly predict or produce a large number of redundant bounding boxes. They perform poorly in the face of issues such as multi-scale, dense small objects, and complex backgrounds in high-resolution remote sensing images. Therefore, a real-time end-to-end high-resolution remote sensing object detection method based on RT-DETR (CDE-DETR) is proposed. Through introducing cascaded group attention, we propose CGA-IFI for intra-scale feature interaction. The DRB-CFFM is designed with a dilated reparam block to facilitate cross-scale feature interaction. Furthermore, we enhance the bounding box regression loss function with EIoU. Experimental results demonstrate that the accuracy mAP value of our method is 2.9% higher than the baseline, FPS is increased by 33.8%. The number of parameters is reduced by 9.9%, and FLOPs is reduced by 16.0%. Compared with other methods, the proposed method has obvious accuracy and lightweight advantages.

Tensor decomposition based fusion methods for hyperspectral and multi-spectral images have recently demonstrated decent fusion performance in the context of remote sensing applications. Considering the huge volume of today’s remote sensing datasets and the involved heavy computation loads, traditional single-machine fusion methods can no longer serve the purpose due to limited storage, communication, and computation capacities. To this end, we propose a distributed and parallel method for Tucker decomposition model based on Apache Spark. Our primary focus is to (i) enhance the generation and update methods for modal matrices to eliminate the bottleneck of insufficient memory; (ii) redesign the initialization and updating procedures of core tensors to adapt to distributed computing and enhance computational efficiency; and (iii) implement group updates to accelerate the overall fusion flow while maintaining fusion accuracy. Experimental results demonstrate that our distributed and parallel fusion method achieves significant speedups over the original model with decent fusion performance.

The diffusion model has recently exhibited remarkable efficacy in image synthesis. Existing diffusion methods rely extensively on abundant training images as data priors, posing challenges for hyperspectral image(HSI) restoration with limited data availability. To tackle this problem, we introduce a novel diffusion approach for HSI restoration, which draws upon the theoretical foundation of denoising diffusion null-space model and works by inferring the parameters of the Spatial-Spectral network during the reverse diffusion process, requiring only a degraded image as input. Through experiments, we showcase the efficacy of this approach in producing impressive results in the realm of hyperspectral image restoration.

When designing task scheduling algorithms in mobile edge computing (MEC), the mobile device (MD)’s mobility becomes an important concern, since the change in MD’s location would affect the data transmission rate, leading to fluctuations in task transmission duration and completion time. In this article, we study a mobility-aware task off-and-downloading scheduling problem in MEC, considering both the communication delay and energy consumption caused by the data offloading and the result downloading. We first formulate a mathematical optimization model of the studied problem and prove its NP-hardness. To explore high-quality task scheduling decisions, we propose a swarm intelligence algorithm-based evolutionary computation (START) framework. The main technical innovations of START include a solution representation of off-and-downloading sequence, an exponential probability model-based mapping operator, and a task dispatching heuristic. Specifically, the solution representation makes START applicable to a wide range of swarm intelligence algorithms. The mapping operator establishes the link between individual space and solution space, in which the critical parameter is determined by a rigorous theoretical analysis. The task dispatching strategy is the only component of the START framework that is relevant to the particular problem, providing the extensibility of applying START to solving other problems. In experiments, we create a real-world MD trajectory dataset MDT-NJUST, and integrate several representative swarm intelligence algorithms to justify the performance of START in solving the scheduling problem. Experimental results also verify the conclusion drawn from the theoretical analysis on critical parameter determination.

Change detection is a concerned task in the interpretation of remote sensing images. The mining of the relationship on change features is usually implicit in the deep learning networks that contain single-branch or two-branch encoders. However, due to the lack of artificial prior design for the relationship on change features, these networks cannot learn enough semantic information on change features and lead to the poor performance. So, we propose a new network architecture explicit change-relation network (ECRNet) for the explicit mining of change-relation features. In our study of the literature, our suggestion is that the change features for change detection should be divided into prechanged image features, postchanged image features, and change-relation features. In order to fully mining these three kinds of change features, we propose the triple branch network combining the transformer and convolutional neural network (CNN) to extract and fuse these change features from two perspectives of global information and local information, respectively. In addition, we design the continuous change-relation (CCR) branch to further obtain the continuous and detailed change-relation features to improve the change discrimination capability of the model. The experimental results show that our network performs better than those of the existing advanced networks by the F1 score improvements of 0.66/0.37/0.70/1.09 on the very high-resolution (VHR) remote sensing datasets of the LEVIR-CD/SVCD/WHU-CD/SYSU-CD. Our source code is available athttps://github.com/DalongZ/ECRNet.

In this letter, we propose a patchwise temporal-spatial feature aggregation (PTFA) network for object detection in satellite video. First, the feature extractor processes the key frame (KF) along with its support frames to ensure comprehensive spatial coverage of potential objects. Subsequently, we model the semantic similarities among instance-level proposals to exploring robust interaction between temporally adjacent support frames and KF. Furthermore, due to the extremely small size of objects in satellite video, we crop the input frames to different patches by the fixed criterion. Then, the temporal-spatial feature aggregation (TSFA) operations are performed on instance-level RoI features, which attains more nuanced and comprehensive descriptors from the explicit high-resolution temporal-spatial features. The patch features are reconstructed to the original one for complementing more valid feature responses. Finally, we compare our PTFA network with many recent works on the SAT-MTB dataset. Extensive experiments demonstrate that our method achieves the state-of-the-art performance than various static image and video object detection (VID) approaches.

Typical high-level vision tasks in hyperspectral image (HSI) processing, such as target detection, often suffer from insufficient information inherent in real-world sampled data. Super-resolution, a powerful tool in HSI low-level vision, is expected to enhance the accuracy of detection results by computationally providing the high-resolution HSI with additional information. However, existing solutions for HSI super-resolution and target detection have always been implemented independently. This conventionally adopted paradigm overlooks the interconnectedness between low-level and high-level visions, inevitably introducing additional errors, redundancies, and inefficiencies. To address this challenge, in this study, we put our efforts into exploring the uncharted continent of hyperspectral remote sensing, that is, realizing the mutual guidance and joint optimization of HSI super-resolution and target detection concurrently within a unified framework. Technically, we first construct different spectral bases to span the target and background subspaces of the underlying high-resolution HSI. Then, we look in-depth at the intrinsic properties of the HSI tensor, henceforth jointly optimizing both tasks by innovatively developing a novel low-cubic-rank tensor approximation model with a unique constrained energy minimization loss. While we have developed efficient algorithms to optimize the proposed model, we also put into place a refinement procedure for spectral bases, aimed at further enhancing the spectral fidelity of the fused results and the compact representation of the target subspace. Finally, empirical studies conducted on synthetic and real-world datasets substantiate that compared with state-of-the-art solutions, the proposed method delivers highly competitive and practical performance in terms of both tasks. Source codes are available at https://github.com/CX-He/HySRTD.git.

Hyperspectral image (HSI) super-resolution with an auxiliary multispectral image (MSI) belongs to the class of inverse problems, where prior knowledge is essential for obtaining the target. Various hand-crafted or deep priors have been developed to enforce the desired solutions. Nevertheless, the spectral distribution knowledge is ignored and still not exploited as a prior. To this end, we design a spectral diffusion model (SDM) to capture the spectral distribution of HSIs and thereby exploit it as a prior for the problem of unsupervised HSI super-resolution. Specifically, we first investigate the spectrum generation problem and extend the diffusion model to fit the 1-D spectral data. Then, we transfer the spectral distribution knowledge of the trained SDM by means of keeping its transition information and thereby induce a regularization term in the framework of maximum a posteriori. At last, we integrate the iterative solving and diffusion generation processes together and employ the Adam to solve the final optimization problem by following the reverse spectral generative sequence. Experimental results conducted on both synthetic and real datasets demonstrate the effectiveness of the proposed approach. The code of the proposed approach is available onhttps://github.com/liuofficial/SDP.

Oriented object detection is an important research topic in remote sensing. The detection of oriented objects in remote sensing images remains a daunting challenge due to their complex backgrounds, various sizes, diverse aspect ratios, and especially arbitrary orientations. In recent years, keypoint-based anchor-free object detectors have demonstrated outstanding performance in this field. However, in current anchor-free detectors, object keypoints are primarily generated using the Gaussian kernel function, which assumes a circular form. This representation falls short in accurately conveying an object’s size and orientation. To address the aforementioned issue, this paper proposes a keypoint-based oriented object detector called MRSDet, which innovatively adopts the Tricube kernel, scales and rotates it, to better generate the center keypoint heatmap of the object. Besides, to improve the model’s detection performance on oriented objects and improve its ability to perceive object keypoints and boundary boxes, we also design a large receptive field mask module (LRFM), which is based on large convolution kernel decomposition and semantic segmentation masks. Taking the BBAVectors method as a baseline, we conduct experiments on multiple types of remote sensing datasets such as HRSC2016, UCAS-AOD and SSDD+ to verify the effectiveness and generalizability of the proposed method.

Unsupervised learning, learning data distributions without needing labeled samples, is a particularly promising approach for solving the challenging task of hyperspectral pansharpening. Inspired by the above, we introduce an innovative generative adversarial network framework (named MFT-GAN) that incorporates a transformer network and multiscale interaction technology. Specifically, MFT-GAN is constructed of one generator and two discriminators. The generator is composed of a multiscale feature guidance branch (MFGB) and a feature interaction fusion branch (FIFB). The former aims at extracting spectral and spatial information branches from low-resolution hyperspectral (LRHS) and panchromatic (PAN) images by convolutional sampling operations, respectively. The latter accomplishes the interaction of spectral and spatial information at different scales through multiscale interaction technology. Then, the multiscale interaction results are fused by level-by-level convolutional sampling and pixel summation operations to generate high-resolution hyperspectral (HRHS) images. The discriminators are composed of a spectral transformer discriminator and a spatial transformer discriminator, designed to maintain structural and parametric balance and learn long-range spectral and spatial correlation relationships. In addition, hybrid loss functions are used to complete the adversarial training of the MFT-GAN to improve its performance. The experiments on simulated and real datasets further demonstrated the effectiveness of the proposed MFT-GAN method.

In recent years, deep learning has shown promising results in change detection (CD) of hyperspectral images (HSIs). However, these algorithms often require a large number of labeled samples, which can be costly in practical applications. This article proposes a few-shot HSI CD method based on a 3-D convolution neural network and normalizing flow. To mitigate the issue of limited training samples, we develop a low-parameter baseline CD model utilizing 3-D convolution neural networks to extract spectral-spatial features while preventing overfitting. In addition, the proposed method analyzes the spectral-spatial distribution of hyperspectral variation by normalizing flow and generates hyperspectral tensor samples that approximate the distribution, thus enhancing CD performance. Furthermore, to improve the sample generation speed, we employ soft labels and label smoothing techniques to assign high-quality labels to the generated samples, thereby increasing the number of available samples. The method is evaluated on three datasets, and the experimental results demonstrate the efficacy of the proposed approach for detecting changes with limited training samples. The proposed method offers a promising solution to the challenge of detecting changes with limited training samples in HSIs.

The effective combination of hyperspectral image (HSI) and light detection and ranging (LiDAR) data can be utilized for land cover classification. Recently, deep learning-based classification methods, especially those utilizing Transformer networks, have achieved remarkable success. However, deep learning classification methods for multi-source data still encounter various technical challenges, such as the comprehensive utilization of multi-scale information, the lightweight network design, and the efficient fusion strategies for heterogeneous data. To address these challenges, we propose a novel and efficient deep neural network, namely multi-scale 3D-2D mixed CNN feature extraction and multi-source data lightweight attention-free fusion network (M2FNet) based on CNN and Transformer. Through end-to-end training, this network effectively combines heterogeneous information from multiple sources, leading to improved performance in joint classification. Specifically, M2FNet employs a multi-scale 3D-2D mixed CNN design to extract both the spatial-spectral features of HSI and the depth-based elevation features of LiDAR data. Subsequently, the extracted features are fed into a novel encoder comprising a feature enhancement module, designed with mathematical morphology and a dilated convolutional module derived from the self-attention of the conventional Transformer encoder (DConvformer), which plays a crucial role in integrating multi-source information within the network. The well-designed architecture enables the network to acquire multi-scale depth and high-order features, significantly reducing the number of training parameters. Comparative experimental results and ablation studies demonstrate that M2FNet outperforms other advanced methods. The source code is publicly available at https://github.com/cupid6868/M2FNet.git.

The spatial resolution of hyperspectral images (HSIs) is usually limited due to internal imaging mechanisms. To obtain imagery with high spectral and high spatial resolutions, which is essential for subsequent HSI processing tasks, a cost-effective approach is to fuse HSI with multispectral images (MSIs). One highly effective fusion method is the convolutional neural network (CNN). However, CNNs have limitations in capturing global information and complex features. Recently, visual transformers (ViTs) have garnered interest for their ability to process non-local information. Despite this, existing HSI-MSI fusion methods suffer from insufficient spatial-spectral feature interaction, resulting in suboptimal fusion quality. To address these challenges, we propose a multiscale deep cross-fusion transformer (MDC-FusFormer) network for HSI and MSI fusion. This network effectively performs the interactive fusion of spatial-spectral features, thereby enhancing the quality of the fused images. MDC-FusFormer employs a three-branch network architecture consisting of two independent progressive feature mining modules (PFMMs), a multiscale deep cross-fusion attention module, and a spatial-spectral feature fusion module. Initially, shallow features at different scales of MSI and HSI are recursively extracted through successive up- and down-sampling using CNNs. These features then interact with the deep cross-modal information at corresponding scales through the attention block. Finally, a multidimensional refinement convolution block (MRCB) is applied to refine the feature information, which is then combined with cascaded up-sampling to reconstruct the high-resolution fused image step by step. Experimental results on five datasets indicate that, compared to nine other methods, MDC-FusFormer delivers superior performance.

In recent years, convolutional neural networks (CNNs) have achieved remarkable success in hyperspectral image (HSI) classification tasks, primarily due to their outstanding spatial feature extraction capabilities. However, CNNs struggle to capture the diagnostic spectral information inherent in HSI. In contrast, vision transformers exhibit formidable prowess in handling spectral sequence information and excelling at capturing long-range correlations between pixels and bands. Nevertheless, due to the information loss during propagation, some existing transformer-based classification methods struggle to form sufficient spectral-spatial information mixing. To mitigate these limitations, we propose a memory-augmented spectral-spatial transformer (MASSFormer) for HSI classification. Specifically, MASSFormer incorporates two efficacious modules, the memory tokenizer (MT) and the memory-augmented transformer encoder (MATE). The former serves to transform spectral-spatial features into memory tokens for storing prior knowledge. The latter aims to extend traditional multi-head self-attention (MHSA) operations by incorporating these memory tokens, enabling ample information blending while alleviating the potential depth decay in the model, and consequently improving the model’s classification performance. Extensive experiments conducted on four benchmark datasets demonstrate that the proposed method outperforms state-of-the-art methods. The source code is available at https://github.com/hz63/MASSFormer for the sake of reproducibility.

Hyperspectral image (HSI) target detection plays a pivotal role in both military and civilian sectors. Nevertheless, this task is fraught with challenges because of the limited availability of target samples and the intricate nature of the background within real-world HSIs. In this study, we present an innovative background learning model based on the orthogonal subspace-guided variational autoencoder, tailored to discern the background distribution in hyperspectral imagery. Given the scarcity of target samples, our model is exclusively trained on background spectral samples, enabling precise modeling of the background distribution. The crux of our approach lies in detecting disparities between the reconstructed HSI and the original HSI, providing a mechanism for faithful target identification. To procure background samples, a coarse detection of the test HSI is first conducted. However, this process proves challenging, as obtaining pristine background pixels is a formidable task. To mitigate the influence of suspicious target samples on the background reconstruction, we employ orthogonal subspace loss on the reconstructed HSI. Extensive experiments conducted on four real-world HSIs substantiate that the proposed framework performs highly competitively and the results outperform other state-of-the-art HSI target detection methods. The source codes of this study are available at https://github.com/CX-He/OS-VAE.

Constrained by the performance of edge devices and real time (RT) processing technology, the existing hyperspectral target detection algorithms often struggle to rapidly distinguish targets from complex background pixels during real-time detection. To address this issue, this article proposes a new real-time cloud-edge selective background energy constrained (CE-SBEC) hyperspectral target detection algorithm. This algorithm aims to obtain detection results in real-time after capturing new data. Moreover, it conducts in-depth analysis based on existing detection results and updates the algorithm’s internal data to enhance its capabilities in terms of global background annihilation (GBA) and complex background suppression (CBS). Consequently, it improves the accuracy of subsequent real-time detection results. To enhance the resource utilization, this article deploys various task nodes of the algorithm separately on both the cloud and the edge, enabling collaborative execution of the CE-SBEC algorithm. In our context, edge devices are airborne equipment designed for the rapid acquisition and processing of data at the site of data collection, while cloud computing devices refer to high-performance computing clusters situated at a significant distance from the data collection site. Experimental results demonstrate that compared with existing detection algorithms, our newly proposed method achieves more accurate detection results while ensuring real-time performance.

Unsupervised domain adaptation (UDA) reduces domain shifts between distributions to enable model generalization to new scenarios. Adversarial domain adaptation (DA) is an effective approach that extracts domain-invariant features through adversarial learning, but such methods often neglect the influence of category differences on domain discrimination. To solve this problem, we construct a new unsupervised domain adaption hyperspectral image (HSI) classification method. The proposed method consists of two modules, namely, the pairing domain discrimination learning module and the multilevel mutual information maximization module. We propose to construct the sample pair as the input of the domain discriminator. We introduce a new label to the sample pair according to the labels of the two samples and use the relationship between samples to reduce the impact of sample category differences on domain discrimination. When extracting the shared features of the two domains, it will inevitably cause the loss of task-related information. This information is retained by maximizing the proposed multilevel mutual information. The experimental results on different datasets show the effectiveness of our method.

In hyperspectral cross-scene classification tasks, it is often challenging to obtain target domain samples during the training phase. Therefore, models need to be trained on one or multiple source domains and achieve good generalization performance on unknown target domains, known as domain generalization. The presence of domain shift limits the model’s generalization across different domains, while the unknown target domain makes it difficult to accurately characterize the distribution differences between domains. To address this issue, we propose a generalization network based on nonlinear sample generation. The network divides the sample features into invariant features and variant features and generates samples by applying nonlinear transformations to the variant features. To ensure the quality of the generated samples, we introduce contrastive learning into the model. It ensures consistency in similarity between the generated samples and the source samples while maintaining a certain degree of dissimilarity. Experiments conducted on four cross-domain adaptive scenarios demonstrate the superior performance of our proposed method.

The detection of oriented objects in remote sensing images remains a daunting challenge due to their complex backgrounds, various sizes, and especially arbitrary orientations. However, most of the existing methods only model the structural features of the images in the spatial domain, while the horizontal convolution kernels limit the model’s ability to perceive object direction information. Furthermore, the frequency features contain rich information about scale, texture, and angle, which can be a good complement to the spatial features. Inspired by this, we propose a multiscale spatial-frequency domain network (MSFN) to utilize spatial-frequency information for oriented object detection, which can be integrated into any convolutional neural network (CNN) architectures seamlessly and perform end-to-end training easily. Firstly, multiscale Haar wavelet transforms are leveraged to extract the multiscale frequency domain features from the image. Subsequently, channel alignment feature fusion module (CA-FFM) is proposed to fuse the high-level semantic features extracted by CNN with the low-level texture features extracted by the wavelet transform in multiscale. Finally, a channel self-attention (CSA)-based spatial-frequency feature perception module (SFPM) is designed to perform self-attention weighted aggregation on the fused features along the channel dimension, thereby constructing a novel spatial-frequency feature extraction backbone network for oriented object detector in remote sensing images. Experimental results on the DOTA and HRSC2016 datasets validate the effectiveness and universality of the proposed method.

Hyperspectral anomaly detection (AD), as a frontier research topic in the field of remotely sensed data processing, aims to identify targets of interest from complex and vast images. Existing AD methods typically involve complex models and many parameters, posing challenges in meeting the requirements of computational efficiency in hyperspectral AD. To address this issue, this article presents an AD acceleration algorithm based on the multivariate Gaussian model as well as its field programmable gate array (FPGA) implementation. By exploiting the parallel processing capabilities of FPGA, we introduce an innovative spectral dimensionality reduction method in which the data processing flow can be accomplished in a distributed manner. Then, we employ an improved linear rotation strategy based on correlation coefficients to accelerate the convergence rate of the proposed AD algorithm. The rotation of Gaussianization in the improved strategy is independent of eigenvalue decomposition, thereby substantially reducing the computational complexity involved during the rotation procedure. Furthermore, we apply a pipeline parallel mechanism to facilitate the FPGA implementation of the AD algorithm and to significantly enhance the computational efficiency. Experimental results on an embedded FPGA platform demonstrate that the FPGA implementation of the hyperspectral AD algorithm proposed in this article achieves a significant acceleration rate with guaranteed high detection accuracy.

Oriented object detection (OOD) in remote sensing images (RSIs) remains a challenging work due to an arbitrary orientation of instance. Learning rotation-invariant features is critical in modeling a fixed descriptor for instances with its rotated variants. However, most existing methods construct the descriptor from the perspectives of data or feature augmentation, but ignore the exploration of potentially useful supervision information inside the detection algorithm. In this paper, we propose an object-wise rotation-invariant semantic representation (ORSR) framework, which synergizes the exploration of latent supervision, rotation-invariant learning, and guided attention mechanism into a unified network to boost the performance of OOD in RSIs. First, supervised by our constructed pseudo ground truth of segmentation masks, a semantic segmentation branch is built along with the detection algorithm to refine the representation of backbone features. Moreover, a consistency loss function is proposed to encourage the segmentation branch to make the fixed predictions for backbone features with its rotated variants. Considering that segmentation predictions remain the same affine transformations before and after rotating, we further construct a Kullback-Leibler (KL) Divergence based similarity loss function for encouraging the network to model the rotation-invariant features. Finally, we separate the ”object” descriptor from the segmentation predictions to extend the implicit constraint in our proposed semantic segmentation branch. The separated ”object” descriptor not only involves the spatial regularizer to emphasize the high-responsive regions in image, but also can be guided by the constructed consistency loss function. We evaluate our proposed ORSR on the challenging DOTA, DIOR-R, and HRSC2016 datasets. Extensive experiments demonstrate that the proposed ORSR achieves competitive performance compared to other single-scale and multi-scale detection methods.

Hyperspectral image (HSI) super-resolution, which aims at improving the spatial quality of HSIs by fusing a low spatial resolution HSI (LR-HSI) with a high spatial resolution multispectral image (HR-MSI), has drawn significant attention. Numerous LR-HSI and HR-MSI (HSI-MSI) fusion algorithms have emerged in recent times, yet they suffer from a lack of generality and integration, which hampers their usability for non-expert users. Moreover, these algorithms encounter significant challenges due to the exponential increase in remote sensing data volume. In this study, we propose a unified cloud-based framework for HSI-MSI fusion based on the general distributed alternating direction method of multipliers that incorporates nonlocal principles and tensor decomposition. The framework not only provides end-users with visualization modeling capabilities equipped with standard and comprehensive components, but also enhances the parallel processing capabilities of cloud computing. We employ a new proposed nonlocal adaptive low-rank coupled tensor canonical polyadic (CP) decomposition algorithm as a case study to evaluate the performance of this framework. Specifically, we establish the LR-HSIs and HR-MSIs relationship using order-4 coupled tensor CP decomposition and suggest an adaptive CP rank estimation method for achieving better super-resolution results. Experimental results on publicly available datasets demonstrate that the proposed parallel distributed optimization algorithm can achieve significant speedup with guaranteed accuracy. The proposed framework enables convenient and efficient processing of large-scale remote sensing data, effectively addressing the challenges associated with handling large data volumes. The source code of our method is released and available online at https://github.com/ZpWaitingForSunshine/DNAC4TCP/.

The recently proposed high-order tensor algebraic framework generalizes the tensor singular value decomposition (t-SVD) induced by the invertible linear transform from order-3 to order-d ( ). However, the derived order-d t-SVD rank essentially ignores the implicit global discrepancy in the quantity distribution of non-zero transformed high-order singular values across the higher modes of tensors. This oversight leads to suboptimal restoration in processing real-world multi-dimensional visual datasets. To address this challenge, in this study, we look in-depth at the intrinsic properties of practical visual data tensors, and put our efforts into faithfully measuring their high-order low-rank nature. Technically, we first present a novel order-d tensor rank definition. This rank function effectively captures the aforementioned discrepancy property observed in real visual data tensors and is thus called the discrepant t-SVD rank. Subsequently, we introduce a nonconvex regularizer to facilitate the construction of the corresponding discrepant t-SVD rank minimization regime. The results show that the investigated low-rank approximation has the closed-form solution and avoids dilemmas caused by the previous convex optimization approach. Based on this new regime, we meticulously develop two models for typical restoration tasks: high-order tensor completion and high-order tensor robust principal component analysis. Numerical examples on order-4 hyperspectral videos, order-4 color videos, and order-5 light field images substantiate that our methods outperform state-of-the-art tensor-represented competitors. Finally, taking a fundamental order-3 hyperspectral tensor restoration task as an example, we further demonstrate the effectiveness of our new rank minimization regime for more practical applications. The source codes of the proposed methods are available at https://github.com/CX-He/DTSVD.git.

Cloud computing has attracted wide attention from both academia and industry, since it can provide flexible and on-demand hardware and software resources as services. Energy consumption of cloud servers is the main concern of cloud service providers since reducing energy consumption can bring them a lower operation cost (and hence a higher profit) and alleviate carbon footprints to the environment. Typically, the common power management techniques for enhancing energy efficiency would make cloud servers more vulnerable to soft errors and hence adversely impact the quality of services. Thus, reliability cannot be ignored in the design of methodologies for improving the energy efficiency of cloud servers. In this article, we aim to minimize the energy consumption of cloud servers under the soft-error reliability constraint by configuring the size and speed of servers. Specifically, we first derive the expected reliability based energy consumption of cloud servers to formulate the reliability-constrained energy minimization problem. We then leverage the reinforcement learning technique to obtain an optimal server configuration solution that maximizes system energy efficiency while maintaining the system reliability constraint. Finally, we perform extensive simulation experiments to analyze the relationship between system energy consumption and server configuration under varying arrival rates and execution requirements of service requests. Comparative experiments are also performed to validate the efficacy of the proposed learning-based server configuration scheme. Results show that compared to a benchmark method, the energy saved by the proposed scheme can reach up to 31.5%.

Hyperspectral target detection (HTD) aims to detect fine targets in hyperspectral images (HSIs). The traditional HTD method in low-resolution hyperspectral image (LR-HSI) is incapable of detecting small targets, clearly and precisely. Accordingly, in this paper, we propose a hyperspectral and multispectral image fusion target detection method based on cloud-edge collaboration. In this method, LR-HSI is first employed for coarse detection with the output of some suspicious target areas. Afterwards, the hyperspectral images and multispectral images (HSI-MSI) fusion is performed on these areas for precise target detection. In order to ensure the efficiency of HTD, we intend to accelerate our method in parallel based on the cloud-edge collaborative architecture. Furthermore, we establish an optimization model and design a greedy strategy to achieve the optimal deployment for minimizing the shortest runtime on the cloud-edge collaborative architecture. The experimental results demonstrate that our proposed method can significantly improve the computational efficiency while ensuring the accuracy.

Object detection in remote sensing images (RSIs) plays an important role both in civil and military fields. Currently, many object detection algorithms in RSIs have shown the excellent capability. However, these methods are designed for the single RGB modality, which cannot cope with the challenges in insufficient illumination or foggy scenarios. Infrared images measure the temperature of the captured objects, and it can avoid the influence of low illumination and fog. In this paper, we propose a novel RGB-Infrared multi-modal remote sensing object detection method termed as RIFuse to address these challenges. RIFuse combines convolutional neural networks (CNNs) and Transformer in a parallel hierarchy, which can efficiently extract the local features of RGB images and the global representations of infrared images. Besides, an adaptive multi-modal feature fusion block (MFF block) is proposed to fuse the features from both branches comprehensively. Extensive experiments demonstrate the superiority of our method for multi-modal object detection on RSIs.

To meet the need for efficient execution of hyperspectral remote sensing image classification algorithms, this paper proposes a fine-grained parallel optimization method for a CPU-GPU heterogeneous hyperspectral image superpixel spectral fusion classification algorithm based on cloud computing. Ray is used as the distributed computing engine to fully utilize the logical control ability and large-scale parallel computing ability of the CPU-GPU heterogeneous platform. We first decouple the superpixel spectral fusion classification algorithm, analyze the data dependence and computing characteristics of sub-tasks, use the GPU to accelerate the algorithm, and then further extend the algorithm to the CPU-GPU heterogeneous platform. At the same time, we establish a scheduling model for algorithm task scheduling problems, specifying the value of the parallelism degree for the algorithm in a fine-grained manner. It is verified by experiments that the parallelization method proposed in this paper can effectively improve the execution efficiency with the premise of the accuracy unreduced.

Classification using multi-source remote sensing data has attracted widespread attention and is playing an increasingly important role in various fields. However, due to the disparity in imaging methods and the informational imbalance among data from various sources, it is still challenging to incorporate complementary advantages. Furthermore, real-world situations frequently involve varied data resolution, and the outcomes of straightforward super-resolution preprocessing are not always helpful for performing subsequent classification tasks. In this paper, we propose a unified framework for joint super-resolution and classification tasks of low resolution hyperspectral images (LR-HSIs) and SAR images. The minimization of the proposed joint loss function in the generative adversarial network (GAN) framework, including super-resolution and classification objective functions, can effectively achieve the multi-task goals. Experimental results on the super-resolution and classification of real LR-HSIs and SAR images demonstrate the effectiveness of the proposed method both visually and quantitatively.

Recently, tensor singular value decomposition (t-SVD) has demonstrated excellent performance in various high-dimensional information processing applications. However, in adapting t-SVD to handle the typical tensor data restoration tasks, such as hyperspectral image (HSI) denoising, the following questions remain inadequately addressed: 1) The existing tensor nuclear norm minimization (TNN) regime treats all tensor singular values alike; thus, it lacks flexibility and dominance in dealing with the sophisticated HSI tensor. 2) The existing t-SVD-based denoising methods can not directly process order-p(p> 3) tensors; thus, they fail to comprehensively exploit the high-dimensional structural correlation of the HSI tensor along different modes. To address the above challenges, in this study, we first generalize a novel weighted order-pTNN minimization regime, which integrates the adaptively reweighting strategy for matrix, third-order, and order-ptensors in a unified architecture. Subsequently, an efficient subspace low-rank learning model is established, using HSI denoising tasks as an application example to corroborate the superiority of the proposed regime in approximating the high-dimensional low-rank structure of natural tensor data. Extensive experimental results substantiate that our effort surpasses existing state-of-the-art low-rank tensor recovery methods in both restoration accuracy and efficiency. The source code is available at https://github.com/CX-He/WTNN.git.

Target detection based on hyperspectral image (HSI) representations has drawn wide attention given its wide variety of features. The matrix-based approach inevitably loses spatial information and fails to explore the intrinsic multimodal structure of an HSI cube. In this paper, we propose a regularized tensor-based model without altering the data structure. We assume that an observed third-order HSI tensor is decomposed into the sum of a Total Variation regularized Low-rank background tensor and a Sparse (TVLrS) target tensor. The two tensors are represented as the mode-3 product of a third-order tensor, called the Tensor Representation Coefficient (TRC), and a spectra dictionary matrix. Then, the model is coined as TVLrS-TRC. The background TRC has a low-rank property, contributing to the low-rankness characterization in our model. Moreover, as the size of the background TRC term is smaller than the background tensor, characterizing its local smoothness via TV regularization reduces the computational cost compared to that of the background tensor. Extensive experiments on two real hyperspectral datasets demonstrate the advantage of the proposed method compared with the state-of-the-art.

Hyperspectral image super-resolution aims to fuse a low spatial resolution hyperspectral image (LR-HSI) and a high spatial resolution multispectral image (HR-MSI) to obtain a high-resolution hyperspectral image (HR-HSI). Tensor-based methods have demonstrated their outstanding ability in constructing the relationship between the LR-HSI and the HR-MSI. This paper introduces a nonlocal hierarchical Tucker decomposition (HTD) model for hyperspectral and multispectral image (HSI-MSI) fusion. First, similar nonlocal patch tensors are clustered according to their similarity in the HR-MSI. Next, the spatial/spectral relationship between the LR-HSI and the HR-MSI is extracted through HTD. The alternating direction method of multipliers (ADMM) is employed to solve the proposed model. Furthermore, to overcome the high computational complexity of the model solver, we propose an efficient distributed and parallel method to accelerate the fusion process. Experimental results demonstrate that the proposed method not only substantially outperforms state-of-the-art HSI-MSI fusion methods, but also achieves a significant acceleration rate.

As the backbone of Industry 4.0, industrial cyber-physical systems (ICPSs) that are geographically dispersed, federated, cooperative, and security-critical systems become the center of interest from both industry and academia. In ICPS, there are huge amounts of devices, such as sensors and actuators, which are embedded and networked together to improve the performance of real-time monitoring and control. Reliability and temperature are two important concerns of these embedded and networked devices in ICPS due to their stringent requirement of reliable execution and long lifespan. In this article, we study the problem of maximizing soft-error reliability of CPU- and GPU-integrated embedded platforms deployed in ICPS under the temperature constraint. To speed up the estimation of soft-error rate (SER) and temperature, we train an artificial neural network (ANN) that is able to quickly and accurately derive the system’s SER and temperature. To solve the temperature-constrained reliability optimization problem, we propose a feedback control-based task scheduling scheme that adaptively determines the number of tasks admitted in the system and the number of replicas for the admitted tasks. We perform a series of simulation experiments to verify the efficacy of our scheme. The experimental results demonstrate that: 1) the estimated SER and temperature derived by our ANN-based method are very close to the ground-truth data and 2) our proposed feedback control-based task scheduling method can improve system reliability by up to 184.2% with a lower peak temperature when compared with one baseline and two state-of-the-art methods. Note to Practitioners—This article is motivated by the safety-critical industrial cyber-physical system (ICPS) applications necessitating reliable execution and long lifespan, which could be realized by increasing reliability and controlling operating temperature. Our goal is to improve the system reliability of CPU- and GPU-integrated multiprocessor systems-on-chip (MPSoCs) deployed in ICPS under the temperature constraint. Most of the existing papers target either reliability or temperature. A few recent papers have focused on reliability and temperature optimization simultaneously. However, they are not designed for ICPS and do not consider the widely accepted CPU- and GPU-integrated MPSoC platforms. This article proposes a machine learning-based approach that trains an artificial neural network (ANN) to facilitate the online estimation of system SER and temperature. Compared to the offline estimation using simulation tools, the online approach is more applicable to real-time ICPS applications. This article also designs a feedback control-based approach for improving system reliability and reducing peak temperature of the CPU- and GPU-integrated MPSoCs by determining the number of tasks to be admitted and the number of replicas for tasks.

In hyperspectral images (HSIs), mixed noise (e.g., Gaussian noise, impulse noise, stripe noise, and deadlines) contamination is a common phenomenon that greatly reduces the visual quality of the image. In recent years, methods combining global and non-local low-rankness have been widely used in the field of HSI denoising. However, most methods apply original space-based denoising strategies (low-rank tensor decomposition, total variation, and tensor sparse representation, etc.) directly to the modeling of non-local low-rank tensors in subspace, without fully exploiting the intrinsic and latent properties of the non-local similar tensors. In this paper, we propose a hybrid prior denoising method based on global tensor low-rankness and non-local SVD-aided group sparsity (GTL_NSGS). This method introduces a novel plug-and-play NSGS denoiser that uses singular value decomposition as assistance to successively explore self-similarity of spatial dimension, low-rankness of spectral dimension, and group sparsity of difference domain in subspace non-local similar tensors. Globally, we utilize the existing three-way log-based tensor nuclear norm (3DLogTNN) to approximate the HSI tensor fibered rank and introduce a difference continuity regularization to obtain a continuous smooth spectral basis. Finally, we combine the Alternating Direction Method of Multipliers (ADMM) with the Augmented Lagrangian Multiplication (ALM) algorithm to solve the proposed model effectively. Extensive experiments on simulated and real data sets demonstrate that the proposed method has superior performance in removing mixed noise compared to state-of-the-art denoisers.

Attention mechanisms improve the classification accuracies by enhancing the salient information for hyperspectral images (HSIs). However, existing HSI attention models are driven by advanced achievements of computer vision, which are not able to fully exploit the spectral-spatial structure prior of HSIs and effectively refine features from a global perspective. In this paper, we propose a unified attention paradigm (UAP) that defines the attention mechanism as a general three-stage process including optimizing feature representations, strengthening information interaction, and emphasizing meaningful information. Meanwhile, we designed a novel efficient spectral-spatial attention module (ESSAM) under this paradigm, which adaptively adjusts feature responses along the spectral and spatial dimensions at an extremely low parameter cost. Specifically, we construct a parameter-free spectral attention block that employs multi-scale structured encodings and similarity calculations to perform global cross-channel interactions, and a memory-enhanced spatial attention block that captures key semantics of images stored in a learnable memory unit and models global spatial relationship by constructing semantic-to-pixel dependencies. ESSAM takes full account of the spatial distribution and low-dimensional characteristics of HSIs, with better interpretability and lower complexity. We develop a dense convolutional network based on efficient spectral-spatial attention (ESSAN) and experiment on three real hyperspectral datasets. Experimental results demonstrate that the proposed ESSAM brings higher accuracy improvement compared to advanced attention models.

Hyperspectral image (HSI) classification is currently a hot topic in the field of remote sensing. The goal is to utilize the spectral and spatial information from HSI to accurately identify land covers. Convolution neural network (CNN) is a powerful approach for HSI classification. However, CNN has limited ability to capture non-local information to represent complex features. Recently, vision transformers (ViTs) have gained attention due to their ability to process non-local information. Yet, under the HSI classification scenario with ultra-small sample rates, the spectral-spatial information given to ViTs for global modeling is insufficient, resulting in limited classification capability. Therefore, in this article, Multi-Attention Joint Convolution Feature Representation with Lightweight Transformer (MAR-LWFormer) is proposed, which effectively combines the spectral and spatial features of HSI to achieve efficient classification performance at ultra-small sample rates. Specifically, we use a three-branch network architecture to extract multi-scale convolved 3D-CNN, EMAP, and LBP features of HSI, respectively, by taking full exploitation of ultra-small training samples. Second, we design a series of multi-attention modules to enhance spectral-spatial representation for the three types of features and to improve the coupling and fusion of multiple features. Third, we propose an explicit feature attention tokenizer to transform the feature information, which maximizes the effective spectral-spatial information retained in the flat tokens. Finally, the generated tokens are input to the designed lightweight transformer for encoding and classification. Experimental results on three datasets validate that MAR-LWFormer has an excellent performance in HSI classification at ultra-small sample rates when compared to several state-of-the-art classifiers.

For unmixing of sequences of hyperspectral images (SHS), spectral variability is an important factor to be considered. However, most existing unmixing methods tend to model the endmember and its variability in spatial domain rather than transform domain. In fact, the intrinsic and invariant features of the spectral curve can be effectively represented by wavelet transform. Therefore, this paper proposes to perform SHS unmixing in the wavelet domain by combing the Bayesian method. Firstly, the assumption of abundance being invariability in both the spatial and wavelet domains is made, then the formulation of unmixing in the wavelet domain using Perturbed Linear Mixing Model (PLMM) is presented. Secondly, based on the Bayesian framework, the likelihood and prior are both given, in which the parameter priors are divided into two parts: low and high frequency wavelet coefficients. Moreover, by considering the sparsity of the high-frequency wavelet coefficients of endmembers, a non-informative prior with zero-mean is designed. Meanwhile, for the coefficients of endmember variability, Gaussian distributions are utilized to represent the steady fluctuation along the temporal dimension. Finally, using the maximum a posterior (MAP) rule, a hierarchical spectral variability unmixing model in wavelet domain is built and solved by the Markov chain Monte Carlo (MCMC) sampling algorithm. Numerical experiments show that the proposed method generates more accurate estimates for endmembers and their variation.

Hyperspectral anomaly detection (AD) task is a typical binary classification problem, and utilizing background prior knowledge is a key technique to solving such problems. The two most commonly used priors for hyperspectral images are low-rank and local smooth properties. Most traditional matrix-based methods use two regularizations to model these two types of priors and integrate them into one model, which makes these two regularizations unable to maximize their effectiveness. In addition, the matrix method also destroys the structure of the hyperspectral images (HSI). To address these issues, this study identified a unique sparsity property in the gradient tensor of HSI. Specifically, the core tensor resulting from the Tucker decomposition of the gradient tensor was observed to exhibit sparsity. This sparsity property, referred to as GCS (the sparsity on the core tensor of the gradient map), effectively captures the structural information of HSI and improves detection performance. The GCS regularization offers the following advantages: 1) GCS regularization uses one term to simultaneously capture both low-rankness and local smoothness, the size of the core tensor represents the low-rank prior to the background, and the ℓ1norm describes the sparsity of gradient map, i.e., the local smoothness of the original data; 2) GCS is a constrained regularization, allowing for the full utilization of information from different dimensions of the HSI when updating the core tensor, i.e., utilizing the spatial and spectral information carried by three-factor matrices of the Tucker decomposition. Finally, extensive experiments validate the superiority of our proposed methods.

Despite the remarkable progress made by the salient object detection of natural sensing images (NSI-SOD), the complex background and scale diversity issues of remote sensing images (RSIs) still pose a substantial obstacle. In this study, we build an end-to-end channel-enhanced remodeling-based network (CRNet) for optical RSIs (ORSIs) to highlight salient objects through feature augmentation. First, the backbone convolutional block is used to suggest the fundamental characteristics. Then, we use the channel enhance module (CEM) to enhance the shallow features. CEM primarily relies on the channel attention mechanism and employs a no-downscaling strategy to produce local cross-channel interaction, which lowers model complexity while enhancing extraction performance. Meanwhile, we use the redefined feature module (RFM) to reconstruct the deep features and generate global attention features by dimensional transformation and feature relationship aggregation to achieve the role of locating salient targets. Finally, the cascade combines the multi-scale features to provide the final saliency map. To further enhance the representational power of the network, we use a hybrid loss function to improve performance. The proposed approach outperforms current state-of-the-art methods, as shown by several experiments on three available datasets. The source code of the proposed CRNet is available publicly at https://github.com/hilitteq/CRNet.git.

Classifying high-dimensional hyperspectral image (HSI) with limited labeled samples is a difficult problem. One effective solution is to leverage knowledge from scenes with well-labeled image (the source domain) to aid training in the target domain. However, since the source and target domains have different category spaces, it is crucial to extract more discriminative features and address domain adaptation challenges. To tackle this issue, we propose a cross-scene classification method for HSIs via generative adversarial networks (GANs) in latent space (GLS). Our method employs autoencoders (AEs) to map the input data to a latent space, where the most effective feature representation is extracted and preserved by deep residual 3D convolutional neural networks (CNN). The unlabeled samples in the target domain are also utilized in the AE which ensure all the samples are considered. We leverage conditional adversarial domain adaptation to overcome the domain shift, and introduce maximum mean discrepancy loss to minimize distribution differences between the two domains, facilitating better domain distribution alignment. We tested our approach on three public datasets and demonstrated that it outperforms existing few-shot learning methods. Our results highlight the effectiveness of our classification method via GANs in latent space for HSIs, and show that it has potential for practical applications.

In recent years, deep learning-based methods have been extensively utilized in remote sensing image scene classification and have achieved remarkable performance. The wide geographical coverage and resolution differences of scene images result in significant within-class diversity and between-class similarity, hindering the further improvement of classification accuracy. Attention-based methods automatically estimate the importance of local regions by learning weight assignments, which effectively enhance the feature extraction capability of the network. However, methods that solely rely on the network to automatically learn weight assignments may introduce biases in the attention calculations. By analyzing the specific contribution of local features to the key components of global semantics, we propose a collaborative dictionary learning network (CDLNet). CDLNet utilizes the collaborative representation method to decompose global features into a set of key semantic vectors to guide the attention learning process of the network. Specifically, we design a semantic summarization module (SSM), which reconstructs global semantic features by optimizing a low-redundancy dictionary. Next, we propose a global semantic attention module (GSAM), which calculates the contribution of local features to the global feature key information based on their correlation with the reconstructed key semantic set. Finally, an attention transfer loss is introduced to further enhance the attention of low-level feature maps. The experimental results on three publicly available datasets demonstrate that CDLNet can effectively improve within-class diversity and between-class similarity by optimizing the attention learning of the network, thereby achieving great promotion in comparison with state-of-the-art methods. The implementation is publicly available at https://github.com/liuofficial/CDLNet.

Significant progress has been achieved in remote sensing image scene classification (RSISC) with the development of convolutional neural networks (CNNs) and vision transformers (ViT). However, high intra-class diversity and inter-class similarity are still enormous challenges for RSISC. Metric learning can effectively improve the discriminative ability of deep representations by constraining the distance between features. Previous metric learning methods only optimize the feature space representation through metric function, ignoring the information interaction between samples. For complex scene images, similarity and discriminative knowledge need to be summarized from the multiple positive and negative pairs. We propose a novel efficient multi-sample contrastive network (EMSCNet) to integrate knowledge from multiple samples. Specifically, we construct a dynamic dictionary with momentum updates to mine positive and negative pairs from the entire dataset. Then, the similarity and discriminative knowledge between samples are summarized by introducing a contrastive module. Finally, the knowledge of the contrastive module is transferred to the backbone classifier through knowledge distillation. The proposed contrastive module can be easily embedded into the training process of CNNs or ViT and removed during inference. Experimental results conducted on three datasets demonstrate the effectiveness of the proposed approach.

The joint use of multisource remote-sensing (RS) data for Earth observation missions has drawn much attention. Although the fusion of several data sources can improve the accuracy of land-cover identification, many technical obstacles, such as disparate data structures, irrelevant physical characteristics, and a lack of training data, exist. In this article, a novel dual-branch method, consisting of a hierarchical convolutional neural network (CNN) and a transformer network, is proposed for fusing multisource heterogeneous information and improving joint classification performance. First, by combining the CNN with a transformer, the proposed dual-branch network can significantly capture and learn spectral–spatial features from hyperspectral image (HSI) data and elevation features from light detection and ranging (LiDAR) data. Then, to fuse these two sets of data features, a cross-token attention (CTA) fusion encoder is designed in a specialty. The well-designed deep hierarchical architecture takes full advantage of the powerful spatial context information extraction ability of the CNN and the strong long-range dependency modeling ability of the transformer network based on the self-attention (SA) mechanism. Four standard datasets are used in experiments to verify the effectiveness of the approach. The experimental results reveal that the proposed framework can perform noticeably better than state-of-the-art methods. The source code of the proposed method will be available publicly athttps://github.com/zgr6010/Fusion_HCT.git.

In recent years, fusing multi-type features poses great potential for oriented object detection (OOD) in remote sensing images (RSIs). Due to the inexplicit operation of modeling orientation variations, convolutional neural networks (CNNs) are difficult to perceive objects under different transformations (angles and scales). In this paper, we propose a novel instance-aware spatial-frequency feature fusion detector (SFFD) for oriented object detection in remote sensing images. First, a layer-wise frequency-domain analysis (L-FDA) module is built along with CNN layers to extract frequency features. Getting rid of the constrains such as horizontal rectangular kernel in CNNs, our L-FDA possesses outstanding ability of locating mutational signals from frequency space. These mutational signals record the scale and angle information of the oriented instances in images. Subsequently, CNN and frequency features are sent into RoI Pooling layer to obtain multi-type instance-level RoI features. Moreover, the proposed instance-aware cross feature fusion (CFF) module explores the interaction between these diverse features which provides an explicit indicator to compensate the orientation information ignored by instance-level CNN features. Finally, our SFFD unifies the proposed L-FDA module and CFF module into the detection network to localize oriented instances in RSIs. We compare our method with many state-of-the-art methods on DOTA, HRSC2016, and NWPU VHR-10 datasets. Experimental results verify the validity of modeling instance-level object relations from frequency-domain and CNNs for OOD.

Low-rank tensor representation philosophy has enjoyed a reputation in many hyperspectral image (HSI) low-level vision applications, but previous studies often failed to comprehensively exploit the low-rank nature of HSI along different modes in low-dimensional subspace, and unsurprisingly handled only one specific task. To address these challenges, in this paper, we figured out that in addition to the spatial correlation, the spectral dependency of HSI also implicitly exists in the coefficient tensor of its subspace, this crucial dependency that was not fully utilized by previous studies yet can be effectively exploited in a cascaded manner. This led us to propose a unified subspace low-rank learning regime with a new tensor cascaded rank minimization, named STCR, to fully couple the low-rankness of HSI in different domains for various low-level vision tasks. Technically, the high-dimensional HSI was first projected into a low-dimensional tensor subspace, then a novel tensor low-cascaded-rank decomposition was designed to collapse the constructed tensor into three core tensors in succession to more thoroughly exploit the correlations in spatial, nonlocal, and spectral modes of the coefficient tensor. Next, difference continuity-regularization was introduced to learn a basis that more closely approximates the HSI's endmembers. The proposed regime realizes a comprehensive delineation of the self-portrait of HSI tensor. Extensive evaluations conducted with dozens of state-of-the-art (SOTA) baselines on eight datasets verified that the proposed regime is highly effective and robust to typical HSI low-level vision tasks, including denoising, compressive sensing reconstruction, inpainting, and destriping. The source code of our method is released at https://github.com/CX-He/STCR.git.

The synthesis of high-resolution (HR) hyperspectral image (HSI) by fusing a low-resolution HSI with a corresponding HR multispectral image has emerged as a prevalent HSI super-resolution (HSR) scheme. Recent researches have revealed that tensor analysis is an emerging tool for HSR. However, most off-the-shelf tensor-based HSR algorithms tend to encounter challenges in rank determination and modeling capacity. To address these issues, we construct nonlocal patch tensors (NPTs) and characterize low-rank structures with coupled Bayesian tensor factorization. It is worth emphasizing that the intrinsic global spectral correlation and nonlocal spatial similarity can be simultaneously explored under the proposed model. Moreover, benefiting from the technique of automatic relevance determination, we propose a hierarchical probabilistic framework based on Canonical Polyadic (CP) factorization, which incorporates a sparsity-inducing prior over the underlying factor matrices. We further develop an effective expectation-maximization-type optimization scheme for framework estimation. In contrast to existing works, the proposed model can infer the latent CP rank of NPT adaptively without tuning parameters. Extensive experiments on synthesized and real datasets illustrate the intrinsic capability of our model in rank determination as well as its superiority in fusion performance.

Combining tensor decomposition and image nonlocal information for remote sensing image fusion method (NCTCP) can effectively preserve the spatial structure of the image, and can obtain good image fusion results, accordingly. However, the NCTCP that is a serial algorithm cannot handle massive remote sensing images due to the computing resources limitation of a single computer. To address this issue, we propose a distributed parallel nonlocal tensor CP decomposition optimization algorithm (DP_NCTCP) based on the Spark platform. The alternating direction method of multipliers(ADMM) in NCTCP is divided into two distributed computing subtasks that can be executed on Spark in parallel to improve the efficiency. Compared with NCTCP, the DP_NCTCP achieves high speedups without the degradation of fusion quality measures accuracy by fusing the real hyperspectral images.

This paper proposes a Rotation-equivariant Attention Feature Fusion Pyramid Networks for Aerial Object Detection named ReAFFPN. ReAFFPN aims at improving the effect of rotation-equivariant features fusion between adjacent layers which suffers from the semantic and scale discontinuity. Due to the particularity of rotational equivariant convolution, general methods are unable to achieve their original effect while ensuring rotation equivariance of the network. To solve this problem, we design a new Rotation-equivariant Channel Attention which has the ability to both generate channel attention and keep rotation equivariance. Then we embed a new channel attention function into Iterative Attentional Feature Fusion (iAFF) module to realize Rotation-equivariant Attention Feature Fusion. Experimental results demonstrate that ReAFFPN achieves a better rotation-equivariant feature fusion ability and significantly improve the accuracy of the Rotation-equivariant Convolutional Networks.

Hyperspectral computational imaging (HCI) is to reconstruct hyperspectral images (HSIs) based on the compressed signals collected by remote sensing and imaging systems. Collaborative Tucker3 tensor decomposition is beneficial for HCI models in reconstructing high-fidelity HSIs. However, the everincreasing amount of compressed data leads to heavy computation burden for tensor decomposition-based HCI models, which may exceed the computing capacity of a single machine. For this reason, this paper proposes a Spark-based distributed and parallel HCI implementation via collaborative Tucker3 tensor decomposition. The proposed implementation decomposes the processing flow of the HCI algorithm into several stages, each of which can be processed in parallel on Spark. In addition, we develop parallel strategies for improving the performance of the redundant computational procedure and data storage procedure, respectively. Experimental results demonstrate that the parallel algorithm not only achieves high accuracy but also improves the computational efficiency when processing large-scale HSI datasets.

Hyperspectral target detection (HTD) is one of the vital issues in hyperspectral image (HSI) processing, and has a pivotal role in remote sensing field. However, it is a challenging task due to redundant information, high dimensionality and complexity of background. In this paper, an effective HTD method based on Tensor Robust Principal Component Analysis (TRPCA) is proposed, which can spot targets from complicated background distribution. Our approach, called constrained energy regularization TRPCA, imposes tensor nuclear norm to describe the low-rank property of back-ground. Considering some target spectra probably lie in the same low-dimensional background subspace, we introduce a constrained energy regularization to enhance the distinction between target and background pixels. Experiments on four datasets illustrate superiority of our method.

With the recent advances in mobile sensing, computing, and communication technologies, mobile cyber–physical systems (MCPSs) become a promising networking paradigm that provides mobile users with various applications and services from cyber space to the physical world. One of the main challenges in the MCPSs is to defend security threats launched by adversaries. Exploiting security services to defend threats is energy consuming, whereas the energy of mobile devices is generally limited since most of mobile devices are battery powered. This necessitates the need to design new methodologies to tackle the tradeoff between security and energy of MCPSs. To this end, this article aims to maximize MCPSs’ security under the constraints of energy and deadline. In this article, we first formulate the MCPS security maximization problem as a mixed-integer nonlinear programming (MINLP) problem and then transform it into a mixed-integer linear programming (MILP) problem without performance degradation. To solve the transformed MILP problem efficiently, we propose a decomposed algorithm to derive the optimum task scheduling solution instead of using MILP solvers that may be very time consuming for MCPSs of large granularity. The derived task scheduling solution decides the assignment, operating frequency, execution order, as well as security service selection for all tasks. We implement a series of simulation-based experiments to validate the proposed decomposed task scheduling scheme. Simulation results demonstrate that the proposed scheme increases system security level by 20.38% and 65.11% on average as compared to a state-of-art approach and a baseline method.

Hyperspectral image super-resolution addresses the problem of fusing a low-resolution hyperspectral image (LR-HSI) and a high-resolution multispectral image (HR-MSI) to produce a high-resolution hyperspectral image (HR-HSI). Tensor analysis has been proven to be an efficient method for hyperspectral image processing. However, the existing tensor-based methods of hyperspectral image super-resolution like the tensor train and tensor ring decomposition only establish an operation between adjacent two factors and are highly sensitive to the permutation of tensor modes, leading to an inadequate and inflexible representation. In this paper, we propose a novel method for hyperspectral image super-resolution by utilizing the specific properties of high-order tensors in fully-connected tensor network decomposition. The proposed method first tensorizes the target HR-HSI into a high-order tensor that has multiscale spatial structures. Then, a coupled fully-connected tensor network decomposition model is proposed to fuse the corresponding high-order tensors of LR-HSI and HR-MSI. Moreover, a weighted-graph regularization is imposed on the spectral core tensors to preserve spectral information. In the proposed model, the superiorities of the fully-connected tensor network decomposition lie in the outstanding capability for characterizing adequately the intrinsic correlations between any two modes of tensors and the essential invariance for transposition. Experimental results on three data sets show the effectiveness of the proposed approach as compared to other hyperspectral image super-resolution methods.

Change detection (CD) is a hot issue in the field of remote sensing. Hyperspectral images (HSI) contain rich spectral information, and have gradually become an important data source in CD. Spectral-spatial combination is a commonly used strategy for suppressing the influence of noise on the spectrum. However, it is difficult to find a feature space that allows both spectral and spatial features to be optimally expressed. Therefore, this letter proposes a bidirectional reconstruction coding network and enhanced residual network for HSI CD (i.e., BRCN-ERN) based on the strategy of completely extracting spectral and spatial features separately and then fusing them together. In the spectral module, we use the spectrum of unchanged pixels at two time points to construct a bidirectional reconstruction network, and use the reconstruction error as a new source of spectral features. In the spatial module, we use advanced band selection algorithms to filter the bands with good spatial information, and design an enhanced two-dimensional residual network to extract the spatial features of the change tensor. Finally, the obtained spectral and spatial feature vectors are fused and inputted into the fully connected classification network to obtain the final CD map. Real HSI experiments show that our proposed BRCN-ERN has a better CD effect and is more effective than most existing algorithms.

Tracking moving objects in remote-sensing videos is becoming increasingly important in remote-sensing analysis. This letter presents a novel object tracking method for remote-sensing videos. We start with using the traditional robust principal component analysis (RPCA) model to extract the moving object from the background. To describe the continuity of moving objects in spatial and temporal directions, we incorporate a 3-D total variation (3DTV) regularization into the RPCA model. Considering that the background is not static and the captured videos will contain noise because of the instability of the sensing camera, our proposed method introduces a certain part of the function to model the noise and capture the changes in background. Experimental results on real videos provided by 2016 IEEE GRSS Data Fusion Contest and 2020 Hyperspectral Object Tracking Challenge demonstrate the advantages of the moving object-tracking method via 3-D TV.

Nowadays, deep learning can play an important role in addressing the issue of hyperspectral anomaly detection (HAD). In order to further utilize the spatial information in hyperspectral images (HSIs), an anomaly detection method for HSIs is presented based on graph regularized variational autoencoder (GRVAE). Firstly, the proposed method uses the superpixel segmentation algorithm to segment the hyperspectral image (HSI) and constructs an adjacency matrix to evaluate the similarity between pixels. Secondly, a variational autoencoder is used to reconstruct the spectral vector of the HSI, and meanwhile, the spatial similarity of the image is shared in the feature space through the graph regularization term. Finally, the reconstructed background and the original input are used to obtain the spectral error map, and then the attribute filtering is used to further refine the detection results. Performed on four data sets of abnormal target data with different shapes and different background complexity, the experiments show that the method has promising anomaly detection performance.

A hyperspectral image sequence can be obtained at different time in the same region from a hyperspectral sensor. The environmental change usually leads to variation in endmember reflectance, which has an important influence on unmixing process. In this article, a Bayesian unmixing model considering spectral variability for hyperspectral sequence is proposed, in which composite prior distributions of abundance and endmember variability are developed. The abundance priors consider the continuity of abundance in the temporal and spatial domains, simultaneously. Specifically, in the spatial domain, a data-adaptive variance of the abundance prior distribution is put forward based on local spatial difference. Moreover, the priors of endmember variability in temporal continuity and spectral smoothness are also exploited. Finally, a joint posterior distribution is obtained by the likelihood function and the parameter prior distributions, which can be calculated by the Markov chain Monte Carlo (MCMC) algorithm. Experiments on synthetic and real data sets demonstrate the effectiveness of the proposed approach in terms of abundance, endmember, and its variability estimation accuracy.

Recently, deep learning has drawn significant attention in hyperspectral image (HSI) classification. With the growth of network depth and feature integration, deep learning demands abundant labeled samples to optimize many parameters. Unfortunately, most hyperspectral data are unlabeled and the available labeled samples are extremely limited. How to obtain richer features under limited training samples is a challenge for HSI classification. To tackle this issue, a new supervised multiscale alternately updated clique network (MSCN) is proposed for HSI classification to fully employ HSI features in different scales. Based on the Clique Block, we design the multiscale alternately updated clique block (MSCB) that applies convolution kernels of various sizes to adaptively exploit the multiscale HSI information and merge them within the block. Meanwhile, the recurrent feedback architecture is introduced to reuse high-level visual information and network parameters. The proposed MSCN includes two MSCBs to capture the multiscale spectral and spatial information in turn. The MSCN improves the information flow and the efficiency of parameter tuning through the feedback mechanism and the cross-utilization of multiscale feature. It not only obtains more abstract HSI information, but also reduces the network depth and the number of parameters, thereby improving the classification accuracy under limited samples. To certify the validity of the proposed MSCN, experiments are conducted on three real HSI datasets and compared with multiple state-of-the-art deep learning-based approaches. The experimental results demonstrate that the presented multiscale network achieves superior performance, especially in the case of a small number of training samples.

This article focuses on hyperspectral image (HSI) super-resolution that aims to fuse a low-spatial-resolution HSI and a high-spatial-resolution multispectral image to form a high-spatial-resolution HSI (HR-HSI). Existing deep learning-based approaches are mostly supervised that rely on a large number of labeled training samples, which is unrealistic. The commonly used model-based approaches are unsupervised and flexible but rely on handcrafted priors. Inspired by the specific properties of model, we make the first attempt to design a model-inspired deep network for HSI super-resolution in an unsupervised manner. This approach consists of an implicit autoencoder network built on the target HR-HSI that treats each pixel as an individual sample. The nonnegative matrix factorization (NMF) of the target HR-HSI is integrated into the autoencoder network, where the two NMF parts, spectral and spatial matrices, are treated as decoder parameters and hidden outputs, respectively. In the encoding stage, we present a pixelwise fusion model to estimate hidden outputs directly and then reformulate and unfold the model’s algorithm to form the encoder network. With the specific architecture, the proposed network is similar to a manifold prior-based model and can be trained patch by patch rather than the entire images. Moreover, we propose an additional unsupervised network to estimate the point spread function and spectral response function. Experimental results conducted on both synthetic and real datasets demonstrate the effectiveness of the proposed approach.

Hyperspectral image (HSI) fusion refers to the reconstruction of a high-resolution HSI by fusing a low-resolution HSI (LR-HSI) and a high-resolution multispectral image (HR-MSI) over the same scene. Recently, researchers have proposed many approaches to handle this issue. However, most of them assume that both the spatial and spectral degradation functions are known, which are often limited or unavailable in reality. This article presents a novel model-driven deep network based on matrix decomposition, which considers spectral correlations and reasonably embeds the well-known observation models. Specifically, the proposed method decomposes the desired HSI into spectral basis and coefficients. The spectral basis can be estimated from the LR-HSI via singular value decomposition. To learn the coefficients, a learning model is constructed by merging the observation models, matrix decomposition, and sparsity into a concise single formulation. For solving the proposed model, a deep framework is built by unrolling the alternating direction method of multipliers (ADMM), dubbed as ADMM-HFNet, where the involved parameters can be learned adaptively. It is worth noting that the spectral basis cannot fully represent the desired HSI. Therefore, another model is constructed here to supplement the approximation error, which can also be embedded in the deep network. After checking on three datasets, it is found that the proposed method stands out from advanced competing techniques in both quality measures and visual effects.

Hyperspectral image (HSI) classification refers to accurately corresponding each pixel in an HSI to a land-cover label. Recently, the successful application of multiscale and multifeature methods has greatly improved the performance of HSI classification due to their enhanced utilization of the available spectral–spatial information. However, as the number of scales and the number of features increases, it becomes more difficult to achieve an optimal degree of fusion for multiple classifiers [e.g., kernel extreme learning machine (KELM)]. On the other hand, a limited sample size of the HSI may cause overfitting problems, which seriously affects the classification accuracy. Therefore, in this article, a novel multi-structure KELM with attention fusion strategy (MSAF-KELM) is proposed to achieve accurate fusion of multiple classifiers for effective HSI classification with ultrasmall sample rates. First, a multi-structure network is built, which combines multiple scales and multiple features to extract abundant spectral–spatial information. Second, a fast and efficient KELM is employed to enable rapid classification. Finally, a weighted self-attention fusion strategy (WSAFS) is introduced, which combines the output weights of each KELM subbranch and the self-attention mechanism to achieve an efficient fusion result on multi-structure networks. We conducted experiments on four types of HSI datasets with different evaluation methods and compared them with several classical and state-of-the-art methods, which demonstrate the excellent performance of our method on ultrasmall sample rates. The code is available athttps://github.com/Fang666666/MSAF-KELMfor reproducibility.

In hyperspectral image (HSI) classification, each pixel sample is assigned to a land-cover category. In the recent past, convolutional neural network (CNN)-based HSI classification methods have greatly improved performance due to their superior ability to represent features. However, these methods have limited ability to obtain deep semantic features, and as the layer’s number increases, computational costs rise significantly. The transformer framework can represent high-level semantic features well. In this article, a spectral–spatial feature tokenization transformer (SSFTT) method is proposed to capture spectral–spatial features and high-level semantic features. First, a spectral–spatial feature extraction module is built to extract low-level features. This module is composed of a 3-D convolution layer and a 2-D convolution layer, which are used to extract the shallow spectral and spatial features. Second, a Gaussian weighted feature tokenizer is introduced for features transformation. Third, the transformed features are input into the transformer encoder module for feature representation and learning. Finally, a linear layer is used to identify the first learnable token to obtain the sample label. Using three standard datasets, experimental analysis confirms that the computation time is less than other deep learning methods and the performance of the classification outperforms several current state-of-the-art methods. The code of this work is available athttps://github.com/zgr6010/HSI_SSFTTfor the sake of reproducibility.

The traditional methods of hyperspectral image (HSI) classification are based on the sufficient labeled data. In real life, we often encounter that the target domain corresponding to the classification task has only a small amount of labeled data, but the source domain has enough labeled data. However, the distribution of the source domain is different from the distribution of the target domain. Thus, the labeled data of the source domain cannot be applied to the target domain directly. This paper proposes a new method to solve the cross-domain few shot problem of HSI classification. In the proposed method, the local spatial alignment and the spectral alignment are simultaneously introduced to transfer the knowledge from the source domain to the target domain. Besides, to extract the domain specific features, we balance the domain invariant features and the domain specific features by a weakly parameter-shared mechanism. The two modules together can narrow the distance between two domains and make the model perform well on the target domain. Experiments conducted on four different target domain data sets demonstrate the effectiveness of our method.

Recently, deep convolutional neural network based hyperspectral and multispectral image fusion methods have shown significant performance. Nevertheless, the rich spatial and spectral details of hyperspectral images (HSIs) have not been fully explored, leaving room for further improve the representation ability of the model. In this paper, we propose an efficient cross-modality self-calibrated network (CMSCN) for hyperspectral and multispectral image fusion. Specifically, we use a cross-modality non-local module to fuse a high-resolution multispectral image (HR-MSI) and a low-resolution hyperspectral image (LR-HSI) to get an enhanced LR-HSI. In addition, a novel cross-scale self-calibrated convolution structure is proposed to explore and exploit multi-scale and hierarchical spatial-spectral features, which can improve the learning ability of the model. The introduced efficient spatial-spectral attention mechanism can calibrate the feature representation at different dimensions, thereby providing more efficient and accurate information for hyperspectral image reconstruction. Extensive experimental results on various hyperspectral images demonstrate the superiority of our method in comparison with the state-of-the-art image fusion methods.

Synthesizing high resolution (HR) hyperspectral image (HSI) by merging a low resolution (LR) HSI with corresponding HR multispectral image (MSI) has become a promising HSI super-resolution scheme. Most existing HSI-MSI fusion methods are effective to some extent, while several challenges remain. First, the spectral response of a given material exhibits considerable variability due to different acquisition time and conditions, however variations in spectral signatures are often neglected. Second, a majority of off-the-shelf methods require predefined degradation operators, which can be unavailable in practice. To tackle above issues, we introduce a novel fusion approach with Bayesian framework. Specifically, we regard the up-sampled LR-HSI as the low frequency component of the underlying HR-HSI. We characterize the texture features of high and low frequency components respectively, which can enlarge modeling capacity and bypass the absence of degradation operators. Furthermore, we depict the relative smoothness of reflectance spectra with Gaussian Process. Extensive experiments on synthesized and real datasets illustrate the superiority of the proposed strategy in terms of fusion performance and robustness to spectral variability.

Recently, deep convolutional neural network-based low-resolution hyperspectral image (LR-HSI) and high-resolution multispectral image (HR-MSI) fusion methods have achieved significant performance improvement. However, the rich spatial and spectral information in HSIs is not fully explored. In this article, we propose a novel cross-scale octave network (CSONet) for hyperspectral and multispectral image fusion. Specifically, we adopt a progressive image fusion structure to effectively extract the spatial and spectral information of HR-MSI at multiple resolutions, thereby efficiently complementing LR-HSI’s information. In addition, the proposed cross-scale octave convolution module can extract rich multiscale spatial feature information and concentrate on more important spatial–spectral features at different scales with the multiscale spatial–spectral attention mechanism. Finally, a multisupervised loss function is used to improve the gradient propagation and enhance the representation ability of the network. Ablation analysis on the benchmark datasets shows the effectiveness of each component in the proposed method. Extensive experimental results on different hyperspectral images demonstrate that the proposed CSONet can achieve superior results and strong generalization ability in comparison with some state-of-the-art LR-HSI and HR-MSI fusion methods.

Object detection in remote sensing images (RSIs) poses great difficulties due to arbitrary orientations, various scales and dense location of the targets over the ground. Recent evidence suggests that encoding the orientation information is of great use for training an accurate object detector for oriented object detection (OOD). In this paper, we propose a new frequency-domain orientation learning (FDOL) module with two main components: the frequency domain feature extraction (FFE) network and an orientation enhanced self-attention layer (OES-Layer). The FFE network models the interactions among spatial locations in the frequency domain to determine the frequency of spatial features. Then, these features are fed into our OES-Layer to learn the orientation information. Moreover, the orientation weights are adopted to guide the feature selection in a self-attention architecture, using them as a control gate to emphasize the spatial responses of target instances. Considering that the original similarity weights (calculated by the self-attention algorithm) do not distinctly model the orientation variation, the considered orientation weights provide an efficient asset to emphasize the orientation of objects. Extensive experiments on the DOTA and HRSC2016 datasets demonstrate that our method achieves state-of-the-art performance among single-scale methods, while achieving competitive performance over multi-scale methods.

Anomaly detection (AD) has become a hot topic in hyperspectral image (HSI) analysis. Anomalies are samples that are significantly different from the surrounding background in space or spectrum. However, the rich spectral-spatial features in HSI are not fully discovered by most traditional AD methods. In this paper, a 3D-convolutional Variational Au-toencoder (3D-CVAE) based AD method is proposed to make full use of the spectral-spatial information. The spectral-spatial features are extracted by the 3D-CVAE encoder and the background is reconstructed using these features through 3D-CVAE decoder. The residual between the original input and the reconstructed background contains the anomalies which can be easily detected by the Reed-Xiaoli(RX) detector in the residual. Experimental results on two HSI datasets demonstrate the advantage of the proposed method.

This special section investigates the state-of-the-art in the analysis and processing of remotely sensed big data employing distributed computing architectures.

This article gives a survey of state-of-the-art methods for processing remotely sensed big data and thoroughly investigates existing parallel implementations on diverse popular high-performance computing platforms. The pros/cons of these approaches are discussed in terms of capability, scalability, reliability, and ease of use. Among existing distributed computing platforms, cloud computing is currently the most promising solution to efficient and scalable processing of remotely sensed big data due to its advanced capabilities for high-performance and service-oriented computing. We further provide an in-depth analysis of state-of-the-art cloud implementations that seek for exploiting the parallelism of distributed processing of remotely sensed big data. In particular, we study a series of scheduling algorithms (GSs) aimed at distributing the computation load across multiple cloud computing resources in an optimized manner. We conduct a thorough review of different GSs and reveal the significance of employing scheduling strategies to fully exploit parallelism during the remotely sensed big data processing flow. We present a case study on large-scale remote sensing datasets to evaluate the parallel and distributed approaches and algorithms. Evaluation results demonstrate the advanced capabilities of cloud computing in processing remotely sensed big data and the improvements in computational efficiency obtained by employing scheduling strategies.

Computational imaging for hyperspectral images (HSIs) is a hot topic in remote sensing and imaging systems. The dual-camera compressive hyperspectral imaging (DCCHI) system has been successfully designed and applied in hyperspectral imaging. However, the corresponding reconstruction algorithms are not well developed. In this paper, under the DCCHI framework, a new reconstruction algorithm is proposed based on the collaborative Tucker3 Tensor decomposition. In actual HSI, similar nonlocal patches always have similar spatial-spectral structures, and thus, these nonlocal patches can share the same spatial and spectral factors in Tucker decomposition. To characterize the similarities simultaneously, the Tucker3 decomposition is used to model the 4-order tensor formed by the similar cubic patches. To keep the spatial structures in the reconstructed HSI consistent with the panchromatic image's spatial structures, we force the spatial factor matrices and the core tensor in the Tucker3 decomposition of the HSI to be identical to the spatial factor matrices and core tensor of the panchromatic image's Tucker3 decomposition. In addition, a spectral quadratic variation constraint is introduced into the spectral factor to characterize the band smoothness. To solve the optimization problem, an alternating direction method of multipliers (ADMM)-based algorithm is designed and each variable is separately solved. Experimental results on a public data set and the remote sensing image demonstrate the advantage of the proposed method.

The large data volume and high algorithm complexity of hyperspectral image (HSI) problems have posed big challenges for efficient classification of massive HSI data repositories. Recently, cloud computing architectures have become more relevant to address the big computational challenges introduced in the HSI field. This article proposes an acceleration method for HSI classification that relies on scheduling metaheuristics to automatically and optimally distribute the workload of HSI applications across multiple computing resources on a cloud platform. By analyzing the procedure of a representative classification method, we first develop its distributed and parallel implementation based on the MapReduce mechanism on Apache Spark. The subtasks of the processing flow that can be processed in a distributed way are identified as divisible tasks. The optimal execution of this application on Spark is further formulated as a divisible scheduling framework that takes into account both task execution precedences and task divisibility when allocating the divisible and indivisible subtasks onto computing nodes. The formulated scheduling framework is an optimization procedure that searches for optimized task assignments and partition counts for divisible tasks. Two metaheuristic algorithms are developed to solve this divisible scheduling problem. The scheduling results provide an optimized solution to the automatic processing of HSI big data on clouds, improving the computational efficiency of HSI classification by exploring the parallelism during the parallel processing flow. Experimental results demonstrate that our scheduling-guided approach achieves remarkable speedups by facilitating the automatic processing of HSI classification on Spark, and is scalable to the increasing HSI data volume.

Classification is one of the major research fields in hyperspectral imagery. Due to the fact that neighboring pixels are more likely to share the same label, it is practical to use spatial information in hyperspectral image to achieve higher accuracy. On the other hand, however, spatial information also leads to higher computational complexity. This paper proposes an efficient implementation of a spatial-spectral kernel sparse representation for hyperspectral image classification base on the multi-GPU platform. The proposed implementation takes advantage of the capability of compute-unified device architecture (CUDA), such as shared memory, streams and peer-to-peer (P2P) transfer of data. In addition, an improvement of performance can be achieved by calculation reorganization and bandwidth usage optimization. Experimental results demonstrate that the proposed method achieves an up to 56.81X speedup in computation time while guaranteeing the classification accuracy.

The environmental change caused by time interval usually leads to the disturbance of endmember reflectance curve, which has an important influence on multi-temporal hyperspectral unmixing process. In this paper, a Bayesian unmixing model considering the spectral variability is proposed, in which a mixed prior distribution of the abundance is constructed. Different form the existing methods, the continuity of the abundance in time and spatial domain is considered simultaneously. In order to describe the smoothness of the abundance, a data-adaptive variance of probability distribution is designed based on the spatial local difference. Then combined with the priors of endmembers and spectral variability, the joint posterior distribution is set up and Markov chain Monte-Carlo(MCMC) algorithm is developed for posterior computation. Experiments on simulated and real datasets demonstrate the effectiveness of the proposed algorithm in terms of abundance estimation and endmember estimation.

Kernel methods, e.g., composite kernels (CKs) and spatial-spectral kernels (SSKs), have been demonstrated to be an effective way to exploit the spatial-spectral information nonlinearly for improving the classification performance of hyperspectral image (HSI). However, these methods are always conducted with square-shaped window or superpixel techniques. Both techniques are likely to misclassify the pixels that lie at the boundaries of class, and thus a small target is always smoothed away. To alleviate these problems, in this paper, we propose a novel patch-based low rank component induced spatial-spectral kernel method, termed LRCISSK, for HSI classification. First, the latent low-rank features of spectra in each cubic patch of HSI are reconstructed by a low rank matrix recovery (LRMR) technique, and then, to further explore more accurate spatial information, they are used to identify a homogeneous neighborhood for the target pixel (i.e., the centroid pixel) adaptively. Finally, the adaptively identified homogenous neighborhood which consists of the latent low-rank spectra is embedded into the spatial-spectral kernel framework. It can easily map the spectra into the nonlinearly complex manifolds and enable a classifier (e.g., support vector machine, SVM) to distinguish them effectively. Experimental results on three real HSI datasets validate that the proposed LRCISSK method can effectively explore the spatial-spectral information and deliver superior performance with at least 1.30% higher OA and 1.03% higher AA on average when compared to other state-of-the-art classifiers.

In this article, we propose a novel tensorial approach, namely, generalized tensor regression, for hyperspectral image classification. First, a simple and effective classifier, i.e., the ridge regression for multivariate labels, is extended to its tensorial version by taking advantages of tensorial representation. Then, the discrimination information of different modes is exploited to further strengthen the capacity of the model. Moreover, the model can be simplified and solved easily. Different from traditional tensorial methods, the proposed model can be utilized to capture not only the intrinsic structure of data in a physical sense but also the generalized relationship of data in a logical sense. Our proposed approach is shown to be effective for different classification purposes on a series of instantiations. Specifically, our experiment results with hyperspectral images collected by the airborne visible/infrared imaging spectrometer, the reflective optics spectrographic imaging system and the ITRES CASI-1500 demonstrate the effectiveness of the proposed approach as compared to other tensor-based classifiers and multiple kernel learning methods.

Hyperspectral (HS) super-resolution, which aims at enhancing the spatial resolution of hyperspectral images (HSIs), has recently attracted considerable attention. A common way of HS super-resolution is to fuse the HSI with a higher spatial-resolution multispectral image (MSI). Various approaches have been proposed to solve this problem by establishing the degradation model of low spatial-resolution HSIs and MSIs based on matrix factorization methods, e.g., unmixing and sparse representation. However, this category of approaches cannot well construct the relationship between the high-spatial-resolution (HR) HSI and MSI. In fact, since the HSI and the MSI capture the same scene, these two image sources must have common factors. In this paper, a nonlocal tensor decomposition model for hyperspectral and multispectral image fusion (HSI-MSI fusion) is proposed. First, the nonlocal similar patch tensors of the HSI are constructed according to the MSI for the purpose of calculating the smooth order of all the patches for clustering. Then, the relationship between the HR HSI and the MSI is explored through coupled tensor canonical polyadic (CP) decomposition. The fundamental idea of the proposed model is that the factor matrices in the CP decomposition of the HR HSI's nonlocal tensor can be shared with the matrices factorized by the MSI's nonlocal tensor. Alternating direction method of multipliers is used to solve the proposed model. Through this method, the spatial structure of the MSI can be successfully transferred to the HSI. Experimental results on three synthetic data sets and one real data set suggest that the proposed method substantially outperforms the existing state-of-the-art HSI-MSI fusion methods.

Hyperspectral image super-resolution addresses the problem of fusing a low-resolution hyperspectral image (LR-HSI) and a high-resolution multispectral image (HR-MSI) to produce a high-resolution hyperspectral image (HR-HSI). In this paper, we propose a novel fusion approach for hyperspectral image super-resolution by exploiting the specific properties of matrix decomposition, which consists of four main steps. First, an endmember extraction algorithm is used to extract an initial spectral matrix from LR-HSI. Then, with the initial spectral matrix, we estimate the spatial matrix, i.e., the spatial-contextual information, from the degraded observations of HR-HSI. Third, the spatial matrix is further utilized to estimate the spectral matrix from LR-HSI by solving a least squares (LS)-based problem. Finally, the target HR-HSI is constructed by combing the estimated spectral and spatial matrixes. In particular, two models are proposed to estimate the spatial matrix. One is a simple case that involves a LS-based problem, and the other is an elaborate case that consists of two fidelity terms and a spatial regularizer, where the spatial regularizer aiming to restrain the range of solutions is achieved by exploiting the superpixel-level low-rank characteristics of HR-HSI. Experiment results conducted on both synthetic and real data sets demonstrate the effectiveness of the proposed approach as compared to other hyperspectral image super-resolution methods.

In this article, the problem of automatically learning multiple parameters for kernel collaborative representation classification (KCRC) is considered. We investigate the KCRC and measure its generalization error via leave-one-out cross-validation (LOO-CV). By taking advantage of the specific properties of KCRC, a closed-form expression is derived for the outputs of LOO-CV. Then, a simple classification rule that provides probabilistic outputs is adopted, and thereby, an effective loss function that is an explicit function with respect to the parameters is proposed as the generalization error. The gradients of the loss function are calculated, and the parameters are learned by minimizing the loss function using a gradient-based optimization algorithm. Furthermore, the proposed approach makes it possible to solve the multiple kernel/feature learning problems of KCRC effectively. Experiment results on six data sets taken from different scenes demonstrate the effectiveness of the proposed approach.

Hyperspectral image (HSI) super-resolution is a hot topic in remote sensing and computer vision. Recently, tensor analysis has been proven to be an efficient technology for HSI image processing. However, the existing tensor-based methods of HSI super-resolution are not able to capture the high-order correlations in HSI. In this article, we propose to learn a high-order coupled tensor ring (TR) representation for HSI super-resolution. The proposed method first tensorizes the HSI to be estimated into a high-order tensor in which multiscale spatial structures and the original spectral structure are represented. Then, a coupled TR representation model is proposed to fuse the low-resolution HSI (LR-HSI) and high-resolution multispectral image (HR-MSI). In the proposed model, some latent core tensors in TR of the LR-HSI and the HR-MSI are shared, and we use the relationship between the spectral core tensors to reconstruct the HSI. In addition, the graph-Laplacian regularization is introduced to the spectral core tensors to preserve the spectral information. To enhance the robustness of the proposed model, Frobenius norm regularizations are introduced to the other core tensors. Experimental results on both synthetic and real data sets show that the proposed method achieves the state-of-the-art super-resolution performance.

As a low-rank tensor modeling, tensor tubal rank has been received more attention in hyperspectral image (HSI) recovery. However, the tubal rank is often approximated by tensor nuclear norm, which leads to modeling bias. To achieve an unbiased approximation and improve the model robustness, in this paper, a non-convex relaxation based HSI low-rank recovery model is proposed. And the model is solved efficiently by alternating direction method of multipliers (ADMM) optimization method. Two HSI datasets are employed to exhibit the superiority of the proposed model over the nuclear norm penalization method in terms of the accuracy and robustness.

Anomaly detection has become a hot topic in hyperspectral image (HSI) processing. Both spatial and spectral features have been proven to be very important for accurate and efficient hyperspectral anomaly detection. The traditional HIS anomaly detection algorithms usually reshape HSI to a matrix, which destroy spatial or spectral structure. In this paper, we propose a novel method of hyperspectral anomaly detection based on LOW RANK AND SPARSE TENSOR DECOMPOSITION (LRASTD). Taking into consideration that HSI data can be essentially regarded as a three-order tensor. HSI is modeled as a background tensor and a sparse anomalies tensor. A tensor nuclear norm is employed to constrain the core tensor, which be designed to characterize the low dimensional structure of the core tensor. Furthermore, a novel sparse tenor norm is proposed to constrain the anomaly targets. Experiments on both simulated and real hyperspectral data sets demonstrate the efficiency and effectiveness of the proposed method.

A new pan-sharpening method based multispectral (MS) image intensity modulation is proposed. Firstly, according to the spectral correlation between MS image and panchromatic (PAN) image, a new spectral-enhanced PAN (SPAN) image is obtained. Then, SPAN image is utilized to calculate the spectral and spatial difference coefficients. Finally, the fused high resolution MS (HRMS) image is generated by modulating the intensity of the upsampled MS image. The experimental results demonstrate that the proposed method performs well both on spatial details preservation and spectral distortion reduction.

Change Detection in Remote sensing image is, in essence, to detect the changes of ground features with regard to time from remote sensing perspective. It is usually realized by analyzing and processing multi-temporal high resolution images. Change Detection based on fully connected conditional random field not only improves the detection accuracy of remote sensing image, but also achieves better robustness. However, with the growth of high-resolution data volumes, this algorithm consumes a huge amount of time and computational resources, and therefore needs to be improved accordingly. Spark is an open-source distributed general- purpose cluster-computing framework. It has powerful memory computing and efficient task scheduling capabilities for complex iterative calculations. Based on Spark, this paper proposes a distributed and parallel method of change detection in remote sensing image based on Fully Connected Conditional Random Field that analyzes the data input form, and proposes a multi-temporal image reading strategy on cloud platforms. This method decomposes the algorithm flow, and performs distributed parallel processing on each stage and makes full use of the processing advantages of data locality to implement a reasonable intermediate data storage. Experimental results demonstrate that this parallel method achieves a promising speedup with high scalability, while guaranteeing remarkable detection accuracy.

Hyperspectral image (HSI) is often contaminated by mixed noise, which severely affects the visual quality and subsequent applications of the data. In this paper, HSI restoration based on low-rank recovery with a local neighborhood weighted spectral-spatial total variation (TV) model is proposed, which focuses on the preservation of spatial structure and spectral fidelity. The low-rank matrix model is adopted to exploit the spectral and spatial correlation information, and the l1-norm is used as a prior to remove the sparse noise. Furthermore, a local spatial neighborhood weighted spectral-spatial TV is utilized to jointly model the spectral-spatial prior information; specifically, the spectral and spatial differences are both considered in the TV term, and the weight is computed by considering the local neighborhood information in the spatial domain. Alternating direction method of multipliers optimization procedure is extended to solve the presented model. Experimental results demonstrate that the proposed method can remove the mixed noise, enhance the structural information simultaneously, and offer the best performance compared with several state-of-the-art HSI restoration methods.

The large amount of data produced by satellites and airborne remote sensing instruments has posed important challenges to efficient and scalable processing of remotely sensed data in the context of various applications. In this paper, we propose a new big data framework for processing massive amounts of remote sensing images on cloud computing platforms. In addition to taking advantage of the parallel processing abilities of cloud computing to cope with large-scale remote sensing data, this framework incorporates task scheduling strategy to further exploit the parallelism during the distributed processing stage. Using a computation- and data-intensive pan-sharpening method as a study case, the proposed approach starts by profiling a remote sensing application and characterizing it into a directed acyclic graph (DAG). With the obtained DAG representing the application, we further develop an optimization framework that incorporates the distributed computing mechanism and task scheduling strategy to minimize the total execution time. By determining an optimized solution of task partitioning and task assignments, high utilization of cloud computing resources and accordingly a significant speedup can be achieved for remote sensing data processing. Experimental results demonstrate that the proposed framework achieves promising results in terms of execution time as compared with the traditional (serial) processing approach. Our results also show that the proposed approach is scalable with regard to the increasing scale of remote sensing data.

This paper presents a hypserspectral image (HSI) super-resolution method which fuses a low-resolution hyperspectral image (LR-HSI) with a high-resolution multispectral image (HR-MSI) to get high-resolution HSI (HR-HSI). The proposed method first extracts the nonlocal similar patches to form a nonlocal patch tensor (NPT). A novel tensor-tensor product (t-product) based tensor sparse representation is proposed to model the extracted NPTs. Through the tensor sparse representation, both the spectral and spatial similarities between the nonlocal similar patches are well preserved. Then, the relationship between the HR-HSI and LR-HSI is built using t-product which allows us to design a unified objective function to incorporate the nonlocal similarity, tensor dictionary learning, and tensor sparse coding together. Finally, Alternating Direction Method of Multipliers (ADMM) is used to solve the optimization problem. Experimental results on three data sets and one real data set demonstrate that the proposed method substantially outperforms the existing state-of-the-art HSI super-resolution methods.

In this paper, we propose a novel cross total variation regularized unidirectional nonlocal low rank tensor approximation method for hyperspectral image denoising. It fully explores the spectral-spatial correlation and non-local self-similarity simultaneously in tensor case and points out that the nonlocal self-similarity is the most important for precisely restoring the HSI. Following the research line in [1], we propose to embed the cross total variation (CrTV) regularization into the unidirectional low rank tensor framework to alleviate the common consistency issue of pixels in overlapped regions. CrTV shows great power to explore the spatial-spectral correlation and has great ability to keep the fine spatial details and preserve the spectra in the course of HSI denoising. The final model can be effectively solved by the alternating direction methods of multipliers (ADMM). Experimental results on HSI data sets validate that the complementary priors (i.e., spatial-spectral correlation and non local self-similarity) really contribute to the performance and also illustrate the superiority of the proposed method when compared with other state-of-the-art denoising methods.

Pan-sharpening is a common image-fusion method. To improve the quality of fused images, a multilevel deep learning Pan-sharpening method is proposed in this paper. In the training phase, we introduce Coupled Sparse Denoising Autoencorder (CSDA) to reconstruct high-Resolution (HR) multispectral (MS) image from low-Resolution (LR) MS image and HR Panchromatic (Pan) image. CSDA has four networks including LM-HP network, HR-MS network, feature mapping network and fine-tuning network. The hidden features in LM-HP network and HR-MS network as well as the mapping function between the two features are learned through joint optimization. In LM-HP and HR-MS networks, the hidden features of image patch pairs are extracted by the sparse autoencoder. A sparse denoising autoencoder is used to build the nonlinear mapping between the extracted features. In the testing phase, the LR-MS and HR-Pan images patches are fed to the CSDA network to reconstruct the fused HR-MS image. The experimental results show that the proposed method is better than the traditional pans-sharpening methods.

Low-rank representation has been widely used as a powerful tool in hyperspectral image (HSI) recovery. The existing studies involving low-rank problems are commonly under the nuclear norm penalization. However, nuclear norm minimization tends to over-shrink the components of rank, which leads to modeling bias. In this paper, a new nonconvex penalty is introduced to obtain an unbiased low-rank approximation. In Addition, local spatial neighborhood weighted spectral-spatial total variation (TV) regularization is introduced to preserve spatial structural information. And sparse l1-norm is used as a constraint to sparse noise. Finally, a novel HSI non-convex low-rank relaxation restoration model is proposed. A number of experiments show that the proposed method can effectively remove the mixed-noise, and result in an unbiased estimate with better robustness.

In this paper, a high spatial resolution (HR) hyperspectral image is inferred from a low spatial resolution (LR) hyperspectral image and a HR panchromatic image by taking advantage of the sparse representation pansharpening (SRP) method. Different from the conventional SRP or joint SRP (JSRP) method, this paper proposes a multitask JSRP method for hyperspectral pansharpening, in order to improve the generalization performance of the model. First, multiple HR/LR dictionary pairs are generated by partitioning the multiple features of the panchromatic image and their corresponding downsampled LR versions into patches. Second, the patch-level sparse representation coefficients of the multiple LR hyperspectral image features are jointly estimated under the multiple LR dictionaries. Finally, the estimated sparse representation coefficients are utilized to reconstruct the HR patches under the original HR dictionary, and the desired HR hyperspectral image is obtained by aggregating the HR patches. Experimental results conducted on two hyperspectral scenes demonstrate the effectiveness of the proposed method.

Anomaly detection in hyperspectral images aims to separate the abnormal pixels from the background, and becomes an important application of hyperspectral data processing. Anomaly detection based on Low-Rank and Sparse Representation (LRASR) can detect abnormal pixels accurately. However, with the growth of the hyperspectral data volumes, this algorithm consumes a huge amount of time and computational resources, and needs to be improved accordingly. Spark is a distributed big data processing platform, and is applicable for complex iterative calculations, because of its powerful in-memory computation and efficient task scheduling. Based on Spark, this paper proposes a distributed and parallel LRASR (called DP-LRASR), which first segments hyperspectral images using narrow dependency of resilient distributed datasets, and afterwards, a parallel clustering algorithm is employed to improve the efficiency, remarkably. Experimental results demonstrate that DP-LRASR achieves a good speedup with high scalability, in the premise of remarkable detection accuracy.

Gas plume detection is a challenging task in the field of remote sensing. Due to sensor development, it is now possible to detect and track chemical gas plumes with hyperspectral video sequences (HVS). For the purpose of high detection accuracy, it is important to explore spectral characteristic, and take full advantages of spatial continuity and temporal consistency in HVS. However, the high computational complexity and the large amount of data limit its application in time-critical scenarios. In this paper, we propose a GPU parallel implementation of gas plume detection in HVS, which properly exploits shared memory and intrinsic concurrency of the CUDA blocks, as well as parallel workload assignment and resource allocation. The experimental results demonstrate the proposed GPU parallel method has a considerable acceleration factor while retaining the same detection accuracy compared with the serial and multicore version.

As the volume and value of hyperspectral remote sensing images increasing, a common hyperspectral imagery repository is urgently required. This paper presents a novel distributed content retrieval method for hyperspectral imagery repository based on cloud computing platform. The proposed method uses a software-as-a-service (SaaS) mode, which provides users with services such as hyperspectral image management, storage and retrieval through a web interface. In order to accelerate the acquisition of the spectral feature information needed in the retrieval process, we optimize the procedure of endmember extraction and the abundance estimation in a distributed way on Spark platform. The generated spectral features meta-data are stored in the MySQL database for imagery retrieval. Finally, we introduce the parallel implementation of distributed retrieval of hyperspectral images based on N-FINDR in details, and evaluate the effectiveness and stability of our proposed method through experiments on real datasets.

Recently, Parallel Intrusion Detection (PID) becomes very popular and its procedure of the parallel processing is called a PID application (PIDA). This PIDA can be regarded as a Bag-of-Tasks (BoT) application, consisting of multiple tasks that can be processed in parallel. Given multiple PIDAs (i.e., BoT applications) to be handled, when the private cloud has insufficiently available resources to afford all tasks, some tasks have to be outsourced to public clouds with resource-used costs. The key challenge here is how to schedule tasks on hybrid clouds to minimize makespan given a limited budget. This problem can be formulated as an Integer Programming model, which is generally NP-Hard. Accordingly, in this paper, we construct an Iterated Local Search (ILS) algorithm, which employs an effective heuristic to obtain the initial task sequence and utilizes an insertion-neighbourhood-based local search method to explore better task sequences with lower makespans. A swap-based perturbation operator is adopted to avoid local optimum. With the objective of improving the proposal’s efficiency without loss of any effectiveness, to calculate task sequences’ objectives, we construct a Fast Task Assignment (FTA) method by integrating an existing Task Assignment (TA) method with an acceleration mechanism designed through theoretical analysis. Accordingly, the proposed ILS is named FILS. Experimental results show that FILS outperforms the existing best algorithm for the considered problem, considerably and significantly. More importantly, compared with TA, FTA achieves a 2.42x speedup, which verifies that the acceleration mechanism employed by FTA is able to remarkably improve the efficiency. Finally, impacts of key factors are also evaluated and analyzed, exhaustively.

Hyperspectral video sequences (HVSs) are well suited for gas plume detection (GPD). The high spectral resolution allows the detection of chemical clouds even when they are optically thin. Processing this new type of video sequences is challenging and requires advanced image and video analysis algorithms. In this paper, we propose a novel method for GPD recorded in HVSs. Based on the assumption that the background is stationary and the gas plume is moving, the proposed method separates the background from the gas plume via a low-rank and sparse decomposition. Furthermore, taking into consideration that the gas plume is continuous in both spatial and temporal dimensions, we include total variation regularization in the constrained minimization problem, which we solve using the augmented Lagrangian multiplier method. After applying the above process to each extracted feature, a novel fusion strategy is proposed to combine the information into a final detection result. Experimental results using real data sets indicate that the proposed method achieves very promising GPD performance.

Anomaly detection plays an important role in remotely sensed hyperspectral image (HSI) processing. Recently, compressive sensing technology has been widely used in hyperspectral imaging. However, the reconstruction from compressive HSI and detection are commonly completed independently, which will reduce the processing's efficiency and accuracy. In this paper, we propose a framework for hyperspectral compressive sensing with anomaly detection which reconstruct the HSI and detect the anomalies simultaneously. In the proposed method, the HSI is composed of the background and anomaly parts in the tensor robust principal component analysis model. To characterize the low-dimensional structure of the background, a novel tensor nuclear norm is used to constrain the background tensor. As the anomaly part is formed by a few anomalous spectra, the anomaly part is assumed to be a tuber-wise sparse tensor. In addition, to enhance the separation of the background and anomaly, we minimize the sum of Mahalanobis distance of the background pixels. Experiments on four HSIs demonstrate that the proposed method outperforms several state-of-the-art methods on both reconstruction and anomaly detection accuracies.

Spectral unmixing is an important technique for remotely sensed hyperspectral image interpretation, of which the goal is to decompose the image into a set of pure spectral components (endmembers) and their abundance fractions in each pixel of the scene. Sparse-representation-based approaches have been widely studied for remotely sensed hyperspectral unmixing. A recent trend is to incorporate the spatial information to improve the spectral unmixing results. Those methods generally assume that the abundances of the pixels are piecewise smooth and fall into a homogeneous region occupied by the same endmembers and their corresponding fractional abundances. However, in real scenarios, abundances may vary abruptly from pixel to pixel. Therefore, the former assumption in most spatial models does not hold. To address this limitation, we propose a new strategy to preserve the spatial details in the abundance maps via a spatial discontinuity weight. Our experimental results, conducted with both simulated and real hyperspectral data sets, illustrate the good potential of our discontinuity-preserving strategy for sparse unmixing, which can greatly improve the abundance estimation results.

Hyperspectral image (HSI) classification is a popular issue in the domain of remote sensing. The fundamental challenges in HSI classification include small number of training samples, high dimensionality of the hyperspectral data and suitable spatial-spectral features. In this paper, we propose a novel multiple features fusion method for HSI classification based on extreme learning machines (ELM). We extract spectral feature via the principal component analysis (PCA), and extract spatial features via local binary pattern (LBP), Gabor feature and extended multiattribute profile (EMAP). Then we utilize probability voting to fuse the multiple features based on extreme learning machine model. Experiment on real HSI demonstrates that the proposed method is superior to some existing methods and it is suitable for small training sample size conditions.

This paper presents a new spatial-spectral classification method for hyperspectral images, which consists of three main techniques. Firstly, fully constrained least squares (FCLS) that is common in hyperspectral unmixing is investigated for hyperspectral image classification in kernel Hilbert space. Secondly, the spatial-spectral information of hyperspectral images is exploited to improve the classification performance of kernel-based FCLS (KFCLS) by taking advantage of a weighted H1 norm-based regularization term. Finally, the spatial and label information of training pixels is furthermore incorporated into KFCLS to deal with the scenarios dominated by limited training pixels. Experimental results on two real hyperspectral images demonstrate the effectiveness of the proposed method.

The hyperspectral image features wide coverage, high dimensional bands and a huge amount of data, which leads to time-consuming computation when processing hyperspectral data. Spark is a distributed big data processing framework, integrated in-memory computation. So Spark is suitable for complex iterative calculation. In order to classify massive hyperspectral data efficiently, the Spark version of the original Spatial Correlation Regularized Sparse Representation Classification (SCSRC) is proposed in this paper. In Distributed Parallel SCSRC (DP-SCSRC), firstly, adjacent hyperspectral image indexes are stored in the same partition of Spark's RDDs to preserve spatial correlation. Secondly, Joint Distributed Matrix (JDM) is created to reduce overhead data synchronization between computing nodes. Experimental results on real hyperspectral data demonstrate that DP-SCSRC achieves a remarkable speedup and is scalable with larger data size.

In this paper, a new classifier under Bayesian framework is proposed to explore homogeneous region based low rank representation in hidden field for classification of hyperspectral imagery (HSI). This classifier integrates low rank representation and superpixel segmentation simultaneously, in which the HSI data is assumed to be lying in a low rank subspace within each homogeneous region of an estimated hidden field. First, the HSI data is projected into the Principal Component space, then the first principal component image is segmented into hundreds of homogeneous regions. Following, the spectral-only supervised Bayesian classifier, i.e., Sparse Multinomial Logistic Regression (SMLR), is utilized for estimating the likelihood probabilities of testing samples, then spatial information is exploited by low rank representation within each superpixel in a hidden field which is approximated to the pre-estimated likelihood probabilities. The proposed model can be easily solved by alternating direction method of multipliers (ADMM). Experimental results on real hyperspectral data, i.e., AVIRIS Indian Pines and ROSIS University of Pavia, show that the proposed classifier outperforms other state-of-the-art classifiers in terms of quantitative assessment and visual effect.

With the widespread application of hyperspectral image, restoration has become an important branch of hyperspectral data processing. Although principal component analysis (PCA), total variation (TV) and soft-thresholding algorithm (PCATV-ST) could restore image effectively, a computing bottleneck may occur with the increase of hyperspectral data volumes. In order to solve the restoration problem effectively and accurately, we put forward a distributed parallel cloud implementation of PCATV-STbased on Spark (PCATV-ST_DP). The proposed method optimizes the way of data transmission and improves the shared memory space, as well as optimizes matrix multiplication. Finally, our experimental results, conducted on real hyperspectral datasets, reveal very high performance for the proposed distributed parallel method.

This letter presents a novel mixed noise (i.e., Gaussian, impulse, stripe noises, or dead lines) reduction method for hyperspectral image (HSI) by utilizing low-rank representation (LRR) on spectral difference image. The proposed method is based on the assumption that all spectra in the spectral difference space of HSI lie in the same low-rank subspace. The LRR on the spectral difference space was exploited by nuclear norm of difference image along the spectral dimension. It showed great potential in removing structured sparse noise (e.g., stripes or dead lines located at the same place of each band) and heavy Gaussian noise. To simultaneously solve the proposed model and reduce computational load, alternating direction method of multipliers was utilized to achieve robust reconstruction. The experimental results on both simulated and real HSI data sets validated that the proposed method outperformed many state-of-the-art methods in terms of quantitative assessment and visual quality.

Thanks to the fast development of sensors, it is now possible to acquire sequences of hyperspectral images. Those hyperspectral video sequences (HVS) are particularly suited for the detection and tracking of chemical gas plumes. In this paper, we present a novel gas plume detection method. It is based on the decomposition of the sequence into a low-rank and a sparse term, corresponding to the background and the plume, respectively, and incorporating temporal consistency. To introduce spatial continuity, a post processing is added using the Total Variation (TV) regularized model. Experimental results on real hyperspectral video sequences validate the effectiveness of the proposed method.

In this paper, a novel kernel low rank representation (KLRR) method for hyperspectral image classification is proposed. Firstly, we extract the global structure characteristics information of the hyperspectral image based on low rank representation (LRR), then use it as a prior to constrain the recovery coefficient matrix. In order to further improve the classification efficiency and deal with the linearly non-separable problems directly, we transformed the linear LRR classifier to a non-linear one by feature space transformation using the kernel trick. The proposed algorithm is solved by the ADMM algorithm effectively. Experimental results on real hyperspectral images demonstrate that the proposed method outperforms many state-of-the-art approaches.

This paper presents a region-based composite kernel framework for spatial-spectral hyperspectral image classification, referred as RCK, by exploiting the local similarities of both the spectral and spatial features via superpixel segmentation. The proposed framework consists of three steps. In the first step, the original hyperspectral image together with its spatial feature image are segmented into several nonoverlapping regions by using an efficient superpixel segmentation algorithm. In the second step, a mean filtering is performed within each region of both the spectral feature image and the spatial feature image to generate the corresponding region-based spectral and spatial features, respectively. In the final step, both the obtained region-based features are combined and incorporated into a probabilistic kernel collaboration classifier by taking advantage of a composite kernel framework. Experimental results on two real hyperspectral images demonstrate the improvement of RCK over the traditional composite kernel framework, as well as its effectiveness as compared to some popular spatial-spectral techniques.

Target detection plays an important role in the field of hyperspectral image (HSI) remote sensing. In this letter, a novel matched subspace detector based on low-rank regularized least squares (LRLS-MSD) is proposed for hyperspectral target detection. As pixels in an HSI have global correlation and can be represented in subspace, the low-rank regularization is introduced in the least squares model. An effective algorithm is presented to solve the problem. Then, the detection results are generated according to the generalized likelihood ratio test with statistical hypotheses. The experimental results suggest an advantage of the low-rank regularization over other classical target detection methods.

This paper presents a new approach for accurate spatial-spectral classification of hyperspectral images, which consists of three main steps. First, a pixelwise classifier, i.e., the probabilistic-kernel collaborative representation classification (PKCRC), is proposed to obtain a set of classification probability maps using the spectral information contained in the original data. This is achieved by means of a kernel extension based on collaborative representation (CR) classification. Then, an adaptive weighted graph (AWG)-based postprocessing model is utilized to include the spatial information by refining the obtained pixelwise probability maps. Furthermore, to deal with scenarios dominated by limited training samples, we modify the postprocessing model by fixing the probabilistic outputs of training samples to integrate the spatial and label information. The proposed approach is able to cover different analysis scenarios by means of a fully adaptive processing chain (based on three steps) for hyperspectral image classification. All the techniques that integrate the proposed approach have a closed-form analytic solution and are easy to be implemented and calculated, exhibiting potential benefits for hyperspectral image classification under different conditions. Specifically, the proposed method is experimentally evaluated using two real hyperspectral imagery data sets, exhibiting good classification performance even when the number of training samples available a priori is very limited.

A novel method for anomaly detection in hyperspectral images (HSIs) is proposed based on low-rank and sparse representation. The proposed method is based on the separation of the background and the anomalies in the observed data. Since each pixel in the background can be approximately represented by a background dictionary and the representation coefficients of all pixels form a low-rank matrix, a low-rank representation is used to model the background part. To better characterize each pixel's local representation, a sparsity-inducing regularization term is added to the representation coefficients. Moreover, a dictionary construction strategy is adopted to make the dictionary more stable and discriminative. Then, the anomalies are determined by the response of the residual matrix. An important advantage of the proposed algorithm is that it combines the global and local structure in the HSI. Experimental results have been conducted using both simulated and real data sets. These experiments indicate that our algorithm achieves very promising anomaly detection performance.

Principal component analysis (PCA) is an important method for feature extraction of hyperspectral remote sensing image. With the development of hyperspectral sensors, the magnitude of hyperspectral data grows quickly, and it is a challenging task to efficiently reduce the data dimension and compress massive data volumes in hyperspectral imaging. In this paper, a distributed parallel optimization of PCA algorithm (PCA_DP) is presented on cloud computing architecture. The realization of the proposed method using Apache Hadoop and MapReduce model is described and evaluated. The experiments conducted on real hyperspectral images of different sizes, demonstrate significant acceleration factor of PCA_DP. It is efficient for massive hyperspectral data processing.

This paper presents a novel method for anomaly detection in hyperspectral image(HSI) based on low-rank representation. In the observed HSI, the anomalies can be separated from the background. Since each pixel in the background can be approximately represented by a background dictionary, and the representation coefficients of the background pixels are correlative, a low-rank representation model is used to model the background part. Besides, to gain robust representation coefficient, the sum-to-one constraint is added. The advantage of the proposed low-rank representation sum-to-one (LRRSTO) method is that it makes use of the global correlation of the background and strength the robustness of the representation. Experiments results have been conducted using both simulated and real data sets. These experiments indicated that our algorithm achieves very promising performance.

In this paper, we propose a new method for hyperspectral image (HSI) classification using multi-layer superpixel graph and loopy belief propagation. A merging algorithm using graph based representation of image is applied to generate multi-scale superpixels in hyperspectral image at first. Then, we build a multi-layer superpixel graph and use loopy belief propagation to transmit messages between the superpixels and compute beliefs at each superpixel in our multi-layer graph for HSI classification. Experimental results with real hyperspectral data set demonstrate that our proposed method provides good performance and is competitive with some of the best available spectral-spatial methods for hyperspectral image classification.

In this letter, we present an efficient parallel implementation of composite kernels in support vector machines (SVMs) for hyperspectral image (HSI) classification. Our implementation makes effective use of commodity graphics processing units (GPUs). Specifically, we port the calculation of composite kernels to GPUs, perform intensive computations based on NVidia's compute unified device architecture, and execute the rest of the operations related with control and small data calculations in the CPU. Our experimental results, conducted using real hyperspectral data sets and NVidia GPU platforms, indicate significant improvements in terms of computational effectiveness, achieving near-real-time performance of spatial-spectral HSI classification for the first time in the literature.

In this letter, a real-time implementation of the logistic regression via variable splitting and augmented Lagrangian (LORSAL) algorithm for sparse multinomial logistic regression is presented on commodity graphics processing units (GPUs) using Nvidia's compute unified device architecture. The proposed parallel method properly exploits the GPU architecture at the low level, including its shared memory, and takes full advantage of the computational power of GPUs to achieve real-time classification performance of hyperspectral images for the first time in the hyperspectral imaging literature. Our experimental results reveal remarkable acceleration factors and real-time performance, while retaining exactly the same classification accuracy with regard to the serial and multicore versions of the classifier.

This paper presents a new approach for hyperspectral image classification exploiting spectral-spatial information. Under the maximum a posteriori framework, we propose a supervised classification model which includes a spectral data fidelity term and a spatially adaptive Markov random field (MRF) prior in the hidden field. The data fidelity term adopted in this paper is learned from the sparse multinomial logistic regression (SMLR) classifier, while the spatially adaptive MRF prior is modeled by a spatially adaptive total variation (SpATV) regularization to enforce a spatially smooth classifier. To further improve the classification accuracy, the true labels of training samples are fixed as an additional constraint in the proposed model. Thus, our model takes full advantage of exploiting the spatial and contextual information present in the hyperspectral image. An efficient hyperspectral image classification algorithm, named SMLR-SpATV, is then developed to solve the final proposed model using the alternating direction method of multipliers. Experimental results on real hyperspectral data sets demonstrate that the proposed approach outperforms many state-of-the-art methods in terms of the overall accuracy, average accuracy, and kappa (k) statistic.

This paper proposes a new hyperspectral images superresolution (HSI-SR) method based on compressive sensing (CS) theory, spatial sparsity and spectral similarity prior. First, according to sparsity and incoherence of CS theory, we propose a new dictionary learning method, ensuring that the learned dictionary not only has less dimensionality to speed up the sparse decomposition, but also satisfies sparsity well. Then, we introduce the spatial sparsity and spectral similarity regularizations into HSI-SR model, which can recover the spatial information effectively and preserve the spectral information well. The experimental results show the proposed method outperforms other well-known methods in terms of both objective measurements and visual evaluation.

This paper presents a novel method to apply homogeneous areas priors adaptively for hyperspectral image classification. Firstly, support vector machine algorithm is utilized to obtain the posterior probability distributions by training the spectral information of the samples. Then, the homogeneous areas generated from the watershed segmentation results are used as new spatial priors. By using Markov Random Field model, we can integrate the spectral information and spatial information which includes the homogeneous areas priors in a unified framework. Compared with neighborhood-generated Markov random field, the adaptive priors strengthen to enforce the segmentation results in homogeneous areas of the neighborhood belong to the same class. Finally, the maximum a posterior segmentation is computed by min-cut based optimization algorithm.

This letter presents a postprocessing algorithm for a kernel sparse representation (KSR)-based hyperspectral image classifier, which is based on the integration of spatial and spectral information. A pixelwise KSR is first used to find the sparse coefficient vectors of the hyperspectral image. Then, a sparsity concentration index (SCI) rule-guided semilocal spatial graph regularization (SSG), called SSG+SCI, is proposed to determine refined sparse coefficient vectors that promote spatial continuity within each class. Finally, these refined coefficient vectors are used to obtain the final classification map. Compared with previous approaches based on similar spatial-spectral postprocessing strategies, SSG+SCI clearly outperforms their results in terms of accuracy and the number of training samples, as it is demonstrated with two real hyperspectral images.

In this paper, we propose a new model for hyperspectral image classification using spectral-spatial information. The main contributions of our paper are that we exploit the posterior distribution from both spectral and spatial information in the original hyperspectral data. The association potential in our model is a sparse multinomial logistic regression (SMLR) classifier and the interaction potential is a spatial-relevant total variation (TV) constraint upon the posterior distribution itself which encourages neighboring pixels to belong to the same class. The proposed model is solved by the alternating direction method of multipliers (ADMM); we enhance the spatial smoothness by expanding the spatial information from the fixed labeled samples to the whole data to further improve the classification accuracy. Experimental results with real hyperspectral data set validate that our proposed approach provides good performance when compared with other state-of-the-art methods.

Hyperspectral unmixing is a typical problem of blind source separation, which can be solved by nonnegative matrix factorization (NMF). Sparsity based NMF will increase the efficiency of unmixing, but its computational complexity limits the possibility of utilizing it in time-critical applications. In this paper, method of parallel hyperspectral unmixing based on sparsity constrained nonnegative matrix factorization on Graphics Processing Units (CSNMF-GPU) is investigated and compared in terms of both accuracy and speed. The realization of the proposed method using Compute Unified Device Architecture (CUDA) on GPU are described and evaluated. The experimental results comparing with the serial implementations based on both simulated and real hyperspectral data demonstrate the effectiveness of the proposed parallel optimization approach.

Sparsity is an intrinsic property of hyperspectral images, which means that the collected pixels can be represented by a part of materials. In this paper, a new sparsity based method for hyperspectral unmixing is proposed, referred to as the constrained sparse nonnegative matrix factorization (CSNMF). First, a novel sparse term which is explored to measure the sparsity of hyperspectral images is introduced to restrict the abundances. Second, minimum distance constraint which is convex is applied to restrict the endmembers. Then the alternating direction method of multipliers (ADMM) is used to solve the proposed CSNMF. The experimental results based on both synthetic mixtures and a real image scene demonstrate the effectiveness of the proposed approach.