Knowledge distillation provides an effective paradigm for developing lightweight remote sensing object detectors. However, existing methods predominantly concentrate on design of foreground feature masks while neglecting the suppression of small-scale targets by dominant large-scale counterparts in cluttered backgrounds during the distillation, which severely degrades detection robustness. To address this, we propose scale-conscious knowledge distillation. Multi-scale feature distillation in this framework decouples the coupled feature in conventional methods into hierarchical multi-scale representation spaces, enabling the student model to capture cross-granularity features from micro-details to macro-structures through differentiated receptive fields. Scale-adaptive output distillation innovatively introduces a dynamic weighting mechanism based on the normalized object area, effectively alleviating the gradient vanishing issue for small objects. Experimental results show that our method consistently improves the performance of student models with different architectures on both the DOTA and DIOR datasets. Source codes are available at https://github.com/RQ-W/SCD.git.

Weakly-supervised object detection (WSOD) learns detectors with only image-level classification annotations. Without precise instance-level labels, most previous WSOD methods in remote sensing images (RSIs) select the highest-scoring proposals as the final detection results, which are confronted by two major challenges: (1) instances with small scale or rare poses are easily neglected; (2) optimizing network by the top-scoring region inevitably overlooks many valuable candidate proposals. To mitigate the above-mentioned challenges, we propose a data-driven bidirectional spatial-adaptive network (BSANet). It contains a forward-reverse spatial dropout (FRSD) module to reduce instance ambiguity induced from extreme scales and poses, as well as crowded scene, and to better excavate the entire instances. From attention learning perspective, the proposed FRSD is conceptually similar to a data-driven hard attention mechanism, which adaptively samples and reconstructs the spatially related regions for mining more latent feature responses. Meanwhile, our FRSD effectively alleviates the inherent problem that non-parametric hard attention learning fashion cannot adapt to different datasets. In addition, we build a soft attention branch to simultaneously model soft pixel-level and hard region-level attention information for exploring the complementary benefit between soft and hard attention learning. We evaluate our BSANet on the challenging NWPU VHR-10.v2 and DIOR datasets. Experimental results demonstrate that our method sets a new state-of-the-art.

Hyperspectral image (HSI) restoration tasks including super-resolution, denoising, and inpainting, present significant challenges due to intrinsic spectral-spatial coupling and limited training data availability. Recent advances in RGB image restoration demonstrate that models pretrained on large-scale datasets acquire exceptional generalization capabilities, suggesting potential cross-modal knowledge transfer solutions for HSI recovery. However, existing approaches exhibit two critical limitations: 1) prohibitive computational costs from mandatory fine-tuning procedures, and 2) inadequate cross-modal adaptation causing spectral distortions. To address these challenges, we propose a Two-Stage Cross-Modal Decoupling Network (CMDN) achieves spectral-faithful HSI restoration without fine-tuning the pretrained RGB prior; instead, we perform unsupervised test-time learning only on a lightweight spectral rectifier for sample-specific spectral calibration. Our methodology introduces two fundamental innovations: First, we develop a theoretically grounded framework using Singular Value Decomposition (SVD) to decouple HSIs into orthogonal spatial coefficients and spectral bases. This decomposition enables strategic reconfiguration of spatial coefficients into pseudo-RGB formats through band reorganization, facilitating direct deployment of frozen RGB-pretrained models for spatial textures recovery while preserving spectral integrity. Second, we propose a Physics Motivated Spectral Rectifier (PMSR) that dynamically adjusts spectral reconstruction weights using spatial gradient priors, correcting spectral deviations through physics-consistent optimization rather than explicit error modeling, thereby achieving superior spectral fidelity. Comprehensive experiments confirm our method's superiority in both spatial reconstruction accuracy and spectral consistency over state-of-the-art techniques. Code is available at: https://github.com/QYo-Liu/CMDN.

Hyperspectral and multispectral image fusion (HMF) enhances spatial-spectral quality by fusing low-resolution hyperspectral images (LR-HSI) with high-resolution multispectral images (HR-MSI). Although recent fusion methods have shown promise in preserving the multi-mode structure of high-dimensional data, existing fusion methods still face some challenges. For tensor-based approaches, conventional mode-wise decomposition, such as order-3 CP or Tucker decomposition, may disrupt intrinsic spatial consistency. Furthermore, although deep learning exhibits powerful feature representation ability, existing deep fusion methods either rely on 'data-driven' deep fusion networks remain insufficiently interpretability with large training data. To address these issues, a novel Self-Expressive High-Order Tensor Unrolling Network (SHOTUN) is proposed for unsupervised HSI-MSI fusion. Within the sparse core tensor decomposition framework, we introduce the intrinsic self-expressive relationships among overlapping image patches as a form of high-order mode representation to preserve spatial structure of the fusion model. During optimization, we adopt an alternative optimizing strategy and design dedicated modules for each sub-problem, yielding an interpretable end-to-end training pipeline. Furthermore, to improve generalization across different sensors, we introduce a pre-training strategy into the unsupervised training for the more accurate estimation of unknown degraded parameters. Extensive experimental results on simulated and real datasets demonstrate the effectiveness of our proposed method. The source code is publicly available at https://github.com/Shawn-H-Wang/SHOTUN.

Hyperspectral images (HSIs) and multispectral images (MSIs) fusion is a hot topic in the remote sensing society. A high-resolution HSI (HR-HSI) can be obtained by fusing a low-resolution HSI (LR-HSI) and a high-resolution MSI (HR-MSI) or RGB image. However, most deep learning-based methods require a large amount of HR-HSIs for supervised training, which is very rare in practice. In this paper, we propose a coupled diffusion posterior sampling (CDPS) method for HSI and MSI fusion in which the HR-HSIs are no longer required in the training process. Because the LR-HSI contains the spectral information and HR-MSI contains the spatial information of the captured scene, we design an unsupervised strategy that learns the required diffusion priors directly and solely from the input test image pair (the LR-HSI and HR-MSI themselves). Then, a coupled diffusion posterior sampling method is proposed to introduce the two priors in the diffusion posterior sampling which leverages the observed LR-HSI and HR-MSI as fidelity terms. Experimental results demonstrate that the proposed method outperforms other state-of-the-art unsupervised HSI and MSI fusion methods. Additionally, this method utilizes smaller networks that are simpler and easier to train without other data.

Task scheduling in mobile edge computing (MEC) is critical for managing latency and energy consumption, especially under user mobility, which poses additional challenges to scheduling decisions. This paper studies the mobility-aware two-stage task scheduling (MTTS) problem in MEC systems to minimize energy consumption at the mobile edge under task deadline constraints. First, the MTTS problem is formulated as a two-stage optimization model consisting of two interrelated sub-problems, associated with task offloading and result downloading, respectively. Then, a dual-population-based evolutionary computation (DORA) framework is proposed that can solve MTTS-type two-stage problems by incorporating a wide range of swarm intelligence algorithms (SIAs). The proposed DORA framework employs two populations that run in parallel, each dedicated to solving one of the two interrelated sub-problems, to enable effective exploration of the solution space and improve computational efficiency. The evolutionary process of each population in DORA consists of three fundamental algorithmic components: mapping, evaluation, and updating. In particular, the mapping component establishes the link between individual space in SIAs and solution space in the MTTS problem, with the critical parameter derived through rigorous theoretical analysis. Furthermore, the inter population collaboration is realized through an asynchronous solution transfer mechanism from the stage-1 population to the stage-2 population, guiding the search toward high-quality final scheduling solutions. Extensive experiments are conducted on a real-world mobile device trajectory dataset, incorporating four well-established SIAs to demonstrate the applicability and effectiveness of DORA.

Hyperspectral and multispectral image (HSI-MSI) fusion involves combining a low-resolution hyperspectral image (LR-HSI) with a high-resolution multispectral image (HR-MSI) to generate a high-resolution hyperspectral image (HR-HSI). Most deep learning-based methods for HSIMSI fusion rely on large amounts of hyperspectral data for supervised training, which is often scarce in practical applications. In this paper, we propose a self-learning Adaptive Residual Guided Subspace Diffusion Model (ARGS-Diff), which only utilizes the observed images without any extra training data. Specifically, as the LR-HSI contains spectral information and the HR-MSI contains spatial information, we design two lightweight spectral and spatial diffusion models to separately learn the spectral and spatial distributions from them. Then, we use these two models to reconstruct HR-HSI from two low-dimensional components, i.e, the spectral basis and the reduced coefficient, during the reverse diffusion process. Furthermore, we introduce an Adaptive Residual Guided Module (ARGM), which refines the two components through a residual guided function at each sampling step, thereby stabilizing the sampling process. Extensive experimental results demonstrate that ARGS-Diff outperforms existing state-of-the-art methods in terms of both performance and computational efficiency in the field of HSI-MSI fusion.

Quantization plays a crucial role in accelerating the inference of large-scale models, and rotational matrices have been shown to effectively improve quantization performance by smoothing outliers. However, end-to-end fine-tuning of rotational optimization algorithms incurs high computational costs and is prone to overfitting. To address this challenge, we propose an efficient distribution-aware rotational calibration method, DartQuant, which reduces the complexity of rotational optimization by constraining the distribution of the activations after rotation. This approach also effectively reduces reliance on task-specific losses, thereby mitigating the risk of overfitting. Additionally, we introduce the QR-Orth optimization scheme, which replaces expensive alternating optimization with a more efficient solution. In a variety of model quantization experiments, DartQuant demonstrates superior performance. Compared to existing methods, it achieves 47$\times$ acceleration and 10$\times$ memory savings for rotational optimization on a 70B model. Furthermore, it is the first to successfully complete rotational calibration for a 70B model on a single 3090 GPU, making quantization of large language models feasible in resource-constrained environments.

With the advancement of object detection technology, few-shot object detection (FSOD) has become a research hotspot. Existing methods face two major challenges: base models have limited generalization to unseen categories, especially with limited few-shot data, where the shared feature representation fails to meet the distinct needs of classification and regression tasks; FSOD is susceptible to overfitting during training. To address these issues, this paper proposes a Multi-Task Decoupled Method (MTDM), which enhances the model’s generalization to new categories by separating the feature extraction processes for different tasks. Additionally, a dynamic adjustment strategy is adopted, which adaptively modifies the IOU threshold and loss function parameters based on variations in the training data, reducing the risk of overfitting and maximizing the utilization of limited data resources. Experimental results show that the proposed hybrid model performs well on multiple few-shot datasets, effectively overcoming the challenges posed by limited annotated data.

Knowledge distillation, which intends to transfer the expertise from a complex teacher model to a concise student model, has achieved impressive success in object detection. However, many existing distillation methods are designed for object detection tasks on natural images and perform poorly on more challenging remote sensing images. In this study, we first attribute the deficiencies of knowledge distillation in remote sensing object detection to two core reasons: 1) lack of relation distillation among different instances and pixels and 2) significant differences in feature magnitude between the teacher-student pair. Then, we propose a dual prediction-guided knowledge distillation framework, which includes relation distillation and output distillation to address two issues, respectively. Prediction-guided relation distillation (PGRD) is proposed to capture the knowledge of global relation at both instance-wise and pixel-wise, then allowing students to better understand and depict the features distribution across different categories. Prediction-guided output distillation (PGOD) is proposed to mitigate the impact of feature magnitude inconsistencies on distillation with classification and location knowledge, then allowing students to directly capture task-relevant information. Finally, experimental results have indicated the consistent effectiveness of our method across anchor-based two-stage, one-stage, and anchor-free detectors with 11 comparison knowledge distillation methods on three remote sensing detection datasets. Source codes are available athttps://github.com/RQ-W/DPGD.git.

Hyperspectral target detection (HTD) can provide detailed information about the objects and materials within a scene and holds significant importance in remote sensing image analysis. Traditional HTD methods often suffer from low detection accuracy when applied to hyperspectral images (HSIs) with low spatial resolution, where the target only occupies a few pixels. This can be addressed by exploiting the higher spatial resolution of multispectral images (MSIs). In addition, many cloud-based HTD methods, which rely on the distributed processing capability of cloud computing to cope with large-scale datasets, may result in long transmission delays that cannot meet real-time requirements. This article suggests a cloud-edge collaborative HTD approach based on the fusion of remotely sensed HSIs and MSIs. We first introduce an HTD algorithm that employs low-rank matrix decomposition and hierarchical constraint energy minimization (hCEM) to fuse a low-resolution HSI (LR-HIS) and a high-resolution MSI (HR-MSI). Aiming at a continuous shooting scenario, we further present a cloud-edge implementation of the HTD algorithm through the collaboration of a cloud cluster and edge servers deployed close to data acquisition devices. The overall processing flow of remotely sensed data fusion in the cloud-edge environment is formulated as a flowshop scheduling-like optimization problem. We develop a co-optimization scheduling algorithm to explore the best resource allocation solutions to the formulated problem. Experimental results on both general-purpose and real-world datasets show that the newly proposed HTD algorithm leads to significant improvements in detection accuracy over traditional methods, and the cloud-edge collaborative approach further enhances computational efficiency.

Currently, hyperspectral and multispectral image fusion methods based on local and global feature learning (e.g., CNN and Transformer) have achieved promising results. However, as the core part of transformer, the computational cost of the self-attention is quadratic with the image size, which severely limits its practical application. In this paper, we propose a spatial-spectral cross mamba network (SSCM) for hyperspectral and multispectral image fusion. By using the mamba structure, our model is able to obtain long-range spatial-spectral information with less computational complexity in comparison with the transformer structure. Specifically, we introduce a spatial-spectral cross mamba block to facilitate the interaction between hyperspectral and multispectral features, effectively enhancing the spatial-spectral feature representation ability of the network. In addition, a cross-scale spatial-spectral learning module based on the U-shaped structure is proposed to effectively extract the long-range high-frequency feature information at different scales. Extensive experimental results demonstrate that our method achieves comparable performance in comparison with some state-of-the-art image fusion methods.

Hyperspectral image (HSI) target detection is an advanced technology within the remote sensing society, widely applied in both civilian and military domains. However, subpixel target detection poses a significantly challenging problem in HSI processing. Existing methods for HSI target detection mostly rely on constructing detectors based on linear mixing model (LMM). Nevertheless, due to multiple scattering, LMM fails to capture the nonlinear features and extensive interactions among various materials commonly present in real HSIs, potentially resulting in poor performance. To address this issue, this work proposes a novel method for HTD based on bilinear mixing model (BMM) in HSI. Specifically, we designed a spectral decomposition network to separate the background and target. To further extract and combine spectral and spatial information of the image for more accurate learning of target and background distributions, a deep network model based on a multiscale feature extraction module is proposed. Finally, to enhance the discrimination between targets and backgrounds, a customized CEM loss is formulated by minimizing the response of the background and maximizing the response of the target. Experimental comparisons on six real HSI datasets validate the superiority of the proposed detector.

Low-rank representation is a popularly used method in remote sensing and image processing systems. Existing low-rank representation-based algorithms often neglect the varying sensitivities of different pixels to the dictionary, which may lead to inaccurate detection. Also, the dense iterative computations involved in these algorithms could induce high computation overheads. This article proposes an efficient low-rank representation method for hyperspectral anomaly detection (HAD) on a central processing unit (CPU)-field programmable gate array (FPGA) hybrid computing platform. The proposed method starts with segmenting the hyperspectral image into superpixels based on the selected number of dictionary categories. The image is divided into center and edge parts using a sliding window along the segmentation boundaries. Then, we use a low-rank and sparse representation (LRASR) to process center pixels that are well-adapted to the dictionary, and a weighted low-rank and collaborative representation model to process edge pixels that are less adaptive to the dictionary. The aforementioned two representation models are integrated to obtain the final reconstructed image. Moreover, we propose the Nesterov acceleration method by incorporating an adaptive step size into the linearized alternating direction method with adaptive penalty (LADMAP). At the hardware level, the acceleration method is deployed on the CPU-FPGA platform to speed up the proposed HAD flow. Experimental results demonstrate that the proposed method outperforms most existing methods in detection accuracy and achieves significant improvement in computational efficiency.

Most remote sensing datasets are annotated with horizontal bounding boxes (HBBs), which conflicts with mainstream oriented object detection methods that require oriented bounding boxes (OBBs). Horizontal box supervised oriented object detection has emerged as a promising solution, but existing methods suffer from two key limitations. First, they apply image-level geometric transformations for consistency learning, which binds object orientation to the global image and limits the model's ability to learn instance-specific orientation features. Second, they rely on data augmentation for orientation awareness while still using conventional horizontal convolutional neural networks (CNNs) for regression, failing to extract orientation-sensitive features effectively. To address these issues, we propose the Instance-Level Orientation Information Enhanced Detector (ILOEDet), which integrates the Instance-Aware Rotated Convolution Module (IARCM) and an Instance-Level Flip Consistency (IFC) mechanism to improve orientation sensitivity. Specifically, IARCM leverages classification and center-ness scores to select high-quality instances and their predicted angles, guiding a rotated convolution operation to embed instance-level orientation information into the feature maps. Meanwhile, IFC introduces a self-supervised branch that flips individual object instances to decouple their orientation from the image background, enforcing instance-level consistency constraints for more robust orientation learning. Experiments on the DOTA, HRSC2016, and DIOR-R datasets demonstrate the effectiveness of our approach.

Fine-grained object detection (FGOD) fundamentally comprises two primary tasks: object detection and fine-grained classification. In natural scenes, most FGOD methods benefit from higher instance resolution and fewer environmental variation, attributing more commonly associated with the latter task. In this paper, we propose a unified paradigm named Detector with Classifier2 (DC2), which provides a holistic paradigm by explicitly considering the end-to-end integration of object detection and fine-grained classification tasks, rather than prioritizing one aspect. Initially, our detection sub-network is restricted to only determining whether the proposal is a coarse-category and does not delve into the specific sub-categories. Moreover, in order to reduce redundant pixel-level calculation, we propose an instance-level feature enhancement (IFE) module to model the semantic similarities among proposals, which poses great potential for locating more instances in remote sensing images (RSIs). After obtaining the coarse detection predictions, we further construct a classification sub-network, which is built on top of the former branch to determine the specific sub-categories of the aforementioned predictions. Importantly, the detection network is performed on the complete image, while the classification network conducts secondary modeling for the detected regions. These operations can be denoted as the global contextual information and local intrinsic cues extractions for each instance. Therefore, we propose a multi-stream feature aggregation (MSFA) module to integrate global-stream semantic information and local-stream discriminative cues. Our whole DC2 network follows an end-to-end learning fashion, which effectively excavates the internal correlation between detection and fine-grained classification networks. We evaluate the performance of our DC2 network on two benchmarks SAT-MTB and HRSC2016 datasets. Importantly, our method achieves the new state-of-the-art results compared with recent works (approximately 7% mAP gains on SAT-MTB) and improves baseline by a significant margin (43.2% $v.s.~36.7$ %) without any complicated post-processing strategies. Source codes of the proposed methods are available at https://github.com/zhengshangdong/DC2.

Hyperspectral image (HSI) and multispectral image (MSI) fusion aims to generate high spectral and spatial resolution hyperspectral image (HR-HSI) by fusing high-resolution multispectral image (HR-MSI) and low-resolution hyperspectral image (LR-HSI). However, existing fusion methods encounter challenges such as unknown degradation parameters, and incomplete exploitation of the correlation between high-dimensional structures and deep image features. To overcome these issues, in this article, an unsupervised blind fusion method for LR-HSI and HR-MSI based on Tucker decomposition and spatial-spectral manifold learning (DTDNML) is proposed. We design a novel deep Tucker decomposition network that maps LR-HSI and HR-MSI into a consistent feature space, achieving reconstruction through decoders with shared parameters. To better exploit and fuse spatial-spectral features in the data, we design a core tensor fusion network (CTFN) that incorporates a spatial-spectral attention mechanism for aligning and fusing features at different scales. Furthermore, to enhance the capacity to capture global information, a Laplacian-based spatial-spectral manifold constraint is introduced in shared-decoders. Sufficient experiments have validated that this method enhances the accuracy and efficiency of hyperspectral and multispectral fusion on different remote sensing datasets. The source code is available at https://github.com/Shawn-H-Wang/DTDNML.

In recent years, due to the rapid development of deep learning, autoencoders (AEs) has become a popular technique for hyperspectral unmixing. However, most of the AE-based unmixing networks only use one network to learn the features, resulting in the imbalance of the endmembers and abundances. Furthermore, the spectral fidelity of the loss function is usually measured in a linear manner by omitting the non-linearity in spectral mixing. Therefore, we propose a two-stream autoencoder blind unmixing network combined with the non-linear Hapke unmixing model. Specifically, we construct a network for solving endmembers then transfer the estimated weight matrices to another designed abundance network. Notably, we introduce a new loss function consisting of three parts: network error estimated using spectral angle distance, reconstruction error based on the non-linear Hapke model, and TV regularization of the endmembers. Experimental results show that our method has certain advantages in endmember extraction and abundance estimation compared to other excellent unmixing methods.

The detection of oriented object detection in remote sensing images remains a daunting challenge due to their complex backgrounds, various sizes, and especially arbitrary orientations. However, most existing methods only model the structural features of images in the spatial domain, while the horizontal convolution kernels limit the model’s ability to perceive object direction information. Furthermore, frequency features contain rich information about scale, texture and angle, which can be a good complement to the spatial features. Inspired by this, we propose a multiscale spatial-frequency domain network (MSFN) to utilize spatial-frequency information for oriented object detection, which can be integrated into any CNN architectures seamlessly and perform end-toend training easily. Besides, we design a channel alignment feature fusion module (CA-FFM) to address the problem of fusing low-level texture and high-level semantic features with a significant difference in channel dimension. Experimental results on HRSC2016 and SSDD dataset demonstrate the effectiveness of the proposed method.

Despite the wide application of deep learning in hyperspectral classification, variations in data collection conditions can lead to domain shift between the training and testing datasets. Traditional hyperspectral classification methods are adversely affected by these distribution differences, resulting in poor generalization performance on the testing set. To overcome this challenge, we present an optimized unsupervised domain adaptation approach based on causal invariance. Our method assumes a causal relationship to reflect the effects of changes in class information and domain information on samples. Based on this causal relationship, we construct a network to separate class-related and domain-related features. To further reduce the negative transfer caused by distribution differences, our model introduces intra-class feature consistency. As a result, our method improves the performance of the model on the target domain. Experimental results on two public hyperspectral datasets demonstrate the superior effectiveness of our method.

High-resolution remote sensing object detection is a research field with significant application value and challenges. However, existing methods cannot directly predict or produce a large number of redundant bounding boxes. They perform poorly in the face of issues such as multi-scale, dense small objects, and complex backgrounds in high-resolution remote sensing images. Therefore, a real-time end-to-end high-resolution remote sensing object detection method based on RT-DETR (CDE-DETR) is proposed. Through introducing cascaded group attention, we propose CGA-IFI for intra-scale feature interaction. The DRB-CFFM is designed with a dilated reparam block to facilitate cross-scale feature interaction. Furthermore, we enhance the bounding box regression loss function with EIoU. Experimental results demonstrate that the accuracy mAP value of our method is 2.9% higher than the baseline, FPS is increased by 33.8%. The number of parameters is reduced by 9.9%, and FLOPs is reduced by 16.0%. Compared with other methods, the proposed method has obvious accuracy and lightweight advantages.

Tensor decomposition based fusion methods for hyperspectral and multi-spectral images have recently demonstrated decent fusion performance in the context of remote sensing applications. Considering the huge volume of today’s remote sensing datasets and the involved heavy computation loads, traditional single-machine fusion methods can no longer serve the purpose due to limited storage, communication, and computation capacities. To this end, we propose a distributed and parallel method for Tucker decomposition model based on Apache Spark. Our primary focus is to (i) enhance the generation and update methods for modal matrices to eliminate the bottleneck of insufficient memory; (ii) redesign the initialization and updating procedures of core tensors to adapt to distributed computing and enhance computational efficiency; and (iii) implement group updates to accelerate the overall fusion flow while maintaining fusion accuracy. Experimental results demonstrate that our distributed and parallel fusion method achieves significant speedups over the original model with decent fusion performance.

The diffusion model has recently exhibited remarkable efficacy in image synthesis. Existing diffusion methods rely extensively on abundant training images as data priors, posing challenges for hyperspectral image(HSI) restoration with limited data availability. To tackle this problem, we introduce a novel diffusion approach for HSI restoration, which draws upon the theoretical foundation of denoising diffusion null-space model and works by inferring the parameters of the Spatial-Spectral network during the reverse diffusion process, requiring only a degraded image as input. Through experiments, we showcase the efficacy of this approach in producing impressive results in the realm of hyperspectral image restoration.

Change detection is a concerned task in the interpretation of remote sensing images. The mining of the relationship on change features is usually implicit in the deep learning networks that contain single-branch or two-branch encoders. However, due to the lack of artificial prior design for the relationship on change features, these networks cannot learn enough semantic information on change features and lead to the poor performance. So, we propose a new network architecture explicit change-relation network (ECRNet) for the explicit mining of change-relation features. In our study of the literature, our suggestion is that the change features for change detection should be divided into prechanged image features, postchanged image features, and change-relation features. In order to fully mining these three kinds of change features, we propose the triple branch network combining the transformer and convolutional neural network (CNN) to extract and fuse these change features from two perspectives of global information and local information, respectively. In addition, we design the continuous change-relation (CCR) branch to further obtain the continuous and detailed change-relation features to improve the change discrimination capability of the model. The experimental results show that our network performs better than those of the existing advanced networks by the F1 score improvements of 0.66/0.37/0.70/1.09 on the very high-resolution (VHR) remote sensing datasets of the LEVIR-CD/SVCD/WHU-CD/SYSU-CD. Our source code is available athttps://github.com/DalongZ/ECRNet.

In this letter, we propose a patchwise temporal-spatial feature aggregation (PTFA) network for object detection in satellite video. First, the feature extractor processes the key frame (KF) along with its support frames to ensure comprehensive spatial coverage of potential objects. Subsequently, we model the semantic similarities among instance-level proposals to exploring robust interaction between temporally adjacent support frames and KF. Furthermore, due to the extremely small size of objects in satellite video, we crop the input frames to different patches by the fixed criterion. Then, the temporal-spatial feature aggregation (TSFA) operations are performed on instance-level RoI features, which attains more nuanced and comprehensive descriptors from the explicit high-resolution temporal-spatial features. The patch features are reconstructed to the original one for complementing more valid feature responses. Finally, we compare our PTFA network with many recent works on the SAT-MTB dataset. Extensive experiments demonstrate that our method achieves the state-of-the-art performance than various static image and video object detection (VID) approaches.

Typical high-level vision tasks in hyperspectral image (HSI) processing, such as target detection, often suffer from insufficient information inherent in real-world sampled data. Super-resolution, a powerful tool in HSI low-level vision, is expected to enhance the accuracy of detection results by computationally providing the high-resolution HSI with additional information. However, existing solutions for HSI super-resolution and target detection have always been implemented independently. This conventionally adopted paradigm overlooks the interconnectedness between low-level and high-level visions, inevitably introducing additional errors, redundancies, and inefficiencies. To address this challenge, in this study, we put our efforts into exploring the uncharted continent of hyperspectral remote sensing, that is, realizing the mutual guidance and joint optimization of HSI super-resolution and target detection concurrently within a unified framework. Technically, we first construct different spectral bases to span the target and background subspaces of the underlying high-resolution HSI. Then, we look in-depth at the intrinsic properties of the HSI tensor, henceforth jointly optimizing both tasks by innovatively developing a novel low-cubic-rank tensor approximation model with a unique constrained energy minimization loss. While we have developed efficient algorithms to optimize the proposed model, we also put into place a refinement procedure for spectral bases, aimed at further enhancing the spectral fidelity of the fused results and the compact representation of the target subspace. Finally, empirical studies conducted on synthetic and real-world datasets substantiate that compared with state-of-the-art solutions, the proposed method delivers highly competitive and practical performance in terms of both tasks. Source codes are available at https://github.com/CX-He/HySRTD.git.

Oriented object detection is an important research topic in remote sensing. The detection of oriented objects in remote sensing images remains a daunting challenge due to their complex backgrounds, various sizes, diverse aspect ratios, and especially arbitrary orientations. In recent years, keypoint-based anchor-free object detectors have demonstrated outstanding performance in this field. However, in current anchor-free detectors, object keypoints are primarily generated using the Gaussian kernel function, which assumes a circular form. This representation falls short in accurately conveying an object’s size and orientation. To address the aforementioned issue, this paper proposes a keypoint-based oriented object detector called MRSDet, which innovatively adopts the Tricube kernel, scales and rotates it, to better generate the center keypoint heatmap of the object. Besides, to improve the model’s detection performance on oriented objects and improve its ability to perceive object keypoints and boundary boxes, we also design a large receptive field mask module (LRFM), which is based on large convolution kernel decomposition and semantic segmentation masks. Taking the BBAVectors method as a baseline, we conduct experiments on multiple types of remote sensing datasets such as HRSC2016, UCAS-AOD and SSDD+ to verify the effectiveness and generalizability of the proposed method.

In recent years, deep learning has shown promising results in change detection (CD) of hyperspectral images (HSIs). However, these algorithms often require a large number of labeled samples, which can be costly in practical applications. This article proposes a few-shot HSI CD method based on a 3-D convolution neural network and normalizing flow. To mitigate the issue of limited training samples, we develop a low-parameter baseline CD model utilizing 3-D convolution neural networks to extract spectral-spatial features while preventing overfitting. In addition, the proposed method analyzes the spectral-spatial distribution of hyperspectral variation by normalizing flow and generates hyperspectral tensor samples that approximate the distribution, thus enhancing CD performance. Furthermore, to improve the sample generation speed, we employ soft labels and label smoothing techniques to assign high-quality labels to the generated samples, thereby increasing the number of available samples. The method is evaluated on three datasets, and the experimental results demonstrate the efficacy of the proposed approach for detecting changes with limited training samples. The proposed method offers a promising solution to the challenge of detecting changes with limited training samples in HSIs.

Hyperspectral image (HSI) target detection plays a pivotal role in both military and civilian sectors. Nevertheless, this task is fraught with challenges because of the limited availability of target samples and the intricate nature of the background within real-world HSIs. In this study, we present an innovative background learning model based on the orthogonal subspace-guided variational autoencoder, tailored to discern the background distribution in hyperspectral imagery. Given the scarcity of target samples, our model is exclusively trained on background spectral samples, enabling precise modeling of the background distribution. The crux of our approach lies in detecting disparities between the reconstructed HSI and the original HSI, providing a mechanism for faithful target identification. To procure background samples, a coarse detection of the test HSI is first conducted. However, this process proves challenging, as obtaining pristine background pixels is a formidable task. To mitigate the influence of suspicious target samples on the background reconstruction, we employ orthogonal subspace loss on the reconstructed HSI. Extensive experiments conducted on four real-world HSIs substantiate that the proposed framework performs highly competitively and the results outperform other state-of-the-art HSI target detection methods. The source codes of this study are available at https://github.com/CX-He/OS-VAE.

Constrained by the performance of edge devices and real time (RT) processing technology, the existing hyperspectral target detection algorithms often struggle to rapidly distinguish targets from complex background pixels during real-time detection. To address this issue, this article proposes a new real-time cloud-edge selective background energy constrained (CE-SBEC) hyperspectral target detection algorithm. This algorithm aims to obtain detection results in real-time after capturing new data. Moreover, it conducts in-depth analysis based on existing detection results and updates the algorithm’s internal data to enhance its capabilities in terms of global background annihilation (GBA) and complex background suppression (CBS). Consequently, it improves the accuracy of subsequent real-time detection results. To enhance the resource utilization, this article deploys various task nodes of the algorithm separately on both the cloud and the edge, enabling collaborative execution of the CE-SBEC algorithm. In our context, edge devices are airborne equipment designed for the rapid acquisition and processing of data at the site of data collection, while cloud computing devices refer to high-performance computing clusters situated at a significant distance from the data collection site. Experimental results demonstrate that compared with existing detection algorithms, our newly proposed method achieves more accurate detection results while ensuring real-time performance.

Unsupervised domain adaptation (UDA) reduces domain shifts between distributions to enable model generalization to new scenarios. Adversarial domain adaptation (DA) is an effective approach that extracts domain-invariant features through adversarial learning, but such methods often neglect the influence of category differences on domain discrimination. To solve this problem, we construct a new unsupervised domain adaption hyperspectral image (HSI) classification method. The proposed method consists of two modules, namely, the pairing domain discrimination learning module and the multilevel mutual information maximization module. We propose to construct the sample pair as the input of the domain discriminator. We introduce a new label to the sample pair according to the labels of the two samples and use the relationship between samples to reduce the impact of sample category differences on domain discrimination. When extracting the shared features of the two domains, it will inevitably cause the loss of task-related information. This information is retained by maximizing the proposed multilevel mutual information. The experimental results on different datasets show the effectiveness of our method.

In hyperspectral cross-scene classification tasks, it is often challenging to obtain target domain samples during the training phase. Therefore, models need to be trained on one or multiple source domains and achieve good generalization performance on unknown target domains, known as domain generalization. The presence of domain shift limits the model’s generalization across different domains, while the unknown target domain makes it difficult to accurately characterize the distribution differences between domains. To address this issue, we propose a generalization network based on nonlinear sample generation. The network divides the sample features into invariant features and variant features and generates samples by applying nonlinear transformations to the variant features. To ensure the quality of the generated samples, we introduce contrastive learning into the model. It ensures consistency in similarity between the generated samples and the source samples while maintaining a certain degree of dissimilarity. Experiments conducted on four cross-domain adaptive scenarios demonstrate the superior performance of our proposed method.

The detection of oriented objects in remote sensing images remains a daunting challenge due to their complex backgrounds, various sizes, and especially arbitrary orientations. However, most of the existing methods only model the structural features of the images in the spatial domain, while the horizontal convolution kernels limit the model’s ability to perceive object direction information. Furthermore, the frequency features contain rich information about scale, texture, and angle, which can be a good complement to the spatial features. Inspired by this, we propose a multiscale spatial-frequency domain network (MSFN) to utilize spatial-frequency information for oriented object detection, which can be integrated into any convolutional neural network (CNN) architectures seamlessly and perform end-to-end training easily. Firstly, multiscale Haar wavelet transforms are leveraged to extract the multiscale frequency domain features from the image. Subsequently, channel alignment feature fusion module (CA-FFM) is proposed to fuse the high-level semantic features extracted by CNN with the low-level texture features extracted by the wavelet transform in multiscale. Finally, a channel self-attention (CSA)-based spatial-frequency feature perception module (SFPM) is designed to perform self-attention weighted aggregation on the fused features along the channel dimension, thereby constructing a novel spatial-frequency feature extraction backbone network for oriented object detector in remote sensing images. Experimental results on the DOTA and HRSC2016 datasets validate the effectiveness and universality of the proposed method.

Hyperspectral anomaly detection (AD), as a frontier research topic in the field of remotely sensed data processing, aims to identify targets of interest from complex and vast images. Existing AD methods typically involve complex models and many parameters, posing challenges in meeting the requirements of computational efficiency in hyperspectral AD. To address this issue, this article presents an AD acceleration algorithm based on the multivariate Gaussian model as well as its field programmable gate array (FPGA) implementation. By exploiting the parallel processing capabilities of FPGA, we introduce an innovative spectral dimensionality reduction method in which the data processing flow can be accomplished in a distributed manner. Then, we employ an improved linear rotation strategy based on correlation coefficients to accelerate the convergence rate of the proposed AD algorithm. The rotation of Gaussianization in the improved strategy is independent of eigenvalue decomposition, thereby substantially reducing the computational complexity involved during the rotation procedure. Furthermore, we apply a pipeline parallel mechanism to facilitate the FPGA implementation of the AD algorithm and to significantly enhance the computational efficiency. Experimental results on an embedded FPGA platform demonstrate that the FPGA implementation of the hyperspectral AD algorithm proposed in this article achieves a significant acceleration rate with guaranteed high detection accuracy.

Over the past few decades, researchers have proposed various hyperspectral unmixing (HU) methods. Among these methods, deep learning (DL) has emerged as a promising approach for HU, providing new opportunities for advancement. However, accurately quantifying the presence of spectral variability factors within a mixture remains a challenging task. Therefore, numerous literatures have concerned the HU with spectral variability, in which the variation spectra are generated through the network. However, there is a lack of the connection between the network and spectral variability, so they fail to provide physically meaningful interpretability of spectral variability. To this end, we use physics-driven model to represent spectral variability and introduce it to the two-stream autoencoder unmixing network, resulting in the improved endmember and abundance estimations. Specifically, the endmember extraction network learn spectral variability parameters associated the dispersion model to generate the variations of spectra, which enhancing physical interpretability of endmember variability. In addition, the abundance estimation autoencoder network, tied to the endmember extraction network by shared weights, estimates abundances using the reconstructed hyperspectral image. Compared with the state-of-the-art HU approaches on three real hyperspectral image datasets, our method outperforms these techniques with improved unmixing accuracy, especially on endmember estimation.

Oriented object detection (OOD) in remote sensing images (RSIs) remains a challenging work due to an arbitrary orientation of instance. Learning rotation-invariant features is critical in modeling a fixed descriptor for instances with its rotated variants. However, most existing methods construct the descriptor from the perspectives of data or feature augmentation, but ignore the exploration of potentially useful supervision information inside the detection algorithm. In this paper, we propose an object-wise rotation-invariant semantic representation (ORSR) framework, which synergizes the exploration of latent supervision, rotation-invariant learning, and guided attention mechanism into a unified network to boost the performance of OOD in RSIs. First, supervised by our constructed pseudo ground truth of segmentation masks, a semantic segmentation branch is built along with the detection algorithm to refine the representation of backbone features. Moreover, a consistency loss function is proposed to encourage the segmentation branch to make the fixed predictions for backbone features with its rotated variants. Considering that segmentation predictions remain the same affine transformations before and after rotating, we further construct a Kullback-Leibler (KL) Divergence based similarity loss function for encouraging the network to model the rotation-invariant features. Finally, we separate the ”object” descriptor from the segmentation predictions to extend the implicit constraint in our proposed semantic segmentation branch. The separated ”object” descriptor not only involves the spatial regularizer to emphasize the high-responsive regions in image, but also can be guided by the constructed consistency loss function. We evaluate our proposed ORSR on the challenging DOTA, DIOR-R, and HRSC2016 datasets. Extensive experiments demonstrate that the proposed ORSR achieves competitive performance compared to other single-scale and multi-scale detection methods.

Hyperspectral image (HSI) super-resolution, which aims at improving the spatial quality of HSIs by fusing a low spatial resolution HSI (LR-HSI) with a high spatial resolution multispectral image (HR-MSI), has drawn significant attention. Numerous LR-HSI and HR-MSI (HSI-MSI) fusion algorithms have emerged in recent times, yet they suffer from a lack of generality and integration, which hampers their usability for non-expert users. Moreover, these algorithms encounter significant challenges due to the exponential increase in remote sensing data volume. In this study, we propose a unified cloud-based framework for HSI-MSI fusion based on the general distributed alternating direction method of multipliers that incorporates nonlocal principles and tensor decomposition. The framework not only provides end-users with visualization modeling capabilities equipped with standard and comprehensive components, but also enhances the parallel processing capabilities of cloud computing. We employ a new proposed nonlocal adaptive low-rank coupled tensor canonical polyadic (CP) decomposition algorithm as a case study to evaluate the performance of this framework. Specifically, we establish the LR-HSIs and HR-MSIs relationship using order-4 coupled tensor CP decomposition and suggest an adaptive CP rank estimation method for achieving better super-resolution results. Experimental results on publicly available datasets demonstrate that the proposed parallel distributed optimization algorithm can achieve significant speedup with guaranteed accuracy. The proposed framework enables convenient and efficient processing of large-scale remote sensing data, effectively addressing the challenges associated with handling large data volumes. The source code of our method is released and available online at https://github.com/ZpWaitingForSunshine/DNAC4TCP/.

The recently proposed high-order tensor algebraic framework generalizes the tensor singular value decomposition (t-SVD) induced by the invertible linear transform from order-3 to order-d ( ). However, the derived order-d t-SVD rank essentially ignores the implicit global discrepancy in the quantity distribution of non-zero transformed high-order singular values across the higher modes of tensors. This oversight leads to suboptimal restoration in processing real-world multi-dimensional visual datasets. To address this challenge, in this study, we look in-depth at the intrinsic properties of practical visual data tensors, and put our efforts into faithfully measuring their high-order low-rank nature. Technically, we first present a novel order-d tensor rank definition. This rank function effectively captures the aforementioned discrepancy property observed in real visual data tensors and is thus called the discrepant t-SVD rank. Subsequently, we introduce a nonconvex regularizer to facilitate the construction of the corresponding discrepant t-SVD rank minimization regime. The results show that the investigated low-rank approximation has the closed-form solution and avoids dilemmas caused by the previous convex optimization approach. Based on this new regime, we meticulously develop two models for typical restoration tasks: high-order tensor completion and high-order tensor robust principal component analysis. Numerical examples on order-4 hyperspectral videos, order-4 color videos, and order-5 light field images substantiate that our methods outperform state-of-the-art tensor-represented competitors. Finally, taking a fundamental order-3 hyperspectral tensor restoration task as an example, we further demonstrate the effectiveness of our new rank minimization regime for more practical applications. The source codes of the proposed methods are available at https://github.com/CX-He/DTSVD.git.

Object detection in remote sensing images (RSIs) plays an important role both in civil and military fields. Currently, many object detection algorithms in RSIs have shown the excellent capability. However, these methods are designed for the single RGB modality, which cannot cope with the challenges in insufficient illumination or foggy scenarios. Infrared images measure the temperature of the captured objects, and it can avoid the influence of low illumination and fog. In this paper, we propose a novel RGB-Infrared multi-modal remote sensing object detection method termed as RIFuse to address these challenges. RIFuse combines convolutional neural networks (CNNs) and Transformer in a parallel hierarchy, which can efficiently extract the local features of RGB images and the global representations of infrared images. Besides, an adaptive multi-modal feature fusion block (MFF block) is proposed to fuse the features from both branches comprehensively. Extensive experiments demonstrate the superiority of our method for multi-modal object detection on RSIs.

Classification using multi-source remote sensing data has attracted widespread attention and is playing an increasingly important role in various fields. However, due to the disparity in imaging methods and the informational imbalance among data from various sources, it is still challenging to incorporate complementary advantages. Furthermore, real-world situations frequently involve varied data resolution, and the outcomes of straightforward super-resolution preprocessing are not always helpful for performing subsequent classification tasks. In this paper, we propose a unified framework for joint super-resolution and classification tasks of low resolution hyperspectral images (LR-HSIs) and SAR images. The minimization of the proposed joint loss function in the generative adversarial network (GAN) framework, including super-resolution and classification objective functions, can effectively achieve the multi-task goals. Experimental results on the super-resolution and classification of real LR-HSIs and SAR images demonstrate the effectiveness of the proposed method both visually and quantitatively.

Recently, tensor singular value decomposition (t-SVD) has demonstrated excellent performance in various high-dimensional information processing applications. However, in adapting t-SVD to handle the typical tensor data restoration tasks, such as hyperspectral image (HSI) denoising, the following questions remain inadequately addressed: 1) The existing tensor nuclear norm minimization (TNN) regime treats all tensor singular values alike; thus, it lacks flexibility and dominance in dealing with the sophisticated HSI tensor. 2) The existing t-SVD-based denoising methods can not directly process order-p(p> 3) tensors; thus, they fail to comprehensively exploit the high-dimensional structural correlation of the HSI tensor along different modes. To address the above challenges, in this study, we first generalize a novel weighted order-pTNN minimization regime, which integrates the adaptively reweighting strategy for matrix, third-order, and order-ptensors in a unified architecture. Subsequently, an efficient subspace low-rank learning model is established, using HSI denoising tasks as an application example to corroborate the superiority of the proposed regime in approximating the high-dimensional low-rank structure of natural tensor data. Extensive experimental results substantiate that our effort surpasses existing state-of-the-art low-rank tensor recovery methods in both restoration accuracy and efficiency. The source code is available at https://github.com/CX-He/WTNN.git.

Target detection based on hyperspectral image (HSI) representations has drawn wide attention given its wide variety of features. The matrix-based approach inevitably loses spatial information and fails to explore the intrinsic multimodal structure of an HSI cube. In this paper, we propose a regularized tensor-based model without altering the data structure. We assume that an observed third-order HSI tensor is decomposed into the sum of a Total Variation regularized Low-rank background tensor and a Sparse (TVLrS) target tensor. The two tensors are represented as the mode-3 product of a third-order tensor, called the Tensor Representation Coefficient (TRC), and a spectra dictionary matrix. Then, the model is coined as TVLrS-TRC. The background TRC has a low-rank property, contributing to the low-rankness characterization in our model. Moreover, as the size of the background TRC term is smaller than the background tensor, characterizing its local smoothness via TV regularization reduces the computational cost compared to that of the background tensor. Extensive experiments on two real hyperspectral datasets demonstrate the advantage of the proposed method compared with the state-of-the-art.

Hyperspectral image super-resolution aims to fuse a low spatial resolution hyperspectral image (LR-HSI) and a high spatial resolution multispectral image (HR-MSI) to obtain a high-resolution hyperspectral image (HR-HSI). Tensor-based methods have demonstrated their outstanding ability in constructing the relationship between the LR-HSI and the HR-MSI. This paper introduces a nonlocal hierarchical Tucker decomposition (HTD) model for hyperspectral and multispectral image (HSI-MSI) fusion. First, similar nonlocal patch tensors are clustered according to their similarity in the HR-MSI. Next, the spatial/spectral relationship between the LR-HSI and the HR-MSI is extracted through HTD. The alternating direction method of multipliers (ADMM) is employed to solve the proposed model. Furthermore, to overcome the high computational complexity of the model solver, we propose an efficient distributed and parallel method to accelerate the fusion process. Experimental results demonstrate that the proposed method not only substantially outperforms state-of-the-art HSI-MSI fusion methods, but also achieves a significant acceleration rate.

Recently, convolutional neural network (CNN) based image super-resolution (SR) methods have achieved significant performance improvement. However, most CNN-based methods mainly focus on feed-forward architecture design and neglect to explore the feedback mechanism, which usually exists in the human visual system. In this paper, we propose feedback pyramid attention networks (FPAN) to fully exploit the mutual dependencies of features. Specifically, a novel feedback connection structure is developed to enhance low-level feature expression with high-level information. In our method, the output of each layer in the first stage is also used as the input of the corresponding layer in the next state to re-update the previous low-level filters. Moreover, we introduce a pyramid non-local structure to model global contextual information in different scales and improve the discriminative representation of the network. Extensive experimental results on various datasets demonstrate the superiority of our FPAN in comparison with the state-of-the-art SR methods.

Attention mechanisms improve the classification accuracies by enhancing the salient information for hyperspectral images (HSIs). However, existing HSI attention models are driven by advanced achievements of computer vision, which are not able to fully exploit the spectral-spatial structure prior of HSIs and effectively refine features from a global perspective. In this paper, we propose a unified attention paradigm (UAP) that defines the attention mechanism as a general three-stage process including optimizing feature representations, strengthening information interaction, and emphasizing meaningful information. Meanwhile, we designed a novel efficient spectral-spatial attention module (ESSAM) under this paradigm, which adaptively adjusts feature responses along the spectral and spatial dimensions at an extremely low parameter cost. Specifically, we construct a parameter-free spectral attention block that employs multi-scale structured encodings and similarity calculations to perform global cross-channel interactions, and a memory-enhanced spatial attention block that captures key semantics of images stored in a learnable memory unit and models global spatial relationship by constructing semantic-to-pixel dependencies. ESSAM takes full account of the spatial distribution and low-dimensional characteristics of HSIs, with better interpretability and lower complexity. We develop a dense convolutional network based on efficient spectral-spatial attention (ESSAN) and experiment on three real hyperspectral datasets. Experimental results demonstrate that the proposed ESSAM brings higher accuracy improvement compared to advanced attention models.

For unmixing of sequences of hyperspectral images (SHS), spectral variability is an important factor to be considered. However, most existing unmixing methods tend to model the endmember and its variability in spatial domain rather than transform domain. In fact, the intrinsic and invariant features of the spectral curve can be effectively represented by wavelet transform. Therefore, this paper proposes to perform SHS unmixing in the wavelet domain by combing the Bayesian method. Firstly, the assumption of abundance being invariability in both the spatial and wavelet domains is made, then the formulation of unmixing in the wavelet domain using Perturbed Linear Mixing Model (PLMM) is presented. Secondly, based on the Bayesian framework, the likelihood and prior are both given, in which the parameter priors are divided into two parts: low and high frequency wavelet coefficients. Moreover, by considering the sparsity of the high-frequency wavelet coefficients of endmembers, a non-informative prior with zero-mean is designed. Meanwhile, for the coefficients of endmember variability, Gaussian distributions are utilized to represent the steady fluctuation along the temporal dimension. Finally, using the maximum a posterior (MAP) rule, a hierarchical spectral variability unmixing model in wavelet domain is built and solved by the Markov chain Monte Carlo (MCMC) sampling algorithm. Numerical experiments show that the proposed method generates more accurate estimates for endmembers and their variation.

Hyperspectral anomaly detection (AD) task is a typical binary classification problem, and utilizing background prior knowledge is a key technique to solving such problems. The two most commonly used priors for hyperspectral images are low-rank and local smooth properties. Most traditional matrix-based methods use two regularizations to model these two types of priors and integrate them into one model, which makes these two regularizations unable to maximize their effectiveness. In addition, the matrix method also destroys the structure of the hyperspectral images (HSI). To address these issues, this study identified a unique sparsity property in the gradient tensor of HSI. Specifically, the core tensor resulting from the Tucker decomposition of the gradient tensor was observed to exhibit sparsity. This sparsity property, referred to as GCS (the sparsity on the core tensor of the gradient map), effectively captures the structural information of HSI and improves detection performance. The GCS regularization offers the following advantages: 1) GCS regularization uses one term to simultaneously capture both low-rankness and local smoothness, the size of the core tensor represents the low-rank prior to the background, and the ℓ1norm describes the sparsity of gradient map, i.e., the local smoothness of the original data; 2) GCS is a constrained regularization, allowing for the full utilization of information from different dimensions of the HSI when updating the core tensor, i.e., utilizing the spatial and spectral information carried by three-factor matrices of the Tucker decomposition. Finally, extensive experiments validate the superiority of our proposed methods.

Classifying high-dimensional hyperspectral image (HSI) with limited labeled samples is a difficult problem. One effective solution is to leverage knowledge from scenes with well-labeled image (the source domain) to aid training in the target domain. However, since the source and target domains have different category spaces, it is crucial to extract more discriminative features and address domain adaptation challenges. To tackle this issue, we propose a cross-scene classification method for HSIs via generative adversarial networks (GANs) in latent space (GLS). Our method employs autoencoders (AEs) to map the input data to a latent space, where the most effective feature representation is extracted and preserved by deep residual 3D convolutional neural networks (CNN). The unlabeled samples in the target domain are also utilized in the AE which ensure all the samples are considered. We leverage conditional adversarial domain adaptation to overcome the domain shift, and introduce maximum mean discrepancy loss to minimize distribution differences between the two domains, facilitating better domain distribution alignment. We tested our approach on three public datasets and demonstrated that it outperforms existing few-shot learning methods. Our results highlight the effectiveness of our classification method via GANs in latent space for HSIs, and show that it has potential for practical applications.

In recent years, fusing multi-type features poses great potential for oriented object detection (OOD) in remote sensing images (RSIs). Due to the inexplicit operation of modeling orientation variations, convolutional neural networks (CNNs) are difficult to perceive objects under different transformations (angles and scales). In this paper, we propose a novel instance-aware spatial-frequency feature fusion detector (SFFD) for oriented object detection in remote sensing images. First, a layer-wise frequency-domain analysis (L-FDA) module is built along with CNN layers to extract frequency features. Getting rid of the constrains such as horizontal rectangular kernel in CNNs, our L-FDA possesses outstanding ability of locating mutational signals from frequency space. These mutational signals record the scale and angle information of the oriented instances in images. Subsequently, CNN and frequency features are sent into RoI Pooling layer to obtain multi-type instance-level RoI features. Moreover, the proposed instance-aware cross feature fusion (CFF) module explores the interaction between these diverse features which provides an explicit indicator to compensate the orientation information ignored by instance-level CNN features. Finally, our SFFD unifies the proposed L-FDA module and CFF module into the detection network to localize oriented instances in RSIs. We compare our method with many state-of-the-art methods on DOTA, HRSC2016, and NWPU VHR-10 datasets. Experimental results verify the validity of modeling instance-level object relations from frequency-domain and CNNs for OOD.

The synthesis of high-resolution (HR) hyperspectral image (HSI) by fusing a low-resolution HSI with a corresponding HR multispectral image has emerged as a prevalent HSI super-resolution (HSR) scheme. Recent researches have revealed that tensor analysis is an emerging tool for HSR. However, most off-the-shelf tensor-based HSR algorithms tend to encounter challenges in rank determination and modeling capacity. To address these issues, we construct nonlocal patch tensors (NPTs) and characterize low-rank structures with coupled Bayesian tensor factorization. It is worth emphasizing that the intrinsic global spectral correlation and nonlocal spatial similarity can be simultaneously explored under the proposed model. Moreover, benefiting from the technique of automatic relevance determination, we propose a hierarchical probabilistic framework based on Canonical Polyadic (CP) factorization, which incorporates a sparsity-inducing prior over the underlying factor matrices. We further develop an effective expectation-maximization-type optimization scheme for framework estimation. In contrast to existing works, the proposed model can infer the latent CP rank of NPT adaptively without tuning parameters. Extensive experiments on synthesized and real datasets illustrate the intrinsic capability of our model in rank determination as well as its superiority in fusion performance.

Combining tensor decomposition and image nonlocal information for remote sensing image fusion method (NCTCP) can effectively preserve the spatial structure of the image, and can obtain good image fusion results, accordingly. However, the NCTCP that is a serial algorithm cannot handle massive remote sensing images due to the computing resources limitation of a single computer. To address this issue, we propose a distributed parallel nonlocal tensor CP decomposition optimization algorithm (DP_NCTCP) based on the Spark platform. The alternating direction method of multipliers(ADMM) in NCTCP is divided into two distributed computing subtasks that can be executed on Spark in parallel to improve the efficiency. Compared with NCTCP, the DP_NCTCP achieves high speedups without the degradation of fusion quality measures accuracy by fusing the real hyperspectral images.

This paper proposes a Rotation-equivariant Attention Feature Fusion Pyramid Networks for Aerial Object Detection named ReAFFPN. ReAFFPN aims at improving the effect of rotation-equivariant features fusion between adjacent layers which suffers from the semantic and scale discontinuity. Due to the particularity of rotational equivariant convolution, general methods are unable to achieve their original effect while ensuring rotation equivariance of the network. To solve this problem, we design a new Rotation-equivariant Channel Attention which has the ability to both generate channel attention and keep rotation equivariance. Then we embed a new channel attention function into Iterative Attentional Feature Fusion (iAFF) module to realize Rotation-equivariant Attention Feature Fusion. Experimental results demonstrate that ReAFFPN achieves a better rotation-equivariant feature fusion ability and significantly improve the accuracy of the Rotation-equivariant Convolutional Networks.

Hyperspectral computational imaging (HCI) is to reconstruct hyperspectral images (HSIs) based on the compressed signals collected by remote sensing and imaging systems. Collaborative Tucker3 tensor decomposition is beneficial for HCI models in reconstructing high-fidelity HSIs. However, the everincreasing amount of compressed data leads to heavy computation burden for tensor decomposition-based HCI models, which may exceed the computing capacity of a single machine. For this reason, this paper proposes a Spark-based distributed and parallel HCI implementation via collaborative Tucker3 tensor decomposition. The proposed implementation decomposes the processing flow of the HCI algorithm into several stages, each of which can be processed in parallel on Spark. In addition, we develop parallel strategies for improving the performance of the redundant computational procedure and data storage procedure, respectively. Experimental results demonstrate that the parallel algorithm not only achieves high accuracy but also improves the computational efficiency when processing large-scale HSI datasets.

Hyperspectral target detection (HTD) is one of the vital issues in hyperspectral image (HSI) processing, and has a pivotal role in remote sensing field. However, it is a challenging task due to redundant information, high dimensionality and complexity of background. In this paper, an effective HTD method based on Tensor Robust Principal Component Analysis (TRPCA) is proposed, which can spot targets from complicated background distribution. Our approach, called constrained energy regularization TRPCA, imposes tensor nuclear norm to describe the low-rank property of back-ground. Considering some target spectra probably lie in the same low-dimensional background subspace, we introduce a constrained energy regularization to enhance the distinction between target and background pixels. Experiments on four datasets illustrate superiority of our method.

Abstract Recently, residual and dense networks have effectively promoted the development of image super‐resolution (SR). However, most dense networks based SR methods do not make full use of dense feature information. To solve this problem, a pyramidal dense attention network for single image super‐resolution is proposed in this paper. In this method, the proposed pyramidal dense learning can gradually increase the width of the densely connected layer inside a pyramidal dense block to extract deep features efficiently. Meanwhile, the adaptive group convolution that the number of groups grows linearly with dense convolutional layers is introduced to relieve the parameter explosion. Besides, a novel joint attention to capture cross‐dimension interaction between the spatial dimensions and channel dimension in an efficient way for providing rich discriminative feature representations is also proposed. Extensive experimental results show that the method achieves comparable performance in comparison with the state‐of‐the‐art SR methods.

Tracking moving objects in remote-sensing videos is becoming increasingly important in remote-sensing analysis. This letter presents a novel object tracking method for remote-sensing videos. We start with using the traditional robust principal component analysis (RPCA) model to extract the moving object from the background. To describe the continuity of moving objects in spatial and temporal directions, we incorporate a 3-D total variation (3DTV) regularization into the RPCA model. Considering that the background is not static and the captured videos will contain noise because of the instability of the sensing camera, our proposed method introduces a certain part of the function to model the noise and capture the changes in background. Experimental results on real videos provided by 2016 IEEE GRSS Data Fusion Contest and 2020 Hyperspectral Object Tracking Challenge demonstrate the advantages of the moving object-tracking method via 3-D TV.

Nowadays, deep learning can play an important role in addressing the issue of hyperspectral anomaly detection (HAD). In order to further utilize the spatial information in hyperspectral images (HSIs), an anomaly detection method for HSIs is presented based on graph regularized variational autoencoder (GRVAE). Firstly, the proposed method uses the superpixel segmentation algorithm to segment the hyperspectral image (HSI) and constructs an adjacency matrix to evaluate the similarity between pixels. Secondly, a variational autoencoder is used to reconstruct the spectral vector of the HSI, and meanwhile, the spatial similarity of the image is shared in the feature space through the graph regularization term. Finally, the reconstructed background and the original input are used to obtain the spectral error map, and then the attribute filtering is used to further refine the detection results. Performed on four data sets of abnormal target data with different shapes and different background complexity, the experiments show that the method has promising anomaly detection performance.

In this article, an adaptive learning framework is established for a deep weighted sparse autoencoder (AE) by resorting to the multiobjective evolutionary algorithm (MOEA). The weighted sparsity is introduced to facilitate the design of the varying degrees of the sparsity constraints imposed on the hidden units of the AE. The MOEA is exploited to adaptively seek appropriate hyperparameters, where the divide-and-conquer strategy is implemented to enhance the MOEA's performance in the context of deep neural networks. Moreover, a sharing scheme is proposed to further reduce the time complexity of the learning process at the slight expense of the learning precision. It is shown via extensive experiments that the established adaptive learning framework is effective, where different sparse models are utilized to demonstrate the generality of the proposed results. Then, the generality of the proposed framework is examined on the convolutional AE and VGG-16 network. Finally, the developed framework is applied to the blind image quantity assessment that illustrates the applicability of the established algorithms.

A hyperspectral image sequence can be obtained at different time in the same region from a hyperspectral sensor. The environmental change usually leads to variation in endmember reflectance, which has an important influence on unmixing process. In this article, a Bayesian unmixing model considering spectral variability for hyperspectral sequence is proposed, in which composite prior distributions of abundance and endmember variability are developed. The abundance priors consider the continuity of abundance in the temporal and spatial domains, simultaneously. Specifically, in the spatial domain, a data-adaptive variance of the abundance prior distribution is put forward based on local spatial difference. Moreover, the priors of endmember variability in temporal continuity and spectral smoothness are also exploited. Finally, a joint posterior distribution is obtained by the likelihood function and the parameter prior distributions, which can be calculated by the Markov chain Monte Carlo (MCMC) algorithm. Experiments on synthetic and real data sets demonstrate the effectiveness of the proposed approach in terms of abundance, endmember, and its variability estimation accuracy.

Recently, deep learning has drawn significant attention in hyperspectral image (HSI) classification. With the growth of network depth and feature integration, deep learning demands abundant labeled samples to optimize many parameters. Unfortunately, most hyperspectral data are unlabeled and the available labeled samples are extremely limited. How to obtain richer features under limited training samples is a challenge for HSI classification. To tackle this issue, a new supervised multiscale alternately updated clique network (MSCN) is proposed for HSI classification to fully employ HSI features in different scales. Based on the Clique Block, we design the multiscale alternately updated clique block (MSCB) that applies convolution kernels of various sizes to adaptively exploit the multiscale HSI information and merge them within the block. Meanwhile, the recurrent feedback architecture is introduced to reuse high-level visual information and network parameters. The proposed MSCN includes two MSCBs to capture the multiscale spectral and spatial information in turn. The MSCN improves the information flow and the efficiency of parameter tuning through the feedback mechanism and the cross-utilization of multiscale feature. It not only obtains more abstract HSI information, but also reduces the network depth and the number of parameters, thereby improving the classification accuracy under limited samples. To certify the validity of the proposed MSCN, experiments are conducted on three real HSI datasets and compared with multiple state-of-the-art deep learning-based approaches. The experimental results demonstrate that the presented multiscale network achieves superior performance, especially in the case of a small number of training samples.

Limited by the shape-fixed kernels, convolutional neural networks (CNNs) are usually difficult to model difform land covers in hyperspectral images (HSIs), leading to inadequate land use. Recently, benefiting from the ability to conduct shape-adaptive convolutions and model complex patterns in graph-structured data, graph convolutional networks (GCNs) have been applied to HSI classification. However, due to the massive computation in GCNs, HSI is usually pretreated into a graph based on a specific superpixel segmentation, which limits the modeling of spatial topologies to the same scale. To break this limitation, we propose a multilevel superpixel structured graph U-Net (MSSGU) to learn multiscale features on multilevel graphs. Specifically, we construct several hierarchical segmentations from fine to coarse by progressively merging adjacent superpixels and then convert them into multilevel graphs. Meanwhile, based on the merging relations between hierarchical superpixels, we establish the pooling and unpooling functions to transfer features from one graph to another, thereby enabling different-level graphs to collaborate in a single network. Different from concatenating different-scale features straightforwardly in the feature fusion stage, MSSGU fuses them in a coarse-to-fine progressive manner, which can generate subtler fusion features adaptive to the pixelwise classification task. Moreover, we use a CNN instead of GCN to extract and fuse the pixel-level features, which greatly reduces the computation. Such a hybrid U-Net can exploit features of HSIs from a multiscale hierarchical perspective, and its performance has been proven competitive with other deep-learning-based methods by extensive experiments on three benchmark datasets.

The traditional methods of hyperspectral image (HSI) classification are based on the sufficient labeled data. In real life, we often encounter that the target domain corresponding to the classification task has only a small amount of labeled data, but the source domain has enough labeled data. However, the distribution of the source domain is different from the distribution of the target domain. Thus, the labeled data of the source domain cannot be applied to the target domain directly. This paper proposes a new method to solve the cross-domain few shot problem of HSI classification. In the proposed method, the local spatial alignment and the spectral alignment are simultaneously introduced to transfer the knowledge from the source domain to the target domain. Besides, to extract the domain specific features, we balance the domain invariant features and the domain specific features by a weakly parameter-shared mechanism. The two modules together can narrow the distance between two domains and make the model perform well on the target domain. Experiments conducted on four different target domain data sets demonstrate the effectiveness of our method.

Recently, deep convolutional neural network based hyperspectral and multispectral image fusion methods have shown significant performance. Nevertheless, the rich spatial and spectral details of hyperspectral images (HSIs) have not been fully explored, leaving room for further improve the representation ability of the model. In this paper, we propose an efficient cross-modality self-calibrated network (CMSCN) for hyperspectral and multispectral image fusion. Specifically, we use a cross-modality non-local module to fuse a high-resolution multispectral image (HR-MSI) and a low-resolution hyperspectral image (LR-HSI) to get an enhanced LR-HSI. In addition, a novel cross-scale self-calibrated convolution structure is proposed to explore and exploit multi-scale and hierarchical spatial-spectral features, which can improve the learning ability of the model. The introduced efficient spatial-spectral attention mechanism can calibrate the feature representation at different dimensions, thereby providing more efficient and accurate information for hyperspectral image reconstruction. Extensive experimental results on various hyperspectral images demonstrate the superiority of our method in comparison with the state-of-the-art image fusion methods.

Synthesizing high resolution (HR) hyperspectral image (HSI) by merging a low resolution (LR) HSI with corresponding HR multispectral image (MSI) has become a promising HSI super-resolution scheme. Most existing HSI-MSI fusion methods are effective to some extent, while several challenges remain. First, the spectral response of a given material exhibits considerable variability due to different acquisition time and conditions, however variations in spectral signatures are often neglected. Second, a majority of off-the-shelf methods require predefined degradation operators, which can be unavailable in practice. To tackle above issues, we introduce a novel fusion approach with Bayesian framework. Specifically, we regard the up-sampled LR-HSI as the low frequency component of the underlying HR-HSI. We characterize the texture features of high and low frequency components respectively, which can enlarge modeling capacity and bypass the absence of degradation operators. Furthermore, we depict the relative smoothness of reflectance spectra with Gaussian Process. Extensive experiments on synthesized and real datasets illustrate the superiority of the proposed strategy in terms of fusion performance and robustness to spectral variability.

Object detection in remote sensing images (RSIs) poses great difficulties due to arbitrary orientations, various scales and dense location of the targets over the ground. Recent evidence suggests that encoding the orientation information is of great use for training an accurate object detector for oriented object detection (OOD). In this paper, we propose a new frequency-domain orientation learning (FDOL) module with two main components: the frequency domain feature extraction (FFE) network and an orientation enhanced self-attention layer (OES-Layer). The FFE network models the interactions among spatial locations in the frequency domain to determine the frequency of spatial features. Then, these features are fed into our OES-Layer to learn the orientation information. Moreover, the orientation weights are adopted to guide the feature selection in a self-attention architecture, using them as a control gate to emphasize the spatial responses of target instances. Considering that the original similarity weights (calculated by the self-attention algorithm) do not distinctly model the orientation variation, the considered orientation weights provide an efficient asset to emphasize the orientation of objects. Extensive experiments on the DOTA and HRSC2016 datasets demonstrate that our method achieves state-of-the-art performance among single-scale methods, while achieving competitive performance over multi-scale methods.

In this paper, we propose a novel tensor-based approach to improve the reconstruction performance for dual-camera compressive hyperspectral imaging. We formulate a coupled tensor decomposition model to maintain the consistency of the spatial structure of the HSI and the panchromatic image. We introduce a regularizer over the core tensor to collaboratively promote global spatial-spectral correlations in HSI. Besides, we incorporate an anisotropic spatial-spectral total variation (SSTV) regularization to characterize the piecewise smooth structure of the HSI. Then the alternating direction method of multipliers (ADMM) algorithm is applied to the optimization problem. Experimental results on a public dataset demonstrate the superiority of the proposed approach.

Anomaly detection (AD) has become a hot topic in hyperspectral image (HSI) analysis. Anomalies are samples that are significantly different from the surrounding background in space or spectrum. However, the rich spectral-spatial features in HSI are not fully discovered by most traditional AD methods. In this paper, a 3D-convolutional Variational Au-toencoder (3D-CVAE) based AD method is proposed to make full use of the spectral-spatial information. The spectral-spatial features are extracted by the 3D-CVAE encoder and the background is reconstructed using these features through 3D-CVAE decoder. The residual between the original input and the reconstructed background contains the anomalies which can be easily detected by the Reed-Xiaoli(RX) detector in the residual. Experimental results on two HSI datasets demonstrate the advantage of the proposed method.

This article gives a survey of state-of-the-art methods for processing remotely sensed big data and thoroughly investigates existing parallel implementations on diverse popular high-performance computing platforms. The pros/cons of these approaches are discussed in terms of capability, scalability, reliability, and ease of use. Among existing distributed computing platforms, cloud computing is currently the most promising solution to efficient and scalable processing of remotely sensed big data due to its advanced capabilities for high-performance and service-oriented computing. We further provide an in-depth analysis of state-of-the-art cloud implementations that seek for exploiting the parallelism of distributed processing of remotely sensed big data. In particular, we study a series of scheduling algorithms (GSs) aimed at distributing the computation load across multiple cloud computing resources in an optimized manner. We conduct a thorough review of different GSs and reveal the significance of employing scheduling strategies to fully exploit parallelism during the remotely sensed big data processing flow. We present a case study on large-scale remote sensing datasets to evaluate the parallel and distributed approaches and algorithms. Evaluation results demonstrate the advanced capabilities of cloud computing in processing remotely sensed big data and the improvements in computational efficiency obtained by employing scheduling strategies.

In this paper, we present a new construction of asymmetric quantum codes (AQCs) by combining classical concatenated codes (CCs) with tensor product codes (TPCs), called asymmetric quantum concatenated and tensor product codes (AQCTPCs) which have the following three advantages. First, only the outer codes in AQCTPCs need to satisfy the orthogonal constraint in quantum codes, and any classical linear code can be used for the inner, which makes AQCTPCs very easy to construct. Second, most AQCTPCs are highly degenerate, which means they can correct many more errors than their classical TPC counterparts. Consequently, we construct several families of AQCs with better parameters than known results in the literature. Third, AQCTPCs can be efficiently decoded although they are degenerate, provided that the inner and outer codes are efficiently decodable. In particular, we significantly reduce the inner decoding complexity of TPCs from$\Omega (n_{2}a^{n_{1}})(a>1)$to$O(n_{2})$by considering error degeneracy, where$n_{1}$and$n_{2}$are the block length of the inner code and the outer code, respectively. Furthermore, we generalize our concatenation scheme by using the generalized CCs and TPCs correspondingly.

Deep Convolutional Neural Networks (CNN) have drawn great attention in image super-resolution (SR). Recently, visual attention mechanism, which exploits both of the feature importance and contextual cues, has been introduced to image SR and proves to be effective to improve CNN-based SR performance. In this paper, we make a thorough investigation on the attention mechanisms in a SR model and shed light on how simple and effective improvements on these ideas improve the state-of-the-arts. We further propose a unified approach called “multi-grained attention networks (MGAN)” which fully exploits the advantages of multi-scale and attention mechanisms in SR tasks. In our method, the importance of each neuron is computed according to its surrounding regions in a multi-grained fashion and then is used to adaptively re-scale the feature responses. More importantly, the “channel attention” and “spatial attention” strategies in previous methods can be essentially considered as two special cases of our method. We also introduce multi-scale dense connections to extract the image features at multiple scales and capture the features of different layers through dense skip connections. Ablation studies on benchmark datasets demonstrate the effectiveness of our method. In comparison with other state-of-the-art SR methods, our method shows the superiority in terms of both accuracy and model size.

Computational imaging for hyperspectral images (HSIs) is a hot topic in remote sensing and imaging systems. The dual-camera compressive hyperspectral imaging (DCCHI) system has been successfully designed and applied in hyperspectral imaging. However, the corresponding reconstruction algorithms are not well developed. In this paper, under the DCCHI framework, a new reconstruction algorithm is proposed based on the collaborative Tucker3 Tensor decomposition. In actual HSI, similar nonlocal patches always have similar spatial-spectral structures, and thus, these nonlocal patches can share the same spatial and spectral factors in Tucker decomposition. To characterize the similarities simultaneously, the Tucker3 decomposition is used to model the 4-order tensor formed by the similar cubic patches. To keep the spatial structures in the reconstructed HSI consistent with the panchromatic image's spatial structures, we force the spatial factor matrices and the core tensor in the Tucker3 decomposition of the HSI to be identical to the spatial factor matrices and core tensor of the panchromatic image's Tucker3 decomposition. In addition, a spectral quadratic variation constraint is introduced into the spectral factor to characterize the band smoothness. To solve the optimization problem, an alternating direction method of multipliers (ADMM)-based algorithm is designed and each variable is separately solved. Experimental results on a public data set and the remote sensing image demonstrate the advantage of the proposed method.

The large data volume and high algorithm complexity of hyperspectral image (HSI) problems have posed big challenges for efficient classification of massive HSI data repositories. Recently, cloud computing architectures have become more relevant to address the big computational challenges introduced in the HSI field. This article proposes an acceleration method for HSI classification that relies on scheduling metaheuristics to automatically and optimally distribute the workload of HSI applications across multiple computing resources on a cloud platform. By analyzing the procedure of a representative classification method, we first develop its distributed and parallel implementation based on the MapReduce mechanism on Apache Spark. The subtasks of the processing flow that can be processed in a distributed way are identified as divisible tasks. The optimal execution of this application on Spark is further formulated as a divisible scheduling framework that takes into account both task execution precedences and task divisibility when allocating the divisible and indivisible subtasks onto computing nodes. The formulated scheduling framework is an optimization procedure that searches for optimized task assignments and partition counts for divisible tasks. Two metaheuristic algorithms are developed to solve this divisible scheduling problem. The scheduling results provide an optimized solution to the automatic processing of HSI big data on clouds, improving the computational efficiency of HSI classification by exploring the parallelism during the parallel processing flow. Experimental results demonstrate that our scheduling-guided approach achieves remarkable speedups by facilitating the automatic processing of HSI classification on Spark, and is scalable to the increasing HSI data volume.

Recently, the graph convolutional network (GCN) has drawn increasing attention in the hyperspectral image (HSI) classification. Compared with the convolutional neural network (CNN) with fixed square kernels, GCN can explicitly utilize the correlation between adjacent land covers and conduct flexible convolution on arbitrarily irregular image regions; hence, the HSI spatial contextual structure can be better modeled. However, to reduce the computational complexity and promote the semantic structure learning of land covers, GCN usually works on superpixel-based nodes rather than pixel-based nodes; thus, the pixel-level spectral–spatial features cannot be captured. To fully leverage the advantages of the CNN and GCN, we propose a heterogeneous deep network called CNN-enhanced GCN (CEGCN), in which CNN and GCN branches perform feature learning on small-scale regular regions and large-scale irregular regions, and generate complementary spectral–spatial features at pixel and superpixel levels, respectively. To alleviate the structural incompatibility of the data representation between the Euclidean data-oriented CNN and non-Euclidean data-oriented GCN, we propose the graph encoder and decoder to propagate features between image pixels and graph nodes, thus enabling the CNN and GCN to collaborate in a single network. In contrast to other GCN-based methods that encode HSI into a graph during preprocessing, we integrate the graph encoding process into the network and learn edge weights from training data, which can promote the node feature learning and make the graph more adaptive to HSI content. Extensive experiments on three data sets demonstrate that the proposed CEGCN is both qualitatively and quantitatively competitive compared with other state-of-the-art methods.

Due to the delay in the row-wise exposure and the lack of stable support when a photographer holds a CMOS camera, video jitter and rolling shutter distortion are closely coupled degradations in the captured videos. However, previous methods have rarely considered both phenomena and usually treat them separately, with stabilization approaches that are unable to handle the rolling shutter effect and rolling shutter removal algorithms that are incapable of addressing motion shake. To tackle this problem, we propose a novel method that simultaneously stabilizes and rectifies a rolling shutter shaky video. The key issue is to estimate both inter-frame motion and intra-frame motion. Specifically, for each pair of adjacent frames, we first estimate a set of spatially variant inter-frame motions using a neighbor-motion-aware local motion model, where the classical mesh-based model is improved by introducing a new constraint to enhance the neighbor motion consistency. Then, different from other 2D rolling shutter removal methods that assume the pixels in the same row have a single intra-frame motion, we build a novel mesh-based intra-frame motion calculation model to cope with the depth variation in a mesh row and obtain more faithful estimation results. Finally, temporal and spatial motion constraints and an adaptive weight assignment strategy are considered together to generate the optimal warping transformations for different motion situations. Experimental results demonstrate the effectiveness and superiority of the proposed method when compared with other state-of-the-art methods.

Classification is one of the major research fields in hyperspectral imagery. Due to the fact that neighboring pixels are more likely to share the same label, it is practical to use spatial information in hyperspectral image to achieve higher accuracy. On the other hand, however, spatial information also leads to higher computational complexity. This paper proposes an efficient implementation of a spatial-spectral kernel sparse representation for hyperspectral image classification base on the multi-GPU platform. The proposed implementation takes advantage of the capability of compute-unified device architecture (CUDA), such as shared memory, streams and peer-to-peer (P2P) transfer of data. In addition, an improvement of performance can be achieved by calculation reorganization and bandwidth usage optimization. Experimental results demonstrate that the proposed method achieves an up to 56.81X speedup in computation time while guaranteeing the classification accuracy.

The environmental change caused by time interval usually leads to the disturbance of endmember reflectance curve, which has an important influence on multi-temporal hyperspectral unmixing process. In this paper, a Bayesian unmixing model considering the spectral variability is proposed, in which a mixed prior distribution of the abundance is constructed. Different form the existing methods, the continuity of the abundance in time and spatial domain is considered simultaneously. In order to describe the smoothness of the abundance, a data-adaptive variance of probability distribution is designed based on the spatial local difference. Then combined with the priors of endmembers and spectral variability, the joint posterior distribution is set up and Markov chain Monte-Carlo(MCMC) algorithm is developed for posterior computation. Experiments on simulated and real datasets demonstrate the effectiveness of the proposed algorithm in terms of abundance estimation and endmember estimation.

Convolutional neural networks (CNNs) have emerged as a powerful tool in remote sensing image analysis. However, the layer-by-layer convolutions (L2Convolutions) in CNNs cannot fully exploit the relativities of pixels in the 3D cube data, especially for hyperspectral images (HSIs). In this paper, a directional message propagation convolutional neural network framework (MPCNN), is proposed for the supervised classification of HSIs. In the proposed framework, we integrate a novel multi-directional message propagation mechanism, namely slice-by-slice convolutions (S2Convolutions), into the hidden feature layers which are generated by L2Convolutions to propagate feature information between feature maps in the same layer. Owing to the S2Convolutions, abundant and discriminative spectral-spatial feature learning can be enhanced compared with traditional CNNs without S2Convolutions. The performance of the proposed MPCNN is evaluated on benchmark dataset of HSIs, and quantitative and qualitative experiments show that the performance of the proposed method outperforms several state-of-the-art methods.

A spatial–spectral residual dense network-based compressive hyperspectral image (HSI) reconstruction method is proposed in this letter. The proposed method contains two networks: residual dense network for hyperspectral image reconstruction (RDNHIR) and spectral difference reconstruction network (SDRN). The RDNHIR network can extract the local features and global hierarchical features by cascading features of all residual dense blocks (RDBs). Then, SDRN takes full advantage of the strong correlation between spectral adjacent bands to better preserve the spectral feature of HSI. Finally, the adjacent spectral difference regularization is introduced into the loss function to further improve the performance. The experimental results show that the proposed method has better reconstruction quality than other state-of-the-art reconstruction methods, especially in the spectral domain.

Hyperspectral (HS) super-resolution, which aims at enhancing the spatial resolution of hyperspectral images (HSIs), has recently attracted considerable attention. A common way of HS super-resolution is to fuse the HSI with a higher spatial-resolution multispectral image (MSI). Various approaches have been proposed to solve this problem by establishing the degradation model of low spatial-resolution HSIs and MSIs based on matrix factorization methods, e.g., unmixing and sparse representation. However, this category of approaches cannot well construct the relationship between the high-spatial-resolution (HR) HSI and MSI. In fact, since the HSI and the MSI capture the same scene, these two image sources must have common factors. In this paper, a nonlocal tensor decomposition model for hyperspectral and multispectral image fusion (HSI-MSI fusion) is proposed. First, the nonlocal similar patch tensors of the HSI are constructed according to the MSI for the purpose of calculating the smooth order of all the patches for clustering. Then, the relationship between the HR HSI and the MSI is explored through coupled tensor canonical polyadic (CP) decomposition. The fundamental idea of the proposed model is that the factor matrices in the CP decomposition of the HR HSI's nonlocal tensor can be shared with the matrices factorized by the MSI's nonlocal tensor. Alternating direction method of multipliers is used to solve the proposed model. Through this method, the spatial structure of the MSI can be successfully transferred to the HSI. Experimental results on three synthetic data sets and one real data set suggest that the proposed method substantially outperforms the existing state-of-the-art HSI-MSI fusion methods.

Hyperspectral image (HSI) super-resolution is a hot topic in remote sensing and computer vision. Recently, tensor analysis has been proven to be an efficient technology for HSI image processing. However, the existing tensor-based methods of HSI super-resolution are not able to capture the high-order correlations in HSI. In this article, we propose to learn a high-order coupled tensor ring (TR) representation for HSI super-resolution. The proposed method first tensorizes the HSI to be estimated into a high-order tensor in which multiscale spatial structures and the original spectral structure are represented. Then, a coupled TR representation model is proposed to fuse the low-resolution HSI (LR-HSI) and high-resolution multispectral image (HR-MSI). In the proposed model, some latent core tensors in TR of the LR-HSI and the HR-MSI are shared, and we use the relationship between the spectral core tensors to reconstruct the HSI. In addition, the graph-Laplacian regularization is introduced to the spectral core tensors to preserve the spectral information. To enhance the robustness of the proposed model, Frobenius norm regularizations are introduced to the other core tensors. Experimental results on both synthetic and real data sets show that the proposed method achieves the state-of-the-art super-resolution performance.

The exploitation of prior of image is very important for the reconstruction of photon-limited Poisson image which is urgent demand and particularly challenging in many application fields. Recently, a low dimensional manifold model has attracted attention in image processing, in which all image patches are treated as samples of a same manifold and the dimension of patch manifold is utilized as a nonlocal regularization prior. But in fact, different image patches often belong to different manifolds, and existing analysis shows that the patch manifolds corresponding to different image components often have different dimensions. Considering this difference, we propose to cluster the image patches into several groups corresponding to certain image components such as cartoon component, texture and edges firstly, and then utilize different low dimensional manifold regularizations for different image patch groups and propose a multi-components low dimensional manifold model for Poisson noisy image reconstruction. Numerical experiments show that our method can improve the result both visually and in terms of the peak-signal-noise-ratio and the featuresimilarity-index-measurement efficiently, especially for the Poisson images with extremely small number of photons.

As a low-rank tensor modeling, tensor tubal rank has been received more attention in hyperspectral image (HSI) recovery. However, the tubal rank is often approximated by tensor nuclear norm, which leads to modeling bias. To achieve an unbiased approximation and improve the model robustness, in this paper, a non-convex relaxation based HSI low-rank recovery model is proposed. And the model is solved efficiently by alternating direction method of multipliers (ADMM) optimization method. Two HSI datasets are employed to exhibit the superiority of the proposed model over the nuclear norm penalization method in terms of the accuracy and robustness.

Anomaly detection has become a hot topic in hyperspectral image (HSI) processing. Both spatial and spectral features have been proven to be very important for accurate and efficient hyperspectral anomaly detection. The traditional HIS anomaly detection algorithms usually reshape HSI to a matrix, which destroy spatial or spectral structure. In this paper, we propose a novel method of hyperspectral anomaly detection based on LOW RANK AND SPARSE TENSOR DECOMPOSITION (LRASTD). Taking into consideration that HSI data can be essentially regarded as a three-order tensor. HSI is modeled as a background tensor and a sparse anomalies tensor. A tensor nuclear norm is employed to constrain the core tensor, which be designed to characterize the low dimensional structure of the core tensor. Furthermore, a novel sparse tenor norm is proposed to constrain the anomaly targets. Experiments on both simulated and real hyperspectral data sets demonstrate the efficiency and effectiveness of the proposed method.

A new pan-sharpening method based multispectral (MS) image intensity modulation is proposed. Firstly, according to the spectral correlation between MS image and panchromatic (PAN) image, a new spectral-enhanced PAN (SPAN) image is obtained. Then, SPAN image is utilized to calculate the spectral and spatial difference coefficients. Finally, the fused high resolution MS (HRMS) image is generated by modulating the intensity of the upsampled MS image. The experimental results demonstrate that the proposed method performs well both on spatial details preservation and spectral distortion reduction.

Change Detection in Remote sensing image is, in essence, to detect the changes of ground features with regard to time from remote sensing perspective. It is usually realized by analyzing and processing multi-temporal high resolution images. Change Detection based on fully connected conditional random field not only improves the detection accuracy of remote sensing image, but also achieves better robustness. However, with the growth of high-resolution data volumes, this algorithm consumes a huge amount of time and computational resources, and therefore needs to be improved accordingly. Spark is an open-source distributed general- purpose cluster-computing framework. It has powerful memory computing and efficient task scheduling capabilities for complex iterative calculations. Based on Spark, this paper proposes a distributed and parallel method of change detection in remote sensing image based on Fully Connected Conditional Random Field that analyzes the data input form, and proposes a multi-temporal image reading strategy on cloud platforms. This method decomposes the algorithm flow, and performs distributed parallel processing on each stage and makes full use of the processing advantages of data locality to implement a reasonable intermediate data storage. Experimental results demonstrate that this parallel method achieves a promising speedup with high scalability, while guaranteeing remarkable detection accuracy.

Sentinel-2 data has become an important tool for current and future earth observation due to its high quality, free availability and world-wide coverage. However, some of the spectral bands are sensed at reduced resolution due to design considerations and sensor hardware limitations. So in this paper we present a super-resolution method based on Convolutional Neural Networks (CNNs) to infer all the 20m spectral bands in the highest available resolution. This is accomplished by using an improved residual network and meanwhile we propose a channel attention mechanism to adaptively rescale the characteristics of the channels by considering the interdependencies among the channels. The proposed solution compares against several alternative methods according to different quality indexes. Our network provides the best results and a compelling visual effect on the sentinel-2 images.

Hyperspectral image (HSI) is often contaminated by mixed noise, which severely affects the visual quality and subsequent applications of the data. In this paper, HSI restoration based on low-rank recovery with a local neighborhood weighted spectral-spatial total variation (TV) model is proposed, which focuses on the preservation of spatial structure and spectral fidelity. The low-rank matrix model is adopted to exploit the spectral and spatial correlation information, and the l1-norm is used as a prior to remove the sparse noise. Furthermore, a local spatial neighborhood weighted spectral-spatial TV is utilized to jointly model the spectral-spatial prior information; specifically, the spectral and spatial differences are both considered in the TV term, and the weight is computed by considering the local neighborhood information in the spatial domain. Alternating direction method of multipliers optimization procedure is extended to solve the presented model. Experimental results demonstrate that the proposed method can remove the mixed noise, enhance the structural information simultaneously, and offer the best performance compared with several state-of-the-art HSI restoration methods.

The large amount of data produced by satellites and airborne remote sensing instruments has posed important challenges to efficient and scalable processing of remotely sensed data in the context of various applications. In this paper, we propose a new big data framework for processing massive amounts of remote sensing images on cloud computing platforms. In addition to taking advantage of the parallel processing abilities of cloud computing to cope with large-scale remote sensing data, this framework incorporates task scheduling strategy to further exploit the parallelism during the distributed processing stage. Using a computation- and data-intensive pan-sharpening method as a study case, the proposed approach starts by profiling a remote sensing application and characterizing it into a directed acyclic graph (DAG). With the obtained DAG representing the application, we further develop an optimization framework that incorporates the distributed computing mechanism and task scheduling strategy to minimize the total execution time. By determining an optimized solution of task partitioning and task assignments, high utilization of cloud computing resources and accordingly a significant speedup can be achieved for remote sensing data processing. Experimental results demonstrate that the proposed framework achieves promising results in terms of execution time as compared with the traditional (serial) processing approach. Our results also show that the proposed approach is scalable with regard to the increasing scale of remote sensing data.

This paper presents a hypserspectral image (HSI) super-resolution method which fuses a low-resolution hyperspectral image (LR-HSI) with a high-resolution multispectral image (HR-MSI) to get high-resolution HSI (HR-HSI). The proposed method first extracts the nonlocal similar patches to form a nonlocal patch tensor (NPT). A novel tensor-tensor product (t-product) based tensor sparse representation is proposed to model the extracted NPTs. Through the tensor sparse representation, both the spectral and spatial similarities between the nonlocal similar patches are well preserved. Then, the relationship between the HR-HSI and LR-HSI is built using t-product which allows us to design a unified objective function to incorporate the nonlocal similarity, tensor dictionary learning, and tensor sparse coding together. Finally, Alternating Direction Method of Multipliers (ADMM) is used to solve the optimization problem. Experimental results on three data sets and one real data set demonstrate that the proposed method substantially outperforms the existing state-of-the-art HSI super-resolution methods.

Pan-sharpening is a common image-fusion method. To improve the quality of fused images, a multilevel deep learning Pan-sharpening method is proposed in this paper. In the training phase, we introduce Coupled Sparse Denoising Autoencorder (CSDA) to reconstruct high-Resolution (HR) multispectral (MS) image from low-Resolution (LR) MS image and HR Panchromatic (Pan) image. CSDA has four networks including LM-HP network, HR-MS network, feature mapping network and fine-tuning network. The hidden features in LM-HP network and HR-MS network as well as the mapping function between the two features are learned through joint optimization. In LM-HP and HR-MS networks, the hidden features of image patch pairs are extracted by the sparse autoencoder. A sparse denoising autoencoder is used to build the nonlinear mapping between the extracted features. In the testing phase, the LR-MS and HR-Pan images patches are fed to the CSDA network to reconstruct the fused HR-MS image. The experimental results show that the proposed method is better than the traditional pans-sharpening methods.

Most traditional compressive sensing (CS) reconstruction methods suffer from the intensive computation caused by iterations. This paper aims at presenting a non-iterative algorithm to reconstruct hyperspectral images (HSI) from patch-based compressively sensed measurements. Our method contains two residual convolutional neural networks. One is reconstruction network for compressive sensing reconstruction and the other is deblocking network for removing the blocky effect, which is caused by patch-based sampling. The reconstruction network can efficiently reconstruct all the bands of HSI jointly, thus the spectral correlation is well preserved. In addition, the deblock performance is enhanced by combining more patches into a larger patch in the deblocking network. Experimental results verify that our method outperforms the state-of-the-art compressive sensing reconstruction methods with patch-based CS measurement.

Recently, the dictionary-aided sparse regression (SR) method for hyperspectral unmixing has received much attention in the field of remote sensing. However, under the assumption that each pixel in the hyperspectral scene can be viewed as a combination of endmembers in the spectral library, most of SR methods ignore the spectral signature mismatches between an actual spectral signature and its corresponding endmember in spectral library. To overcome this problem, we proposed a joint optimizing unmixing model called DSPCSR which includes dictionary sparse pruning and collaborative sparse regression. By exploiting the sparse property of spectral mismatch error and the collaborative sparse property of the abundance matrix, the DSPCSR can provide good robustness and performance. Experiments on the synthetic and real datasets show that the proposed DSPCSR can achieve better performance compared with several state-of-art algorithms.

Low-rank representation has been widely used as a powerful tool in hyperspectral image (HSI) recovery. The existing studies involving low-rank problems are commonly under the nuclear norm penalization. However, nuclear norm minimization tends to over-shrink the components of rank, which leads to modeling bias. In this paper, a new nonconvex penalty is introduced to obtain an unbiased low-rank approximation. In Addition, local spatial neighborhood weighted spectral-spatial total variation (TV) regularization is introduced to preserve spatial structural information. And sparse l1-norm is used as a constraint to sparse noise. Finally, a novel HSI non-convex low-rank relaxation restoration model is proposed. A number of experiments show that the proposed method can effectively remove the mixed-noise, and result in an unbiased estimate with better robustness.

Anomaly detection in hyperspectral images aims to separate the abnormal pixels from the background, and becomes an important application of hyperspectral data processing. Anomaly detection based on Low-Rank and Sparse Representation (LRASR) can detect abnormal pixels accurately. However, with the growth of the hyperspectral data volumes, this algorithm consumes a huge amount of time and computational resources, and needs to be improved accordingly. Spark is a distributed big data processing platform, and is applicable for complex iterative calculations, because of its powerful in-memory computation and efficient task scheduling. Based on Spark, this paper proposes a distributed and parallel LRASR (called DP-LRASR), which first segments hyperspectral images using narrow dependency of resilient distributed datasets, and afterwards, a parallel clustering algorithm is employed to improve the efficiency, remarkably. Experimental results demonstrate that DP-LRASR achieves a good speedup with high scalability, in the premise of remarkable detection accuracy.

Gas plume detection is a challenging task in the field of remote sensing. Due to sensor development, it is now possible to detect and track chemical gas plumes with hyperspectral video sequences (HVS). For the purpose of high detection accuracy, it is important to explore spectral characteristic, and take full advantages of spatial continuity and temporal consistency in HVS. However, the high computational complexity and the large amount of data limit its application in time-critical scenarios. In this paper, we propose a GPU parallel implementation of gas plume detection in HVS, which properly exploits shared memory and intrinsic concurrency of the CUDA blocks, as well as parallel workload assignment and resource allocation. The experimental results demonstrate the proposed GPU parallel method has a considerable acceleration factor while retaining the same detection accuracy compared with the serial and multicore version.

Generally, the improvement in resolution will lead to larger data volume and higher data dimension for Hyperspectral image, which raise a higher requirement for previous image processing algorithms. In this paper, a novel coupled spectral-spatial tensor representation framework (CSSTR) is proposed for denoising of hyperspectral images. Specifically, the proposed method is applied to describe the spectral-spatial features which decomposes a third-order tensor into the sum of several component tensors, with each component tensor being the outer product of a matrix and a vector. Owing to the spatial-spectral constraint fed back from the tensor representation method, CSSTR can capture the structural correlations and inherent feature information of data. Finally, several experiments were conducted to illustrate the advantage of the proposed algorithm.

As the volume and value of hyperspectral remote sensing images increasing, a common hyperspectral imagery repository is urgently required. This paper presents a novel distributed content retrieval method for hyperspectral imagery repository based on cloud computing platform. The proposed method uses a software-as-a-service (SaaS) mode, which provides users with services such as hyperspectral image management, storage and retrieval through a web interface. In order to accelerate the acquisition of the spectral feature information needed in the retrieval process, we optimize the procedure of endmember extraction and the abundance estimation in a distributed way on Spark platform. The generated spectral features meta-data are stored in the MySQL database for imagery retrieval. Finally, we introduce the parallel implementation of distributed retrieval of hyperspectral images based on N-FINDR in details, and evaluate the effectiveness and stability of our proposed method through experiments on real datasets.

Hyperspectral video sequences (HVSs) are well suited for gas plume detection (GPD). The high spectral resolution allows the detection of chemical clouds even when they are optically thin. Processing this new type of video sequences is challenging and requires advanced image and video analysis algorithms. In this paper, we propose a novel method for GPD recorded in HVSs. Based on the assumption that the background is stationary and the gas plume is moving, the proposed method separates the background from the gas plume via a low-rank and sparse decomposition. Furthermore, taking into consideration that the gas plume is continuous in both spatial and temporal dimensions, we include total variation regularization in the constrained minimization problem, which we solve using the augmented Lagrangian multiplier method. After applying the above process to each extracted feature, a novel fusion strategy is proposed to combine the information into a final detection result. Experimental results using real data sets indicate that the proposed method achieves very promising GPD performance.

Anomaly detection plays an important role in remotely sensed hyperspectral image (HSI) processing. Recently, compressive sensing technology has been widely used in hyperspectral imaging. However, the reconstruction from compressive HSI and detection are commonly completed independently, which will reduce the processing's efficiency and accuracy. In this paper, we propose a framework for hyperspectral compressive sensing with anomaly detection which reconstruct the HSI and detect the anomalies simultaneously. In the proposed method, the HSI is composed of the background and anomaly parts in the tensor robust principal component analysis model. To characterize the low-dimensional structure of the background, a novel tensor nuclear norm is used to constrain the background tensor. As the anomaly part is formed by a few anomalous spectra, the anomaly part is assumed to be a tuber-wise sparse tensor. In addition, to enhance the separation of the background and anomaly, we minimize the sum of Mahalanobis distance of the background pixels. Experiments on four HSIs demonstrate that the proposed method outperforms several state-of-the-art methods on both reconstruction and anomaly detection accuracies.

A single sensor camera can capture scenes by means of color filter array. Each pixel samples only one of the three primary colors. Color demosaicking (CDM) is a process of reconstruction a full color image from this sensor data. In this paper, we propose a novel CDM scheme based on learned simultaneous sparse coding over nonlocal tensor representation. First, similar 2D patches are grouped to form a three-order tensor, that is, 3D array. Then, three sub-dictionaries, which characterize the coherent structures that appear in each dimension of the grouped tensor, are learned jointly by using Tucker decomposition. The consequent coefficient tensor is imposed by the grouped-block-sparsity constraint, which forces the similar patches to share the same atoms of the dictionaries in their sparse decomposition. Experimental results demonstrate the effectiveness both in the average CPSNR and visual quality.

Two-stage non-local methods represented by the Poisson non-local means (PNLM) method [1] and the non-local principal component analysis (NLPCA) [2] method perform well for photon-limited Poisson image reconstruction, where patch-similarity computation in the non-local reconstruction stage is guided and affected by a pre-reconstructed image obtained at the first stage. In this paper, we propose a new method to provide a better pre-reconstructed image with low computational cost. Firstly, we propose an adaptive method for fitting the linear relationship between image intensity and Poisson parameter; Secondly, we obtain an initial estimated image according to this relationship. Lastly, we obtain new pre-reconstructed image by adjusting the initial estimation according to Skellam distribution. Numerical experiments show that our method can provide a better pre-reconstructed image, and therefore can improve the performance of the PNLM method and reduce the computational cost of the NLPCA method efficiently.

Hyperspectral image (HSI) classification is a popular issue in the domain of remote sensing. The fundamental challenges in HSI classification include small number of training samples, high dimensionality of the hyperspectral data and suitable spatial-spectral features. In this paper, we propose a novel multiple features fusion method for HSI classification based on extreme learning machines (ELM). We extract spectral feature via the principal component analysis (PCA), and extract spatial features via local binary pattern (LBP), Gabor feature and extended multiattribute profile (EMAP). Then we utilize probability voting to fuse the multiple features based on extreme learning machine model. Experiment on real HSI demonstrates that the proposed method is superior to some existing methods and it is suitable for small training sample size conditions.

The hyperspectral image features wide coverage, high dimensional bands and a huge amount of data, which leads to time-consuming computation when processing hyperspectral data. Spark is a distributed big data processing framework, integrated in-memory computation. So Spark is suitable for complex iterative calculation. In order to classify massive hyperspectral data efficiently, the Spark version of the original Spatial Correlation Regularized Sparse Representation Classification (SCSRC) is proposed in this paper. In Distributed Parallel SCSRC (DP-SCSRC), firstly, adjacent hyperspectral image indexes are stored in the same partition of Spark's RDDs to preserve spatial correlation. Secondly, Joint Distributed Matrix (JDM) is created to reduce overhead data synchronization between computing nodes. Experimental results on real hyperspectral data demonstrate that DP-SCSRC achieves a remarkable speedup and is scalable with larger data size.

With the widespread application of hyperspectral image, restoration has become an important branch of hyperspectral data processing. Although principal component analysis (PCA), total variation (TV) and soft-thresholding algorithm (PCATV-ST) could restore image effectively, a computing bottleneck may occur with the increase of hyperspectral data volumes. In order to solve the restoration problem effectively and accurately, we put forward a distributed parallel cloud implementation of PCATV-STbased on Spark (PCATV-ST_DP). The proposed method optimizes the way of data transmission and improves the shared memory space, as well as optimizes matrix multiplication. Finally, our experimental results, conducted on real hyperspectral datasets, reveal very high performance for the proposed distributed parallel method.

Nonlocal image representation methods, including group-based sparse coding and block-matching 3-D filtering, have shown their great performance in application to low-level tasks. The nonlocal prior is extracted from each group consisting of patches with similar intensities. Grouping patches based on intensity similarity, however, gives rise to disturbance and inaccuracy in estimation of the true images. To address this problem, we propose a structure-based low-rank model with graph nuclear norm regularization. We exploit the local manifold structure inside a patch and group the patches by the distance metric of manifold structure. With the manifold structure information, a graph nuclear norm regularization is established and incorporated into a low-rank approximation model. We then prove that the graph-based regularization is equivalent to a weighted nuclear norm and the proposed model can be solved by a weighted singular-value thresholding algorithm. Extensive experiments on additive white Gaussian noise removal and mixed noise removal demonstrate that the proposed method achieves a better performance than several state-of-the-art algorithms.

Recently, it has been an increasing interest in modeling abnormal temporal dynamics of functional interactions in psychiatric disorders. However, the accuracy of differentiating attention-deficit/hyperactivity disorder (ADHD) children form normal children has still much space for improvement. To further improve the accuracy, the key issue is to extract more effective features from original fMRI data. In this paper, we propose a novel local feature extraction method named Local Binary Encoding Method (LBEM) that can effectively characterize functional interaction patterns (FIPs). In particular, we show that the proposed method can well discriminate the functional interaction abnormalities, which is composed of a Bayesian connectivity change point model, a local feature extraction method and a kernel Extreme Learning Machine (ELM)-based classifier. The experiment on a real dataset of 23 ADHD children and 45 normal control (NC) children has shown that our method achieved better classification performance compared to the existing methods.

Thanks to the fast development of sensors, it is now possible to acquire sequences of hyperspectral images. Those hyperspectral video sequences (HVS) are particularly suited for the detection and tracking of chemical gas plumes. In this paper, we present a novel gas plume detection method. It is based on the decomposition of the sequence into a low-rank and a sparse term, corresponding to the background and the plume, respectively, and incorporating temporal consistency. To introduce spatial continuity, a post processing is added using the Total Variation (TV) regularized model. Experimental results on real hyperspectral video sequences validate the effectiveness of the proposed method.

In this paper, a novel kernel low rank representation (KLRR) method for hyperspectral image classification is proposed. Firstly, we extract the global structure characteristics information of the hyperspectral image based on low rank representation (LRR), then use it as a prior to constrain the recovery coefficient matrix. In order to further improve the classification efficiency and deal with the linearly non-separable problems directly, we transformed the linear LRR classifier to a non-linear one by feature space transformation using the kernel trick. The proposed algorithm is solved by the ADMM algorithm effectively. Experimental results on real hyperspectral images demonstrate that the proposed method outperforms many state-of-the-art approaches.

Target detection plays an important role in the field of hyperspectral image (HSI) remote sensing. In this letter, a novel matched subspace detector based on low-rank regularized least squares (LRLS-MSD) is proposed for hyperspectral target detection. As pixels in an HSI have global correlation and can be represented in subspace, the low-rank regularization is introduced in the least squares model. An effective algorithm is presented to solve the problem. Then, the detection results are generated according to the generalized likelihood ratio test with statistical hypotheses. The experimental results suggest an advantage of the low-rank regularization over other classical target detection methods.

A novel method for anomaly detection in hyperspectral images (HSIs) is proposed based on low-rank and sparse representation. The proposed method is based on the separation of the background and the anomalies in the observed data. Since each pixel in the background can be approximately represented by a background dictionary and the representation coefficients of all pixels form a low-rank matrix, a low-rank representation is used to model the background part. To better characterize each pixel's local representation, a sparsity-inducing regularization term is added to the representation coefficients. Moreover, a dictionary construction strategy is adopted to make the dictionary more stable and discriminative. Then, the anomalies are determined by the response of the residual matrix. An important advantage of the proposed algorithm is that it combines the global and local structure in the HSI. Experimental results have been conducted using both simulated and real data sets. These experiments indicate that our algorithm achieves very promising anomaly detection performance.

Principal component analysis (PCA) is an important method for feature extraction of hyperspectral remote sensing image. With the development of hyperspectral sensors, the magnitude of hyperspectral data grows quickly, and it is a challenging task to efficiently reduce the data dimension and compress massive data volumes in hyperspectral imaging. In this paper, a distributed parallel optimization of PCA algorithm (PCA_DP) is presented on cloud computing architecture. The realization of the proposed method using Apache Hadoop and MapReduce model is described and evaluated. The experiments conducted on real hyperspectral images of different sizes, demonstrate significant acceleration factor of PCA_DP. It is efficient for massive hyperspectral data processing.

This paper presents a novel method for anomaly detection in hyperspectral image(HSI) based on low-rank representation. In the observed HSI, the anomalies can be separated from the background. Since each pixel in the background can be approximately represented by a background dictionary, and the representation coefficients of the background pixels are correlative, a low-rank representation model is used to model the background part. Besides, to gain robust representation coefficient, the sum-to-one constraint is added. The advantage of the proposed low-rank representation sum-to-one (LRRSTO) method is that it makes use of the global correlation of the background and strength the robustness of the representation. Experiments results have been conducted using both simulated and real data sets. These experiments indicated that our algorithm achieves very promising performance.

Image dehazing is an important issue that interests both image processing and computer vision. In this study, image dehazing is modelled as an example‐based learning problem, and a novel dehazing algorithm using two‐dimensional (2D) canonical correlation analysis (CCA) is proposed. By assuming that the hazy‐free image patches are smooth and the pixel intensities in the same patch are approximate to constant, the authors deduce an underlying linear correlation between the observed hazy image patches and corresponding transmission patches. By maximising the correlation between the patch‐pairs of hazy image and corresponding transmission map, 2D CCA is able to learn a subspace to reconstruct the reliable transmission. Thus, given a test hazy image, the transmission map is aggregated by the nearest neighbour patches in the subspace and then globally refined by a local mean adaptive guided filter. The final hazy‐free image is obtained by using the dichromatic atmospheric model. Experimental results demonstrate the efficiency of the proposed method in single image dehazing.

The authors propose a fast and effective solution for automatic segmentation of white matter lesions by using T1 and fluid‐attenuated inversion recovery (FLAIR) image modalities with no need for manual segmentation and atlas registration. Initially, a brain tissue segmentation method is used to segment the T1 image into cerebrospinal fluid (CSF), grey matter and white matter. Based on the obtained tissue segmentation results, the region of interest (ROI) of the FLAIR image is created by subtracting the CSF from the FLAIR image. Subsequently, the authors calculate the z ‐score of the intensities in the ROI and define a threshold to perform a preliminary identification of abnormalities from normal tissues. The abnormalities obtained at this stage are used as the prior knowledge for the modified level‐set technique. The proposed level set method here is applied based on local Gaussian distribution to precisely detect the boundaries of the white matter lesions in the ROI. The level set method based on local Gaussian distribution fitting energy is robust to the intensity inhomogeneity of MR data and therefore capable of precisely extracting the boundaries of white matter lesions. Experimental analysis and quantitative comparisons with the peak‐seeking and state‐of‐the‐art white matter lesion segmentation (WMLS) techniques demonstrate that the algorithm is a stable and effective approach which significantly outperforms other trusted solutions for white matter lesion segmentation.

Colour image enhancement is an important preprocessing phase of many image analysis tasks such as image segmentation, pattern recognition and so on. This study presents a new local brightness adaptive variational model using Wasserstein distance for colour image enhancement. Under the perceptually inspired variational framework, the proposed energy functional consists of an improved contrast energy term and a Wasserstein dispersion energy term. To better adjust image dynamic range, the authors propose a local brightness adaptive contrast energy term using the average brightness of image local patch as the local brightness indicator. To restore image true colours, a Wasserstein distance‐based dispersion energy term is used to measure the statistical similarity between the original image and the enhanced image. The proposed energy functional is minimised by using a gradient descent algorithm. Two objective measures are used to quantitatively measure the enhancement quality. Experimental results demonstrate the efficiency of the proposed model for removing colour cast and haze, enhancing contrast, recovering details and equalising low key images.

A deep neural network (DNN)-based new pansharpening method for the remote sensing image fusion problem is proposed in this letter. Research on representation learning suggests that the DNN can effectively model complex relationships between variables via the composition of several levels of nonlinearity. Inspired by this observation, a modified sparse denoising autoencoder (MSDA) algorithm is proposed to train the relationship between high-resolution (HR) and low-resolution (LR) image patches, which can be represented by the DNN. The HR/LR image patches only sample from the HR/LR panchromatic (PAN) images at hand, respectively, without requiring other training images. By connecting a series of MSDAs, we obtain a stacked MSDA (S-MSDA), which can effectively pretrain the DNN. Moreover, in order to better train the DNN, the entire DNN is again trained by a back-propagation algorithm after pretraining. Finally, assuming that the relationship between HR/LR multispectral (MS) image patches is the same as that between HR/LR PAN image patches, the HR MS image will be reconstructed from the observed LR MS image using the trained DNN. Comparative experimental results with several quality assessment indexes show that the proposed method outperforms other pan-sharpening methods in terms of visual perception and numerical measures.

In this letter, we present an efficient parallel implementation of composite kernels in support vector machines (SVMs) for hyperspectral image (HSI) classification. Our implementation makes effective use of commodity graphics processing units (GPUs). Specifically, we port the calculation of composite kernels to GPUs, perform intensive computations based on NVidia's compute unified device architecture, and execute the rest of the operations related with control and small data calculations in the CPU. Our experimental results, conducted using real hyperspectral data sets and NVidia GPU platforms, indicate significant improvements in terms of computational effectiveness, achieving near-real-time performance of spatial-spectral HSI classification for the first time in the literature.

In this letter, a real-time implementation of the logistic regression via variable splitting and augmented Lagrangian (LORSAL) algorithm for sparse multinomial logistic regression is presented on commodity graphics processing units (GPUs) using Nvidia's compute unified device architecture. The proposed parallel method properly exploits the GPU architecture at the low level, including its shared memory, and takes full advantage of the computational power of GPUs to achieve real-time classification performance of hyperspectral images for the first time in the hyperspectral imaging literature. Our experimental results reveal remarkable acceleration factors and real-time performance, while retaining exactly the same classification accuracy with regard to the serial and multicore versions of the classifier.

This paper presents a new approach for hyperspectral image classification exploiting spectral-spatial information. Under the maximum a posteriori framework, we propose a supervised classification model which includes a spectral data fidelity term and a spatially adaptive Markov random field (MRF) prior in the hidden field. The data fidelity term adopted in this paper is learned from the sparse multinomial logistic regression (SMLR) classifier, while the spatially adaptive MRF prior is modeled by a spatially adaptive total variation (SpATV) regularization to enforce a spatially smooth classifier. To further improve the classification accuracy, the true labels of training samples are fixed as an additional constraint in the proposed model. Thus, our model takes full advantage of exploiting the spatial and contextual information present in the hyperspectral image. An efficient hyperspectral image classification algorithm, named SMLR-SpATV, is then developed to solve the final proposed model using the alternating direction method of multipliers. Experimental results on real hyperspectral data sets demonstrate that the proposed approach outperforms many state-of-the-art methods in terms of the overall accuracy, average accuracy, and kappa (k) statistic.

This paper proposes a new hyperspectral images superresolution (HSI-SR) method based on compressive sensing (CS) theory, spatial sparsity and spectral similarity prior. First, according to sparsity and incoherence of CS theory, we propose a new dictionary learning method, ensuring that the learned dictionary not only has less dimensionality to speed up the sparse decomposition, but also satisfies sparsity well. Then, we introduce the spatial sparsity and spectral similarity regularizations into HSI-SR model, which can recover the spatial information effectively and preserve the spectral information well. The experimental results show the proposed method outperforms other well-known methods in terms of both objective measurements and visual evaluation.

Kernel regression has been shown to be a powerful image denoising technique. In this paper, a three-dimensional (3-D) kernel regression hyperspectral image (HSI) denoising mechanism is proposed. The main contributions of this paper can be summarized as follows: Three orientation vectors and the corresponding coefficients are presented, which are adaptive for each pixel based on the innovation of 2-D structure tensor. An adaptive-driven 3-D tensor matrix is proposed for kernel regression, in which the spatial geometric structure and spectrum continuity are both considered. The proposed adaptive kernel regression is applied to HSI denoising. Both stimulated and real data experiments indicate that the proposed method can work well in detail preservation and noise removal.

This paper presents a novel method to apply homogeneous areas priors adaptively for hyperspectral image classification. Firstly, support vector machine algorithm is utilized to obtain the posterior probability distributions by training the spectral information of the samples. Then, the homogeneous areas generated from the watershed segmentation results are used as new spatial priors. By using Markov Random Field model, we can integrate the spectral information and spatial information which includes the homogeneous areas priors in a unified framework. Compared with neighborhood-generated Markov random field, the adaptive priors strengthen to enforce the segmentation results in homogeneous areas of the neighborhood belong to the same class. Finally, the maximum a posterior segmentation is computed by min-cut based optimization algorithm.

This letter presents a postprocessing algorithm for a kernel sparse representation (KSR)-based hyperspectral image classifier, which is based on the integration of spatial and spectral information. A pixelwise KSR is first used to find the sparse coefficient vectors of the hyperspectral image. Then, a sparsity concentration index (SCI) rule-guided semilocal spatial graph regularization (SSG), called SSG+SCI, is proposed to determine refined sparse coefficient vectors that promote spatial continuity within each class. Finally, these refined coefficient vectors are used to obtain the final classification map. Compared with previous approaches based on similar spatial-spectral postprocessing strategies, SSG+SCI clearly outperforms their results in terms of accuracy and the number of training samples, as it is demonstrated with two real hyperspectral images.

In this paper, we propose a variational Bayesian method for Retinex to simulate and interpret how the human visual system perceives color. To construct a hierarchical Bayesian model, we use the Gibbs distributions as prior distributions for the reflectance and the illumination, and the gamma distributions for the model parameters. By assuming that the reflection function is piecewise continuous and illumination function is spatially smooth, we define the energy functions in the Gibbs distributions as a total variation function and a smooth function for the reflectance and the illumination, respectively. We then apply the variational Bayes approximation to obtain the approximation of the posterior distribution of unknowns so that the unknown images and hyperparameters are estimated simultaneously. Experimental results demonstrate the efficiency of the proposed method for providing competitive performance without additional information about the unknown parameters, and when prior information is added the proposed method outperforms the non-Bayesian-based Retinex methods we compared.

We propose a structured feature active contour model based on the level set method for image segmentation. We make the following two contributions. First, an adaptive data fitting term detects intensity variation based on the direction and global information. Second, integrated with a structured gradient vector flow (SGVF) method, we formulate a new regularization term with respect to the level set function via the duality formulation to penalize the length of active contour. We compare the proposed method to the classical active contour methods and demonstrate through the experiments on synthetic and medical images.

In this paper, we propose an improved version of CS-based model for inverse synthetic aperture radar (ISAR) imaging, which can sustain strong clutter noise and provide high quality images with extremely limited measurements. Different from traditional l1norm based CS ISAR imaging models, the essential of our model is to use the Gini index to measure the sparsity of signals. We also develop an iteratively re-weighted algorithm to find the solution of our model and reconstruct sparse signals from compressed samples. Experimental results of point targets and complex scene show that our approach significantly reduces the number of measurements needed for exact reconstruction and effectively suppresses the noise and outperforms l1norm based methods.

In this paper, we propose a new model for hyperspectral image classification using spectral-spatial information. The main contributions of our paper are that we exploit the posterior distribution from both spectral and spatial information in the original hyperspectral data. The association potential in our model is a sparse multinomial logistic regression (SMLR) classifier and the interaction potential is a spatial-relevant total variation (TV) constraint upon the posterior distribution itself which encourages neighboring pixels to belong to the same class. The proposed model is solved by the alternating direction method of multipliers (ADMM); we enhance the spatial smoothness by expanding the spatial information from the fixed labeled samples to the whole data to further improve the classification accuracy. Experimental results with real hyperspectral data set validate that our proposed approach provides good performance when compared with other state-of-the-art methods.

Hyperspectral unmixing is a typical problem of blind source separation, which can be solved by nonnegative matrix factorization (NMF). Sparsity based NMF will increase the efficiency of unmixing, but its computational complexity limits the possibility of utilizing it in time-critical applications. In this paper, method of parallel hyperspectral unmixing based on sparsity constrained nonnegative matrix factorization on Graphics Processing Units (CSNMF-GPU) is investigated and compared in terms of both accuracy and speed. The realization of the proposed method using Compute Unified Device Architecture (CUDA) on GPU are described and evaluated. The experimental results comparing with the serial implementations based on both simulated and real hyperspectral data demonstrate the effectiveness of the proposed parallel optimization approach.

We propose a novel compressive sensing (CS) image reconstruction method based on iterative directional total variation (TV) refinement. As is generally known, classical TV-based CS reconstruction methods tend to produce over-smoothed image edges and texture details, since they favor piece-wise constant solutions. Hence, directional TV is introduced to describe the sparsity of the image gradient in order to overcome this drawback. However, it is difficult to estimate orientation field robustly and accurately from CS measurements. Inspired by vectorial ROF model, orientation field refinement model is presented and introduced into CS reconstruction. Extended experiments show that the proposed CS reconstruction method has a better improvement in the quality of the reconstructed image details over related TV-based CS reconstruction methods.

Sparsity is an intrinsic property of hyperspectral images, which means that the collected pixels can be represented by a part of materials. In this paper, a new sparsity based method for hyperspectral unmixing is proposed, referred to as the constrained sparse nonnegative matrix factorization (CSNMF). First, a novel sparse term which is explored to measure the sparsity of hyperspectral images is introduced to restrict the abundances. Second, minimum distance constraint which is convex is applied to restrict the endmembers. Then the alternating direction method of multipliers (ADMM) is used to solve the proposed CSNMF. The experimental results based on both synthetic mixtures and a real image scene demonstrate the effectiveness of the proposed approach.

Compressive Sensing Deconvolution (CS Deconvolution) is a new challenge problem encountered in a wide variety of image processing fields. A compound variational regularization model which combined total variation and curve let-based sparsity prior is proposed to recovery blurred image from compressive measurements. We propose a novel fast algorithm using variable-splitting and Dual Douglas-Rachford operator splitting methods. Experiments demonstrate our proposed algorithm can obtain high-resolution data from highly incomplete measurements.

In this paper, a new region-based active contour model is proposed for image segmentation. Different from the general region-based active contour models, this model partitions the regions of interests in images depending on the local statistics of the intensity and the magnitude of gradient in the neighborhood of the contour. Inspired by the structure tensor method, an improved regularization term is defined through the duality formulation to penalize the length of region boundaries. Experiments on medical images demonstrate the proposed model outperforms the classical segmentation models in terms of efficiency and accuracy.

Image restoration is a core topic of image processing. In this paper, we consider a variational restoration model consisting of Poisson singular integral (PSI) and curvelet-type decomposition space seminorm as regularizer. The PSI is used to impose a priori constraint on appropriate Lipschitz spaces, wherein a wide class of nonsmooth images can be accommodated. The seminorm of curvelet-type decomposition space is equivalent to the weighted curvelet coefficients which optimal represent smooth and edge parts of image with sparsity. We propose efficient algorithm to solve the optimization problem based on the Douglas-Rachford splitting (DRS) technique. Experimental results demonstrate that our proposed method can preserve important image features, such as edges and textures.

Multiplicative noise removal has been drawn a greatly attention recently. Firstly, this paper proposes a new non-convex variational model for multiplicative noise removal under the Weberized TV regularization framework. Then we propose and study another surrogate strictly convex objective functional for Weberized TV regularization based multiplicative noise removal model. Finally, we adopt the recently proposed inverse scale space approach to estimate the underlying image under total variation (TV) regularization, in which a relaxation technique with two evolution equations is applied. Our experimental results show that the quality of images denoised is quite good and the detail information of restored images is well preserved by the proposed algorithm.

Compressed sensing (CS) is a new area of signal processing for simultaneous signal sampling and compression. Most of existing methods for CS image reconstruction are suitable for piecewise smooth image, but do not behave well on texture-rich natural image. In this paper, a new optimization problem for CS image reconstruction is proposed, in which different regularization terms are introduced for different morphological components of image. Furthermore, an alternating iterative algorithm is presented to solve the relevant optimization problem. Experimental results show that the proposed method can be applied to reconstruct texture-rich images besides piecewise smooth ones, and outperforms the existing methods on preserving detail feature.

A generalized Mumford-Shah model for color image restoration and edge detection is established in this paper. First the color images are considered as the manifold based on the "surface based method", then a new physical quantity in the form of vector product, which describes the differences of the gradient between different channels, is introduced into the regularization term of the "object", then a generalized energy functional is proposed. Finally, we proposed a fast PDE's numerical iterative algorithm utilizing the steepest descent method and half-point scheme. Experimental results show the proposed model has good ability to overcoming the bad phenomena of color fluctuations along edges caused in the straightforward extension of the original Mumford-Shah model in vectorial cases.