Animation production workflows centred around motion capture techniques require animators to edit motions based on a set of keyframes. However, most existing keyframe selection methods are optimisation-based, which suffer from the issues of flexibility and efficiency. In this paper, a novel deep reinforcement learning method with dual agents are proposed for unsupervised keyframe selection. First, an S-Agent and an R-Agent evaluate the actions of selection and refinement, respectively. A deep spatio-temporal network, namely graph keyframe evaluation network (GKEN), is proposed for the agents. Then, an animation specified reward is devised based on reconstruction, which fulfills three important properties of the animation workflow: incremental reward, order insensitivity and non-diminishing returns. During the inference, it is no longer necessary to compute the reconstruction, which significantly decreases the run-time latency. Experiments on the CMU MoCap dataset demonstrate the efficiency of the proposed method without clearly compromising the effectiveness compared with the state-of-the-art methods. • A deep reinforcement learning with dual-agent to identify motion keyframes. • A spatio-temporal deep agent with graph convolutions and transformers. • Comprehensive experiments and human demonstrations for MoCap keyframing.

Large Language Models (LLMs) increasingly employ alignment techniques to prevent harmful outputs. Despite these safeguards, attackers can circumvent them by crafting prompts that induce LLMs to generate harmful content. Current methods typically target exact affirmative responses, suffering from limited convergence, unnatural prompts, and high computational costs. We introduce semantic representation attacks, a novel paradigm that fundamentally reconceptualizes adversarial objectives against aligned LLMs. Rather than targeting exact textual patterns, our approach exploits the semantic representation space that can elicit diverse responses that share equivalent harmful meanings. This innovation resolves the inherent trade-off between attack effectiveness and prompt naturalness that plagues existing methods. Our Semantic Representation Heuristic Search (SRHS) algorithm efficiently generates semantically coherent adversarial prompts by maintaining interpretability during incremental search. We establish rigorous theoretical guarantees for semantic convergence and demonstrate that SRHS achieves unprecedented attack success rates (89.4% averaged across 18 LLMs, including 100% on 11 models) while significantly reducing computational requirements. Extensive experiments show that our method consistently outperforms existing approaches.

Hyperspectral imaging offers significant potential for precise object tracking, yet the scarcity of dataset volumes specifically tailored for hyperspectral tracking algorithms hinders progress, particularly for deep models with complex structures. Additionally, current deep learning-based hyperspectral trackers typically enhance model accuracy via online or adversarial learning, adversely affecting tracking speed. To address these challenges, this paper introduces the Constrained Object Adaptive Learning hyperspectral Tracker (COALT), an effective parameter-efficient fine-tuning tracker tailored for hyperspectral tracking. COALT integrates Pixel-level Object Constrained Spectral Prompt (POCSP) and Temporal Sequence Trajectory Prompt (TSTP) through Adaptive Learning with Parameter-efficient Fine-tuning (ALPEFT), enabling a transformer-based tracker to capture detailed spectral features and relationships in hyperspectral image sequences through trainable rank decomposition matrices. Specifically, POCSP is designed to retain optimal spectral information with low internal correlation and high object representativeness, enabling rapid image reconstruction. Then, the most representative spectral template and search are fused into a single stream as spectral prompts for the Encoder and Decoder layers. Concurrently, the previous coordinates within the same sequence are tokenized and utilized as temporal prompts by TSTP in the decoder layers. The model is trained with ALPEFT to optimize spectral information learning, which substantially reduces the number of training parameters, alleviating overfitting issues arising from limited data. Meanwhile, the proposed tracker not only retains the ability of pre-trained model to estimate object trajectories in an autoregressive manner but also effectively utilizes spectral information and enhances target location perception during the fine-tuning process. Extensive experiments and evaluations are conducted on two public hyperspectral tracking datasets. The results demonstrate that the proposed COALT tracker achieves satisfactory performance with leading processing speed. The code will be available at https://github.com/PING-CHUANG/COALT.

Hyperspectral imaging technology is of crucial importance to improve the performance of object tracking in many remote sensing surveillance areas. Previous methods primarily focused on feature fusion strategies by employing additional enhancement modules. However, these methods commonly lack contextual understanding to distinguish the target from the background and totally ignore the category information of the targets. To address these limitations, a novel hyperspectral object tracker is proposed to incorporate context-aware learning and category consistency tracker (CCTrack), which can adaptively learn context-aware representations in hyperspectral scenarios to obtain global target information with memory storage, while constructing an interframe category consistency constraint to enhance tracking process. Specifically, CCTrack integrates an adaptive context-aware learning (ACL) mechanism, which includes a feature decoupling module (FDM) to extract specific representations from decoupled features, and a Mamba layer to retain and update long-range dependencies. To align with prior knowledge of target recognition and motion patterns, an alignment transformation module (ATM) is employed with the ACL mechanism, fully leveraging spatial-spectral representations. In addition, category consistency constraint modules (C3Ms) are introduced to enforce category consistency across frames by computing the similarities between the target features and the corresponding category name, serving as the constraint to improve tracking performance. Extensive experiments over the hyperspectral object tracking (HOT) benchmark covering various remote sensing scenarios demonstrate that CCTrack outperforms state-of-the-art methods by a significant margin.

Few-shot object detection (FSOD) enables the detection of novel-class objects in remote sensing images (RSIs) with limited labeled samples. Although convolutional neural networks (CNNs) are commonly used for this task, they suffer from two inherent constraints. First, their limited local receptive field fails to capture global context within a single image and the relational dependencies between query and support images. Second, an additional feature alignment mechanism is typically required to bridge the gap between query and support images. To address these challenges, this work introduces a novel cross-attention and multicorrelation aided transformer (CAMCFormer) FSOD framework tailored for global feature representation and multicorrelation modeling in complex and large-scale RSIs. Specifically, a long-distance cross-attention module (LDCAM) is devised to capture dependencies between distant elements across query and support images at each feature extraction layer. This module facilitates the exchange of contextual information between images, resulting in more comprehensive feature representations and eliminating the need for separate feature alignment and fusion modules. Multicorrelation aided heads (MAHs) are constructed to enhance detection performance further to model various relational aspects, i.e., channel-correlation detection head (CCDH), spatial-correlation detection head (SCDH), and cross-attention detection head (CADH). These aided heads contribute to more robust and accurate classification and localization. Comprehensive experiments have been conducted, demonstrating the superiority of the proposed framework compared to several state-of-the-art detectors, highlighting its potential as an effective solution for FSOD in remote sensing scenarios.

In recent years, graph convolutional networks (GCNs) have gained increasing attention in hyperspectral image (HSI) classification due to their good ability to model the pairwise relationships between two pixels. However, it is difficult to effectively model more complex relationships among multiple pixels with simple graphs. To solve this problem, we propose a novel tensorized high-order hypergraph convolutional network (TH2GCN) for HSI classification. Specifically, the hypergraph structure is employed to effectively model complex spatial relationships between pixels in HSIs, and we propose a new tensor-based algebraic representation of hypergraphs as a powerful strategy for describing the high-order interaction structures of the hypergraph. Besides, by extending the adjacency matrix-based GCN to the tensor domain and exploiting the tensor decomposition, the TH2GCN method is designed to efficiently extract high-order discriminative information from the hypergraph at low complexity for improving HSI classification performance. Furthermore, the construction of the adjacency tensor on all the data requires a huge amount of memory, especially for large-scale remote sensing images. To this end, the TH2GCN is trained and tested for HSI data in a minibatch fashion. Experimental results on three HSI datasets prove that the performance of the proposed method outperforms the comparison methods.

Spectral variability inevitably presents in hyperspectral images (HSIs), resulting in significant unmixing errors when using the conventional linear mixture model (LMM). Though several variants of LMM have been proposed to encounter such spectral variability, they cannot well model the complex characteristics of spectral variability, and the performance of these variants strongly depends on the prior knowledge of the scene. In this article, spectral variability within an image is classified into class-dependent variability and class-independent one, which can be tackled by a novel fully linear mixture model (FLMM) introducing a class-dependent multiplicative scaling term, a class-dependent additive perturbation term, and a class-independent variability term into the conventional LMM. Moreover, a spectral variability-aware cascaded autoencoder (SVACA) is designed to realize the automatic learning and representation of unmixing targets and spectral variability in different hyperspectral scenarios, which consists of a class-independent variability autoencoder and a cascaded class-dependent variability autoencoder. Such a network is able to handle different spectral variability autonomously without any scene prior by parallel inference structure. Experimental results over synthetic and real hyperspectral datasets demonstrate that the proposed SVACA network not only outperforms several state-of-the-art unmixing networks but also presents a stronger capability to handle spectral variability within HSIs.

Deep learning-based object detection algorithms have achieved significant success in the field of computer vision. However, the wide range of target sizes in remote sensing images poses a challenge for single algorithms to detect objects of varying sizes effectively. To address this issue, this paper proposes an end-to-end object detection algorithm for remote sensing images based on Scale Adaptive and Frequency Fusion DETR (SAFF-DETR), which designs a frequency feature enhancement and fusion mechanism to handle targets of varying sizes within a single framework. First, in order to improve the Transformer-based detectors’ ability to small targets, a multibranch representation fusion (MRF) module is proposed to fuse shallow layer frequency representations, boosting the network’s ability to perceive small targets. Furthermore, Cross-layer Spatial and Channel Frequency Attention (CSFA and CCFA) is designed to enable efficient frequency feature interaction across multi-scale features, enhancing the representation capability for targets of different sizes. Moreover, by integrating the two aforementioned attention mechanisms, the Cross-layer Channel-Spatial-wise Frequency Fusion (CCSFF) structure is introduced to realize global feature interactions in one step without repetitive up and down-sampling operations, by which patch division-based Transformer architecture is designed to enhance scale adaptability for object detection. Experimental results over several benchmark datasets demonstrate that the proposed SAFF-DETR can handle extremely varying-size targets and outperforms several SOTA algorithms. The source code will be available at https://github.com/Jianglin-Zhao/SAFF-DETR.

Transformer architecture has demonstrated significant potential in hyperspectral object tracking by leveraging global correlation learning to accurately represent the data distribution. However, existing hyperspectral object trackers based on transformer models typically rely on costly pre-trained models, making them prone to crashing due to overfitting when tuned on small-scale hyperspectral videos, greatly limiting their performance. To address this challenge, in this paper, a Hybrid Transformer with Adaptive Content and Position Embedding (HTACPE) tracker is proposed to improve the learning efficiency of the tracking model, and fully explore the spectral-spatial information. Specifically, an Adaptive Content and Position Embedding Module (ACPEM) is designed to dynamically learn the balance between focusing on positional and content-based information, which allows the model to effectively handle datasets of various sizes. To enhance the spectral-spatial information, a Spectral Grouping Module (SGM) is designed to learn the highfrequency information in complex scenarios, thereby enhancing diversified features. It operates in parallel with the ACPEM feature learning module. Furthermore, a Dynamic Reliability Refinement Module (DRRM) is incorporated to address challenges related to accurate object position perception, iteratively refining prediction parameters to enhance the reliability of the model. Extensive experiments demonstrate that the proposed HTACPE achieves satisfactory tracking performance both qualitatively and quantitatively, especially with insufficient training data.

The reconstruction from three to dozens of spectral bands, known as spectral super resolution (SSR) has achieved remarkable progress with the continuous development of deep learning. However, the reconstructed hyperspectral images (HSIs) still suffer from the spatial degeneration due to the insufficient retention of high-frequency (HF) information during the SSR process. To remedy this issue, a novel Wavelet-based Hybrid Asymmetric Network (WHANet) is proposed to establish a RGB-to-HSI translation in wavelet domain, thus reserving and emphasizing the HF features in hyperspectral space. Basically, the backbone is designed in a hybrid asymmetric structure that learns the exact representations of decomposed wavelet coefficients in hyperspectral domain in a parallel way. Innovatively, a CNN-based HF reconstruction module (HFRM) and a transformer-based low frequency (LF) reconstruction module (LFRM) are delicately devised to perform the SSR process individually, which are able to process the discriminative wavelet coefficients contrapuntally. Furthermore, a hybrid loss function incorporated with the Fast Fourier loss (FFL) is proposed to directly regularize and emphasis the missing HF components. Eventually, experimental results over three benchmark datasets and one remote sensing dataset demonstrate that our WHANet is able to reach the state-of-the-art performance quantitatively and qualitatively.

Deep Neural Networks (DNNs) have demonstrated remarkable effectiveness in remote sensing (RS) image processing. However, they remain vulnerable to adversarial examples, which are generated by adding tiny but purposeful perturbations to clean examples. Such vulnerabilities in critical applications like environmental monitoring and urban planning can lead to significant negative consequences. To mitigate the interference of adversarial examples on DNNs, in this paper, a feature decoupling based adversarial examples detection (FD-AED) method for RS images is proposed, where non-robust features are employed in the detection process. Specifically, a loss function is designed for the de-coupler to disentangle the features into robust and non-robust features. Non-robust features are particularly useful because they often contain subtle clues that distinguish between clean and adversarial examples. By focusing on these non-robust features, the adversarial example detector can more effectively capture the differences between clean and adversarial examples. Experimental results indicate that the proposed FD-AED method effectively decouples robust and non-robust features, achieving more precise and reliable detection of adversarial examples.

Few-shot object detection (FSOD) on remote sensing images (RSIs) has garnered significant research interest due to its ability to detect novel classes using very few training examples from challenging remote sensing scenarios. Meta-learning FSOD methods, based on Faster R-CNN and YOLO structures, utilize a two-branch Siamese network as the backbone and compute the similarity between image regions for effective detection. However, almost all methods rely on extracting features using convolutional neural networks (CNNs). Inspired by the improved performance of transformer backbones for downstream tasks, a transformer-based FSOD method is proposed, which employs a transformer backbone with asymmetric-batched cross-attention for the two-branch feature extraction. Our model can improve the classification performance by introducing a Multi-Relation Matching (MRM) head for FSOD to enhance the similarity relation matching learning between two branches. Comprehensive experiments on DIOR benchmarks demonstrate the effectiveness of our model.

Few-shot object detection on remote sensing images aims to detect novel objects with limited annotated samples. Data augmentation has become an effective approach in addressing the issue of few-shot object detection by increasing the number of samples. However, existing methods mainly employ data augmentation in the image domain and do not fully utilize the semantic feature information. In this paper, we introduce a method called diverse regional feature generation for few-shot object detection on remote sensing images, which introduces the WGAN model as the feature generator to generate diverse positive and negative features for the base and novel classes. Our method contains the intra-class diversity module and the inter-class dispersion module to train the classification head. Specifically, the intra-class diversity module is to generate diverse visual features for each category, while the inter-class dispersion module aims to push the features from different categories far away to model the real data distribution. The proposed algorithm is evaluated on the publicly available DIOR dataset and the experimental results show that our method significantly outperforms the other state-of-the-art methods.

A cycle-consistent sparse unmixing network based on deep image prior (C2SU-DIP) is proposed in this paper, to reduce the complexity of sparse unmixing (SU) algorithm and the loss of details in hyperspectral images (HSIs) simultaneously. In the proposed C2SU-DIP network, the complex design of regularization terms in sparse unmixing is avoided, meanwhile, details of abundances are effectively retained. It employs DIP-based sparse unmixing network as the backbone, and the learning process of the network replaces the regularization term design. Furthermore, cycle consistency is introduced by cascading two backbone networks, and a cycle consistency constrained loss function is designed for image detail preservation. Experimental results illustrate that the newly proposed C2SU-DIP network is capable of obtaining competitive unmixing results compared with several representative spectral unmixing methods.

In this paper, a dual-path optimized fusion network based on spectral unmixing (DPOSU) is proposed for the fusion of hyperspectral image (HSI) and multispectral image (MSI). Based on the spectral mixing model of HSI, an endmember optimization model and an abundance optimization model are constructed respectively. Combining with the observation model, a fusion model for HSI and MSI is then derived. To address the unknown spectral and spatial degradation matrices in the optimization models, a dual-path optimization network is constructed to iteratively update endmember and abundance. Comprehensive experimental results illustrate that the proposed DPOSU network outperforms several typical traditional fusion methods as well as some representative deep learning based fusion methods both visually and quantitatively.

SAR-to-optical image translation (S2OIT) improves the interpretability of SAR images, providing a clearer visual insight that can significantly enhance remote sensing applications. Compared to supervised S2OIT methods that are limited by the paired dataset, unsupervised methods have shown more advantages in practical applications. However, the existing unsupervised S2OIT approaches, designed for unpaired datasets, often struggle to generalize well to scenes that are significantly different from the training data, potentially leading to mistranslations in diverse scenarios. To address the above issues, we propose a scene-embedded generative adversarial network for semi-supervised S2OIT called ScE-GAN, which utilizes the scene category labels in addition to unpaired image dataset, thus effectively improving the robustness of S2OIT under different scenes without increasing complex network structure and learning cost. In particular, a scene information fusion generator (SIFG) is proposed to learn the relationship between the image and the scene directly through scene category guidance and multihead attention, enhancing its ability to adapt to scene changes. Moreover, a scene-assisted discriminator (SAD) is presented cooperating with the generator to ensure both image authenticity and scene accuracy. Extensive experiments on two challenging datasets SEN1-2 and QXS-SAROPT demonstrate that our method outperforms the state-of-the-art methods in both objective and subjective evaluations. Our code and more details are available athttps://github.com/lr-dddd/ScE-GAN.

Image-to-image (I2I) translation methods based on Generative Adversarial Networks (GANs) have shown general solutions for aerial visible-to-infrared image translation (AVIT) task. Though existing approaches have made impressive results, they still struggle to produce diverse or high-realism translated aerial infrared images (AIIs). In this paper, a novel model is proposed to achieve both diverse and realistic AVIT, named DR-AVIT. Specifically, we introduce disentangled representation learning to disentangle the image representation of aerial visible images (AVIs) and AIIs into a domain-invariant semantic structure space and two domain-specific imaging style spaces. By leveraging this disentanglement, our model can perform the translation process conditioned on semantic structure information derived from the input AVI and randomly sampled imaging style features from the AII domain to obtain diverse outputs. Furthermore, a new constraint is present to encourage GANs to learn efficient mappings between AVI and AII domains by integrating geometry-consistency constraint and a dual learning framework, named dual geometry-consistency constraint. Coping with these two designs, our method exhibits superiority in both realism and diversity of the translation results over several state-of-the-art I2I translation methods on AVIID dataset and two new benchmark datasets for AVIT, which are obtained by extracting data from publicly available datasets. Code of DR-AVIT and proposed benchmark datasets are available at https://github.com/silver-hzh/DR-AVIT.

Multispectral (MS) pansharpening can improve the spatial resolution of MS images by fusing panchromatic (PAN) images, which have important applications in the fields of smart agriculture and environmental monitoring. However, existing supervised algorithms treat the original MS images as ground truth and generate training data under Wald’s protocol, resulting in a gap between the learned degradation process of the model and reality. This leads to the model having poor generalization and impractical. Unsupervised pansharpening methods often struggle to fully explore the rich information contained in images, leading to suboptimal pansharpening outcomes. In this work, we propose an unsupervised pansharpening algorithm based on double-cycle consistency that can learn directly from the original MS images without relying on artificially simulated degradation processes. Specifically, the network with cross-domain correlation information interaction is developed to achieve a deep fusion of spatial and spectral features. To address the inaccurate degradation mechanism representation of MS images, a spatial information extraction module based on scale invariance is developed to achieve an accurate representation. Meanwhile, double-cycle consistency loss is proposed to reduce the information loss caused by simulated degradation during the cycle process. Experimental results show that this method outperforms existing unsupervised pansharpening methods in both quantitative and qualitative evaluation of full-resolution images.

Graph convolutional networks (GCNs) have garnered extensive attention in the realm of hyperspectral image (HSI) classification. However, due to the problem of over-smoothing caused by deep GCN, most of the existing GCN-based methods are limited to constructing shallow networks, thus only able to extract superficial features. Moreover, when existing shallow GCNs extend to a more deeper structure, the number of learnable parameters increase linearly, thus leading to poor generalization performance under limited training samples. To address the aforementioned issues, a Separable Deep Graph Convolutional Network Integrated with CNN and Prototype Learning (SDGCP) is proposed for HSI classification, which can extract effective global structural information of HSI without increasing the number of trainable parameters. Specifically, the spectral and spatial features, adaptively selected by the attention module, are encoded into the structure of a graph by the graph encoder with the assistance of the pixel-to-region mapping obtained from the simple linear iterative clustering (SLIC). Then, a separable deep graph convolution module, composed of feature extraction and deep feature propagation, is adopted to capture the long-range contextual relationships from HSI encoded as graph data, which is combined with locally complementary information extracted by CNN after decoding. Finally, to further boost the performance of classification under limited labeled samples, prototype learning with regularization terms is utilized to enhance the intra-class compactness and inter-class separability of feature representations. Extensive experiments on three standard HSI data sets demonstrate the superiority of the proposed SDGCP over the state-of-the-art (SOTA) methods.

Transformers can model global context to enhance the performance of hyperspectral classification. However, the explored global information is generally confined to the spatial neighborhood of target pixels. In order to fully leverage global correlation across broader areas, a deformable graph transformer (DGT) is proposed for hyperspectral classification, in which the global information within an entire image is explored to improve the classification performance. Specifically, DGT layers are designed to adaptively sample virtual nodes at varying distances from an initial graph constructed from an image, by which the global spatial information can be explored using a deformable graph self-attention (DGSA) mechanism. Moreover, a learnable absolute position encoding (LAPE) module is constructed to enhance the spatial context awareness of DGT by integrating positional information into the graph nodes. In addition, graph structure encoding and graph topology encoding are further designed as inductive biases for the graph, by which both local structural information and global topological information of the HSI are captured to enhance the feature extraction capability of the DGT layer. Ultimately, through the stacking of multiple DGT layers, a composite feature fusion learning (CFFL) module is employed to fully utilize the simple low-level and complex abstract high-level features extracted from different layers. Extensive experiments on four datasets demonstrate the superiority and robustness of the proposed DGT over several state-of-the-art (SOTA) methods in terms of various evaluation criteria.

Band selection (BS), which effectively reduces spectral dimensionality, stands out as a leading focus within hyperspectral image (HSI) analysis. Self-representation (SR) has surfaced as a favored technique in this domain due to its applicability to BS and unsupervised nature. However, the existing SR-based BS approaches only leverage either spatial or spectral relationships, with few integrating both while concentrating on the representation level rather than the selection level. In addition, employing all spatial pixels for spatial relationship utilization leads to considerable computational complexity. Therefore, this article proposes joint spatial and spectral graph-based consistent SR (JSSGCSR) to more effectively exploit spatial and spectral relationships for BS, which separately conducts SR to handle each view of spatial and spectral graphs to better consider two different structure characteristics, and ultimately integrates two SR results to achieve a unified and robust representative band set by imposing consistent sparsity pattern on their joint representation coefficients. In addition, the spatial and spectral relationships are integrated into different data spaces, that is, spectral graph SR and spatial graph SR are, respectively, conducted in the original HSI and the segmented and pooled HSI, which not only reduces the influence of superpixel segmentation on spectral relationships, but also improves the efficiency of spatial relationship utilization. Experimental results on three benchmark datasets have demonstrated the effectiveness of the proposed JSSGCSR in HSI classification tasks.

Learning discriminative features is of crucial for hyperspectral image (HSI) classification. Though metric learning has been applied to learn effective features in HSI classification tasks, existing metric loss functions only consider distance among features of sample pairs but ignore the feature centers and boundaries in the embedding feature space, which limits the discrimination of learned features. In this paper, a novel metric loss function named center-boundary metric loss (CBML) is proposed to learn more discriminative features so as to improve HSI classification performance. Unlike the existing metric loss functions, CBML not only considers the distance between sample pairs to enhance intra-class similarity and inter-class separability but also pays more attention to the feature centers and boundaries in the embedding feature space that could greatly determine and affect the category of features. Specifically, CBML forces the distance of a sample to its corresponding feature center to be explicitly smaller than that to samples from other classes by a predefined threshold. As a result, the boundaries of different classes will separate an actual distance, which improves the discrimination of learned features. Moreover, in order to improve the training efficiency, a cross mini-batch sampling strategy is further proposed to break through the limitation within the mini-batch by using features between several contiguous mini-batches to sample pairs without increasing the size of the mini-batch. Accordingly, the sampling range of sample pairs is greatly expanded, and the training data is more fully exploited. Experimental results over four benchmark datasets with a typical network for HSI classification demonstrate our proposed method outperforms several state-of-the-arts.

Recent developments in adversarial attack have witnessed the success of background attack against object detectors. However, most existing methods attack detectors by luring targets into background. Therefore, an innovative Background Attack framework via Dual-adversarial-induced Error Identification (BADEI) is proposed to attack detectors by deceiving background as targets, as well as deceiving targets as background, where the attack performance can be greatly enhanced by these two kinds of induced error identification. Specifically, a mechanism that generates the adversarial background is proposed to result in dual error detection, where the background can conceal the specified targets and cause the misclassification of the adversarial pattern in the background as a specific category. Moreover, an unoccluded training strategy (UTS) that leverages the target mask of an image is introduced to strategically place adaptive adversarial background beneath the targets while optimizing and updating the pixel values of the background outside the target region, which can enhance attack effectiveness for adversarial background, significantly degrade the targets’ average accuracy, and enhance the robustness of background. Finally, a dual deceptive loss function (D2LF) is carefully formulated to generate false negatives (FNs) and false positives (FPs) to achieve untargeted attacks for hiding objects as well as targeted attacks for erroneously recognizing objects. Extensive experiments and comparative analysis of various victim network models on two datasets (including DOTA dataset and RSOD dataset) confirm that the proposed framework exhibits superior performance over the state-of-the-art methods in both digital and physical scenarios.

Metalearning has been widely applied to solve the few-shot object detection (FSOD) problem in natural scenes, which performs similarity measurement and information aggregation of the support set and the query set. However, regarding remote sensing images (RSIs), many difficulties caused by their disparities need to be further addressed, such as inconsistencies in imaging scale, direction, and background between support and query images. These result in feature misalignment and attention bias, interfering with model performance. In this article, a multilevel information interaction (MLII) strategy is proposed for FSOD to alleviate feature misalignment and attention bias. Information interactions are conducted within multiple scales of features and highlight similar regions of query and support features. A semantic enhancement module (SEM) is proposed to assist MLII in extracting key information and achieving more discriminative feature representation. Moreover, a feature cross-aggregation module (FCM) with separate classification losses is designed to train the detector to identify objects that coexist in query and support images. Extensive experiments demonstrate that the proposed method outperforms several state-of-the-art few-shot object detectors over commonly used benchmark datasets, i.e., DIOR and NWPU-10.

Dozens of spectral bands of hyperspectral images (HSIs) have been successfully reconstructed from only three color band images using deep neural networks according to their powerful nonlinear mapping capability. However, the existing deep-learning-based approaches tend to directly reconstruct HSIs from RGB inputs without emphasizing the discriminative intrinsic properties of different materials, resulting in certain distortion in reconstructed spectra. In this article, an intrinsic image decomposition (IID)-based spectral super-resolution (SSR) framework is proposed to reconstruct spectra of pixels from their reflectance feature and shading feature separately, by which the intrinsic properties can be emphasized during spectral reconstruction. Specifically, a dual hierarchical regression network (DHRNet) is designed for the proposed IID-based SSR task, in which a shading feature extraction module (SFEM) based on dense structure and a reflectance feature extraction module (RFEM) with attention mechanism are first, respectively, designed to reconstruct spectral information from reflectance feature and shading feature, and a feature enhancement module (FEM) is consequently devised to further improve the coarse combined estimation. Ultimately, a novel hybrid loss combining smooth$\boldsymbol {l}_{1}$loss, spectral angel mapper (SAM), and gradient prior is also presented to restrain the spectral distortion while enhancing the sharpness of the reconstructed HSI. Experimental results over three datasets demonstrate the superiority of our proposed framework.

Feature representation is crucial for hyperspectral image (HSI) classification. However, existing convolutional neural network (CNN)-based methods are limited by the convolution kernel and only focus on local features, which causes it to ignore the global properties of HSIs. Transformer-based networks can make up for the limitations of CNNs because they emphasize the global features of HSIs. How to combine the advantages of these two networks in feature extraction is of great importance in improving classification accuracy. Therefore, a cross-attention fusion network bridging CNN and Transformer (CAF-Former) is proposed, which can fully utilize the advantages of CNN in local features and Transformer’s long time-dependent feature learning for hyperspectral classification. In order to fully explore the local and global information within an HSI, a Dynamic-CNN branch is proposed to effectively encode local features of pixels, while a Gaussian Transformer branch is constructed to accurately model the global features and long-range dependencies. Moreover, in order to fully interact with local and global features, a cross-attention fusion (CAF) module is proposed as a bridge to fuse the features extracted by the two branches. Experiments over several benchmark datasets demonstrate that the proposed CAF-Former significantly outperforms both CNN-based and Transformer-based state-of-the-art networks for HSI classification.

Aerial person detection (APD) is vital for enhancing search and rescue (SaR) operations, particularly when locating victims in remote, poorly-lit areas. Despite advancements in detection technologies, achieving a balance between detection speed and accuracy on mobile devices in “edge AI” continues to pose challenges. In this article, a lightweight distillation network (APDNet) is proposed for edge deployment of APD, which enables real-time inference as well as minimizes accuracy loss during model transfer. The proposed APDNet employs a distillation network between varying-depth backbones and integrates an 8-bit quantized optimizer to reduce the floating-point operations of network parameters. Specifically, in the teach-assistant distillation (TAD) stage, small student models using random weight initialization are trained with pseudo-labels generated by deeper teacher models, facilitating consistent learning for a more accurate, lighter model. Moreover, a low-precision quantization (LPQ) stage incorporates an offline, quantization-aware training strategy that dynamically adjusts the ranges of weight and activation function float-point values, reducing computational complexity. In order to compensate for the potential accuracy decline, a pluggable tracker updates the position and feature information of persons frame-by-frame, with tracking results integrated with detection outputs to enhance accuracy. Extensive experiments on the Heridal, Manipal-UAV, and VTSaR datasets confirm the effectiveness of APDNet, demonstrating its superior performance in edge-based APD.

Oriented object detection has gained increasing attention due to its ability to detect objects with arbitrary orientations in the field of remote sensing (RS) images. However, the laborious task of annotating oriented bounding boxes (OBBs) presents significant challenges for training a fully supervised arbitrary-oriented object detector. Some existing approaches apply annotated horizontal bounding boxes (HBBs) as weakly supervised signals, in which even HBB annotations require significant human efforts. In this article, we propose a point-to-mask-to-HBB-to-OBB (PMHO) method that achieves weakly supervised oriented object detection by requiring only single-point object annotations. Specifically, we first take the input images and the given annotated points as prompts to obtain the initial segmentation masks, and then present a neighboring mask combination scheme to address the over-segmentation issue and a point-centric mask selection strategy to filtrate the related masks. Based on positive and negative proposal bags transferred from the mask regions, the pseudo-HBB generation network which consists of a classification branch for classification and an instance branch for the localization of individual proposals, aims to generate pseudo HBBs for each object. For further refinement of pseudo HBBs, we present a pseudo-HBB filtering strategy with K-means clustering on features extracted by a CNN model pretrained on a large-scale offline RS dataset. To train the oriented object detector, an inscribed ellipse constraint is proposed to measure the regression loss between predicted OBBs and the given pseudo HBBs. Extensive experiments including OBB detection, pseudo-HBB generation, and ablation studies are conducted on public DIOR and DIOR-R datasets, which demonstrates that our method achieves state-of-the-art performance.

Object detection in remote sensing images identifies and extracts the acquired Earth surface information, providing data support and research basis for multiple fields. Remote sensing image object detection based on knowledge distillation (KD) can transfer the knowledge of a large teacher model to a smaller student model, achieving the effect of low parameter volume and high accuracy. Mainstream methods directly imitate teacher features to improve student performance, ignoring the generation of high-ranking features through teacher features instructing student feature maps in this knowledge transfer process. In this article, an adaptive composite feature generation (ACFG) strategy is proposed to achieve end-to-end trainable KD for object detection in remote sensing images, in which the robustness of feature points under composite masks is improved through adaptive feature mapping. In particular, a composite mask generator (CMG) module is proposed to select student instance-related features and point background features. Furthermore, a global and local projection layer (GLPL) module is proposed to connect the local information and global information of the feature map under the mask generator to adaptively realize the global recovery mapping of the feature map with partial feature points. Finally, balanced decoupling loss (BDL) is improved to handle foreground and background loss separately, so that the two decoupled features can better enable the student model to learn instance-related information. Note that the proposed ACFG is capable of conducting KD for both single-stage and two-stage object detectors. Experimental results using both anchor-based and anchor-free detectors on the DIOR dataset and DOTA dataset demonstrate that the proposed ACFG clearly achieved better performance than several state-of-the-art (SOTA) algorithms for KD.

Infrared image acquisition is extremely challenging due to the huge spend of manpower and material resources. This paper presents a novel approach for generating paired RGB-infrared images by leveraging feature disentanglement and cross-modality reconstruction. The proposed model effectively extracts unique style features and shared content features from input RGB or infrared images and integrates these features to generate synthetic images corresponding to their respective modalities. This method aims to bridge the gap between different imaging modalities. Experimental evaluations were conducted on the ThermalWorld, LLVIP and SYSU-MM01 datasets, demonstrating the superiority of our method over recent image generation networks in terms of generated infrared image quality. The effectiveness of the model is further evidenced by its superior performance in a pedestrian re-identification task.

Deep Neural Networks (DNNs) have been extensively utilized in aerial detection. However, DNNs are susceptible and vulnerable to adversarial examples Recently, physical attacks have gradually garnered attention due to their effectiveness and practicality, which pose great threats to some security-critical applications. In this paper, we take the first attempt to perform physical attacks in contextual form against aerial detection in the physical world. We propose an innovative contextual attack method against aerial detection in real scenarios, which achieves powerful attack performance and transfers well between various aerial object detectors without smearing or blocking the interested objects. Based on the findings that the targets’ contextual information plays an important role in aerial detection by observing the detectors’ attention maps, we fully use the contextual feature of the interested targets to elaborate background perturbations for the uncovered attacks in physical scenarios. Experiments with proportional scaling are conducted to evaluate the effectiveness of the proposed method, demonstrating its superiority in terms of both attack efficacy and physical practicality.

Recently, Transformers have been widely used in many computer vision tasks and have shown promising results. However, like convolutional neural networks (CNNs), Transformers cannot handle rotational variations well, thus hindering its further application in the field of remote sensing. In this paper, we design a rotation-invariant Transformer (RIFormer) to alleviate the abovementioned problem. Moreover, we propose a novel rotation-invariant position embedding (RIPE) to encode positional information of features, and this position-dependent features learned by RIPE is robust to rotations. The experimental results show that proposed RIFormer with RIPE can effectively learn rotation-invariant features compared to the state-of-the-art methods with limited parameters. We provide an open-source implementation of our method. It is publicly available at https://github.com/psychAo/RIFormer.

The training of deep networks for hyperspectral target detection (HTD) is usually confronted with the problem of limited samples and in extreme cases, there might be only one target sample available. To address this challenge, we propose a novel approach with dual networks in this letter. First, a training set that is not fully accurate but representative enough regarding both targets and backgrounds is built through predetection and clustering. Then, two types of neural networks, that is, one generative adversarial network (GAN) and one convolutional neural network (CNN), which focus on spectral and spatial features of hyperspectral images (HSIs), are utilized for target detection. After that, the results of the two networks are fused, with the final detection result obtained. Experiments on real HSIs indicate that the proposed approach manages to perform HTD with only one target sample and is able to yield a more robust detection performance compared to other approaches.

Recently, the Transformer-based technique has emerged as a promising solution for modeling contextual information in Remote Sensing (RS) scenes and has found widespread applications in RS scene classification. However, how to make full use of intermediate features learned in Transformers is of crucial importance in the RS scene classification tasks. Therefore, this paper proposes a Hierarchical Feature Fusion of Transformer with Patch Dilating (HFFT-PD), which aims to capture rich contextual information from hierarchical features to enhance the performance of RS scene classification. Specifically, the HFFT-PD model consists of a Hierarchical Transformer Merging (HTM) block and a Lightweight Adaptive Channel Compression (LACC) module, in which the HTM is specially designed for the Transformer architecture to bridge the semantic gaps between features from different hierarchical blocks, and the LACC accounts for the significance of distinct channels in the ultimate classification features. In addition, a brand-new Patch Dilating strategy is uniquely designed for the Transformer paradigm, functioning as a reassembly operator predicated on patch features. Contrasting with conventional upsampling techniques, Patch Dilating facilitates upsampling without requiring supplementary information, while concurrently preserving the semantic content of local spatial structure. Extensive and rigorous experiments conducted on the UCM, AID, and NWPU-45 datasets, with training ratios of 80%, 50%, and 20% respectively, demonstrate that our proposed HFFT-PD outperforms the baseline at least by 0.59%, 0.44%, and 0.99% respectively, showcasing the significant superiority of our HFFT-PD over contemporary state-of-the-art methodologies.

With the wide application of remote sensing images (RSIs) in military and civil fields, remote sensing object detection (RSOD) has gradually become a hot research direction. However, we observe two main challenges for remote sensing object detection, namely the complicated background and the small objects issues. Given the different appearances of generic objects and remote sensing objects, the detection algorithms designed for the former usually cannot perform well for the latter. We propose a novel global to local scale-aware detection network (GLSANet) for remote sensing object detection, aiming to solve the above mentioned two challenges. Firstly, we design a global semantic information interaction module (GSIIM) to excavate and reinforce the high-level semantic information in the deep feature map, which alleviates the obstacles of complex background on foreground objects. Secondly, we optimize the feature pyramid network to improve the performance of multiscale object detection in RSIs. Finally, a local attention pyramid (LAP) is introduced to highlight the feature representation of small objects gradually while suppressing the background and noise in the shallower feature maps. Extensive experiments on three public datasets demonstrate that the proposed method achieves superior performance compared with the state-of-the-art detectors, especially on small object detection dataset. Specifically, our algorithm reaches 94.57% mAP on NWPU VHR-10 dataset, 95.93% mAP on RSOD dataset and 77.9% mAP on DIOR dataset, respectively.

Patch-based physical attacks have increasingly aroused concerns. However, most existing methods focus on obscuring targets captured on the ground, and some of these methods are simply extended to deceive aerial detectors. They smear the targeted objects in the physical world with the elaborated adversarial patches, which can only slightly sway the aerial detectors’ prediction and with weak attack transferability. To address the above issues, a novel Contextual Background Attack (CBA) framework is proposed to fool aerial detectors in the physical world, which can achieve strong attack efficacy and transferability in real-world scenarios even without smudging the interested objects at all. Specifically, the targets of interest, i.e. the aircraft in aerial images, are adopted to mask adversarial patches. The pixels outside the mask area are optimized to make the generated adversarial patches closely cover the critical contextual background area for detection, which contributes to gifting adversarial patches with more robust and transferable attack potency in the real world. To further strengthen the attack performance, the adversarial patches are forced to be outside targets during training, by which the detected objects of interest, both on and outside patches, benefit the accumulation of attack efficacy. Consequently, the sophisticatedly designed patches are gifted with solid fooling efficacy against objects both on and outside the adversarial patches simultaneously. Extensive proportionally scaled experiments are performed in physical scenarios, demonstrating the superiority and potential of the proposed framework for physical attacks. We expect that the proposed physical attack method will serve as a benchmark for assessing the adversarial robustness of diverse aerial detectors and defense methods. The code has been released at https://github.com/JiaweiLian/CBA.

Transformer has been widely applied in image processing tasks as a substitute for Convolutional Neural Networks (CNNs) for feature extraction due to its superiority in global context modeling and flexibility in model generalization. However, the existing transformer-based methods for semantic segmentation of Remote Sensing (RS) images are still with several limitations, which can be summarized into two main aspects: 1) the transformer encoder is generally combined with CNN-based decoder, leading to inconsistency in feature representations; 2) the strategies for global and local context information utilization are not sufficiently effective. Therefore, in this paper, a Global-Local Transformer Segmentor (GLOTS) framework is proposed for semantic segmentation of RS images to acquire consistent feature representations by adopting transformers for both encoding and decoding, in which a Masked Image Modeling (MIM) pretrained transformer encoder is adopted to learn semantic-rich representations of input images, and a multi-scale global-local transformer decoder is designed to fully exploit the global and local features. Specifically, the transformer decoder uses a feature separation-aggregation module (FSAM) to utilize the feature adequately at different scales and adopts a global-local attention module (GLAM) containing Global Attention Block (GAB) and Local Attention Block (LAB) to capture the global and local context information respectively. Furthermore, a Learnable Progressive Upsampling Strategy (LPUS) is proposed to restore the resolution progressively, which can flexibly recover the fine-grained details in the upsampling process. Experimental results on the three benchmark RS datasets demonstrate that the proposed GLOTS is capable of achieving better performance with some state-of-the-art methods, and the superiority of the proposed framework is also verified by ablation studies. The code will be available at https://github.com/lyhnsn/GLOTS.

Band selection which can reduce the spectral dimensionality effectively, has become one of the most popular topics in hyperspectral image (HSI) analysis. Recently, sparse representation based band selection (BS) has emerged as a popular tool. The existing sparse models mainly focus on minimizing reconstruction error and sparsity, while do not fully exploit the unique correlations among hundreds of continuous bands, which may cause representative bands missed and highly-correlated bands selected. Therefore, this paper proposes the spectral correlation based diverse band selection (SCDBS) for HSIs to improve representativeness and diversity of the selected bands. Specifically, a correlation derived weight is used to perform weighted sparse reconstruction to select the bands that are more correlated to the whole HSI, and a correlation minimization term is designed to remove the highly-correlated bands simultaneously. In addition, the proposed method imposes an adjustable sparse constraint by using an ℓ2,0

Convolutional neural networks (CNNs) have been demonstrated to be powerful tools to automatically learn effective features from large datasets. Though features learned in CNNs are approximately scale-, translation-, and position-invariant, and their capacity in dealing with image rotations remains limited. In this article, a novel cyclic polar coordinate convolutional layer (CPCCL) is proposed for CNNs to handle the problem of rotation invariance for feature learning. First, the proposed CPCCL converts rotation variation into translation variation using polar coordinates transformation, which can easily be handled by CNNs. Moreover, cyclic convolution is designed to completely handle the translation variation converted from rotation variation by conducting convolution in a cyclic shift mode. Note that the proposed CPCCL is capable of generalization and can be used as a preprocessing layer for classification CNNs to learn the rotation-invariant feature. Extensive experiments over three benchmark datasets demonstrate that the proposed CPCCL can clearly handle the rotation-sensitive problem in traditional CNNs and outperforms several state-of-the-art rotation-invariant feature learning algorithms.

Spectral super-resolution (SR), which reconstructs high spatial-resolution hyperspectral images (HSIs) from RGB inputs, has been demonstrated to be one of the effective computational imaging techniques to acquire HSIs. Though deep neural networks have shown their superiority in such a complex mapping problem, existing networks generally involve a very complex structure with huge amounts of parameters, resulting in giant memory occupation. In this article, a lightweight multiresolution feature fusion network (MRFN) is proposed, which adopts a multiresolution feature extraction and fusion framework to fully explore RGB inputs in different scales of resolution. Specifically, a lightweight feature extraction module (LFEM), which adopts cheap convolution and attention mechanisms, is constructed to explore different scales of features under a lightweight structure. Moreover, a hybrid loss function is proposed by encountering not only pixel-value level reconstruction error but also spectral continuity and fidelity. Experiments over three benchmark datasets, i.e., CAVE, Interdisciplinary Computational Vision Laboratory (ICVL), and NTIRE2022 datasets, have demonstrated that the proposed MRFN can reconstruct HSIs from RGB inputs in higher quality with fewer parameters and computational floating-point operations (FLOPs) compared with several state-of-the-art networks.

Despite deep neural networks (DNNs) have been widely applied in remote sensing (RS) scene classification and achieved satisfying performance, the vulnerability of DNNs towards adversarial examples significantly degrades their performance. Moreover, the relatively limited labeled samples of RS scene classification make DNNs more likely to overfit, leading to weak generalizability and noise sensitivity. This may result in DNNs being more vulnerable to adversarial examples. Consequently, the defense of adversarial examples is of crucial importance to improve both the generalizability and robustness of DNNs in the RS scene classification task. However, few studies have been conducted on defense for RS scene classification, especially ignoring the intrinsic characteristics of RS images. In this paper, an effective defense framework for RS scene classification, named reconstruction-assisted and distance-optimized adversarial training (RDAT), is proposed to defend adversarial examples. In order to solve the problems caused by high interclass similarity, a distance-optimized (DO) strategy is designed for adversarial training to strengthen the learning of underfitting content, increase the interclass distance, and improve the robustness of the networks. Furthermore, in order to generate high quality samples for adversarial training, a reconstruction-assisted (RA) block is proposed to eliminate adversarial perturbations in adversarial examples. Specifically, in this block, by swin transformer (SwinT) block and multi-scale convolution (MSC) block, SwinT-MSC-UNet (SMUNet) is constructed to fully extract global and multi-scale local features to adapt to the characteristics of RS images with large variance of ground object scales. Extensive experiments on the benchmark datasets, i.e., UC Merced (UCM) and Aerial Image Dataset (AID), have demonstrate that the proposed RDAT can effectively resist multiple adversarial attacks and yield superior results than other defense methods for RS scene classification.

Transformer-based networks, which can well model the global characteristics of inputted data using the attention mechanism, have been widely applied to hyperspectral image (HSI) classification and achieved promising results. However, the existing networks fail to explore complex local land cover structures in different scales of shapes in hyperspectral remote sensing images. Therefore, a novel network named multiscale and cross-level attention learning (MCAL) network is proposed to fully explore both the global and local multiscale features of pixels for classification. To encounter local spatial context of pixels in the transformer, a multiscale feature extraction (MSFE) module is constructed and implemented into the transformer-based networks. Moreover, a cross-level feature fusion (CLFF) module is proposed to adaptively fuse features from the hierarchical structure of MSFEs using the attention mechanism. Finally, the spectral attention module (SAM) is implemented prior to the hierarchical structure of MSFEs, by which both the spatial context and spectral information are jointly emphasized for hyperspectral classification. Experiments over several benchmark datasets demonstrate that the proposed MCAL obviously outperforms both the convolutional neural network (CNN)-based and transformer-based state-of-the-art networks for hyperspectral classification.

Remote sensing cross-modal text-image retrieval (RSCTIR) has recently received unprecedented attention due to its advantages of flexible input and efficient query on enormous remote sensing (RS) images. However, most RSCTIR methods focus obsessively on the cross-modal semantic alignment between text and image modalities and are easily stuck in the dilemma of information redundancy, leading to the degradation of retrieval accuracy. To address the issues, we construct a novel RSCTIR framework based on mask-guided relation modeling with entity loss (MGRM-EL), to fully explore uni-modal feature learning on entities and relations in the cross-modal model learning process. Specifically, we take advantage of the Transformer encoder architecture for its ability of capturing long-distance dependencies from the global view, and build two uni-modal (visual and textual) Transformer encoders combined with convolutional neural network (CNN) to extract the spatial inter-region relations of images as well as long-term inter-word relations of texts for prominent feature embedding of visual and semantic representations. A mask-guided attention strategy is further introduced to learn the salient regions and words, with the aim of enhancing the RSCTIR model’s uni-modal learning ability and eliminating unnecessary and redundant information of each modality. Unlike existing methods that simply compute the semantic similarity between images and texts on the loss functions, we present a novel uni-modal entity loss, which treats each image as an image entity and merges similar texts into a text entity, to learn the independent distribution of entities in each modality. We conduct extensive experiments on public RSCTIR benchmarks including RSICD and RSITMD datasets, which demonstrate the state-of-the-art performance of the proposed method on the RSCTIR task.

Freezing of gait (FoG) is one of the most common symptoms of Parkinson's disease, which is a neurodegenerative disorder of the central nervous system impacting millions of people around the world. To address the pressing need to improve the quality of treatment for FoG, devising a computer-aided detection and quantification tool for FoG has been increasingly important. As a non-invasive technique for collecting motion patterns, the footstep pressure sequences obtained from pressure sensitive gait mats provide a great opportunity for evaluating FoG in the clinic and potentially in the home environment. In this study, FoG detection is formulated as a sequential modelling task and a novel deep learning architecture, namely Adversarial Spatio-temporal Network (ASTN), is proposed to learn FoG patterns across multiple levels. ASTN introduces a novel adversarial training scheme with a multi-level subject discriminator to obtain subject-independent FoG representations, which helps to reduce the over-fitting risk due to the high inter-subject variance. As a result, robust FoG detection can be achieved for unseen subjects. The proposed scheme also sheds light on improving subject-level clinical studies from other scenarios as it can be integrated with many existing deep architectures. To the best of our knowledge, this is one of the first studies of footstep pressure-based FoG detection and the approach of utilizing ASTN is the first deep neural network architecture in pursuit of subject-independent representations. In our experiments on 393 trials collected from 21 subjects, the proposed ASTN achieved an AUC 0.85, clearly outperforming conventional learning methods.

Semantic segmentation has been widely researched for high level analysis of High Spatial Resolution (HSR) remote sensing images, where Convolutional Neural Network (CNN) is the mainstream method. However, the transformer with attention mechanism has its unique capacity of extracting global information which is generally ignored by CNN models. In this paper, a Swin Transformer with UPer head (STUP) is proposed to tackle with semantic segmentation problem on a challenging remote sensing land-cover dataset called LoveDA, which owns complex background samples and inconsistent classes distributions. The proposed STUP combines the Swin Transformer with Uper Head in the form of an encoder-decoder structure, to extract features of HSR images for segmentation. Furthermore, Focal Loss is adopted to handle the unbalanced distribution problem in the training step. Experimental results demonstrate that the proposed STUP clearly outperforms several state-of-the-art models.

In recent years, convolutional neural networks (CNNs) have been successfully applied in hyperspectral image (HSI) classification tasks. However, the spatial-spectral features within an HSI have not been well explored using convolutions in CNNs. In the paper, a novel end-to-end hierarchical spatial-spectral transformer (HSST) is proposed for HSI classification, in which effective spatial-spectral features are emphasized using multi-head self-attention mechanism (MHSA). MHSA module captures better internal correlation of HSI data than the traditional convolution operation and can compute weighting scores for spatial and spectral context of pixels. Furthermore, a hierarchical architecture is designed to reduce a large number of parameters in the original transformer-style networks while still achieving satisfying classification results. Experimental results over two benchmark HSI datasets demonstrated the proposed HSST obviously outperforms several state-of-the-art deep learning-based HSI classification algorithms.

Hyperspectral videos, which provide extra spectral characteristics besides spatial and temporal information, can improve the performance of object tracking using spectral signatures. However, there is a lack of labeled hyperspectral videos to support deep learning based model design. On the contrary, object tracking in the color space has been well developed in the past decade with many benchmark tracking models, e.g., SiamBAN. Therefore, how to transfer models designed in the color space to the hyperspectral space is of great importance. In this paper, hyperspectral videos are reduced into 3 bands using a band reduction algorithm, by which the existing well-trained trackers can be directly used. Specifically, Gaussian Information Entropy (GIE) is used to transform a hyperspectral video into a 3-band pseudo-color video, by which hyperspectral object tracking is conducted in an unsupervised mode. Experimental results demonstrate that object trackers designed in the color space can be transferred to hyperspectral videos using band reduction algorithms and the GIE based reduction is more effective than several well-known band reduction algorithms when using SiamBAN.

Spectral super-resolution (SR), which generally reconstructs hyperspectral images (HSIs) from RGB inputs, has attracted lots of attention recently. In this paper, a spectral SR algorithm based on intrinsic image decomposition (IID) is proposed, in which RGB images are decomposed into reflectance images and shading images to fully explore RGB features for HSI reconstruction. Considering that features of the reflectance image are only related to the material of objects, the sparsity of material reflectivity is used to reconstruct the reflectance image of HSI. Moreover, an convonlutional neural network (CNN) is constructed to reconstruct shading parts of HSI. Finally, these two reconstructed results are fused to generate the high spectral resolution HSI and an enhancement network is also designed to further improve the recontruction performance. Experimental results with two benchmark datasets, ICVL and CAVE, demonstrate that the performance of the proposed algorithm is superior to several state-of-the-art spectral SR algorithms.

Most object detection methods based on deep learning require large amounts of labeled data and can detect only the categories in the training set. Such issues significantly limit applications in remote sensing scenarios where it usually needs to recognize novel, unseen objects given very few training examples. To address these limitations, a novel meta-learning-based object detection method using Faster R-CNN framework is proposed for optical remote sensing image. Specifically, a diversity measurement module is proposed to measure diversity information between support images and query images on base classes so as to acquire more meta-knowledge. Experiments on DIOR dataset demonstrate our method has achieved superior performance than state-of-the-art meta-learning detection models in the field of remote sensing.

Collaborative representation (CR) has been demonstrated to be very effective for hyperspectral image classification. However, insufficient diversity of training samples often results in limited classification accuracy under small-training-sample conditions, especially when diverse spectral variation is presented in testing samples. In order to alleviate such a problem, a spectral variation augmented-based linear mixed model (SV-LMM) is proposed, in which the spectral variation is extracted by conducting singular value decomposition (SVD) over training samples. Such spectral variation is further utilized to extend the CR for hyperspectral classification. Experiments over two benchmark datasets, i.e., the Pavia Center dataset and the University of Houston dataset, demonstrate that the proposed extended CR-based classifier (ECRC) clearly improves the performance of conventional CRC for hyperspectral classification and outperforms several state-of-the-art algorithms.

Deep learning-based methods have drawn great attention in hyperspectral unmixing and obtained promising performance due to their powerful learning capability. However, few existing networks explicitly deal with the spectral variability inevitably present in hyperspectral images, limiting their fitting performance. In this letter, a spectral variability augmented two-stream network (SVATN) is designed to explicitly address the problem of spectral variability in a deep convolutional network for sparse unmixing. Specifically, the proposed SVATN maps a random input to coefficients of spectral variability in addition to abundances of endmembers, in which spectral variability is accommodated by the linear mixture model as an augmented item. Moreover, a spatial-spectral correlation-based variability extraction method (SSCVE) is proposed to construct a spectral variability library, which serves as priors in the loss function to optimize the proposed SVATN. Experiments over synthetic and real data sets demonstrate the superiority of the proposed SVATN over several state-of-the-art methods. The code of our proposed method is released at: https://github.com/MeiShaohui/SVATN.

Deep neural networks (DNNs) have become essential for aerial detection. However, DNNs are vulnerable to adversarial examples, which pose great security concerns for security-critical systems. Researchers recently devised adversarial patches to evaluate the vulnerability of DNNs-based aerial detection methods physically. Nonetheless, adversarial patches generated by existing algorithms are not strong enough and extremely time-consuming. Moreover, the complicated physical factors are not accommodated well during the optimization process. In this paper, a novel adaptive-patch-based physical attack (AP-PA) framework is proposed to alleviate the above problems, which achieves state-of-the-art performance in both accuracy and efficiency. Specifically, the AP-PA aims to generate adversarial patches that are adaptive in both physical dynamics and varying scales, and by which the particular targets can be hidden from being detected. Furthermore, the adversarial patch is also gifted with attack effectiveness against all targets of the same class with a patch outside the target (No need to smear targeted objects) and robust enough in the physical world. In addition, a new loss is devised to consider more available information of detected objects to optimize the adversarial patch, which can significantly improve the patch’s attack efficacy (Average precision drop up to 87.86% and 85.48% in white-box and black-box settings, respectively) and optimizing efficiency. We also establish one of the first comprehensive, coherent, and rigorous benchmarks to evaluate the attack efficacy of adversarial patches on aerial detection tasks. Finally, several proportionally scaled experiments are performed physically to demonstrate that the elaborated adversarial patches can successfully deceive aerial detection algorithms in dynamic physical circumstances. The code is available at https://github.com/JiaweiLian/AP-PA.

Convolutional neural networks (CNNs) have achieved excellent feature extraction capabilities in remotely sensed hyperspectral image (HSI) classification. This is due to their ability to learn representative spatial and spectral features. However, it is difficult for conventional computers to classify HSIs quickly enough for practical use in many applications, mainly because of the large number of calculations and parameters needed by deep learning-based methods. Although several weight quantization methods achieved remarkable results in network compression, the network acceleration effect is still not significant because a full exploration of the potential of network acceleration brought by network weight quantization is still absent from the literature. In this article, a new step activation quantization method is proposed to constrain the input of the network layer of the CNN so that the data can be represented by low-bit integers. As a result, floating-point operations can be replaced with integer operations to greatly accelerate the forward (inference) step of the network. Specifically, nonlinear uniform quantization is adopted in this work to restrain the input of the CNN in the forward inference of the step activation quantization layer, and two functions (constant and tanh-like) are used in the backpropagation step to avoid gradient vanishing and noise. Our newly proposed step activation quantization acceleration method is applied to a CNN for HSI with two well-known benchmark data sets and the experimental results demonstrate that the proposed method is very effective in terms of both memory savings and computation acceleration, with only a slight decrease in classification accuracy. Specifically, our method reduces memory requirements in$13.6\times $and obtains around$10\times {}$speedup with regard to the original real-valued network version.

Deep neural networks have been widely applied to hyperspectral image (HSI) classification areas, in which recurrent neural network (RNN) is one of the most typical networks. Most of the existing RNN-based classifiers treat the spectral signature of pixels as an ordered sequence, in which only unidirectional correlation along the wavelength direction of adjacent bands is considered. However, each band image is related to not only its preceding band images but also its successive band images. In order to fully explore such bidirectional spectral correlation within an HSI, in this article, a bidirectional long short-term memory (Bi-LSTM)-based network is designed for HSI classification. Moreover, a spatial–spectral attention mechanism is designed and implemented in the proposed Bi-LSTM network to emphasize the effective information and reduce the redundant information among spatial–spectral context of pixels, by which the performance of classification can be greatly improved. Experimental results over three benchmark HSIs, i.e., Salinas Valley, Pavia Centre, and Pavia University, demonstrate that our proposed Bi-LSTM obviously outperforms several state-of-the-art unidirectional RNN-based classification algorithms. Moreover, the proposed spatial–spectral attention mechanism can further improve the classification accuracy of our proposed Bi-LSTM algorithm by effectively weighting spatial and spectral context of pixels. The source code of the proposed Bi-LSTM algorithm is available athttps://github.com/MeiShaohui/Attention-based-Bidirectional-LSTM-Network.

Hyperspectral image (HSI) classification is a critical task with numerous applications in the field of remote sensing. Although convolutional neural networks have achieved remarkable success in computer vision, they are still limited in the ability to model long-term dependencies due to small receptive fields. Recently, vision transformers have been used in HSI classification, where multi-head self-attention (MHSA), as the key feature extractor of transformers, learns global dependencies in long-range positions and bands of HSI pixels. Existing vision transformers for classifying HSIs with a large number of bands, however, have some limitations in that features extracted by MHSA may exhibit over-dispersion. In this article, we propose a Group-Aware Hierarchical Transformer (GAHT) for HSI classification, which confines MHSA to the local spatial–spectral context by introducing a new grouped pixel embedding (GPE) module. The GPE emphasizes local relationships within HSI spectral channels, resulting in a global–local fashion from a spatial–spectral context for HSI classification. In addition, we construct our transformer in a hierarchical manner, which can significantly improve classification accuracy with only a few parameters. Extensive experiments on four benchmark HSI datasets demonstrate that the proposed method outperforms state-of-the-art HSI classification algorithms. The source code is available athttps://github.com/MeiShaohui/Group-Aware-Hierarchical-Transformer.

Due to variation of imaging conditions, spectra of the same type of ground objects usually exhibit certain discrepancy, leading to intra-class spectral distance increase and inter-class distance decrease. As a result, classification accuracy is greatly affected, especially in cases with few labeled samples. For representation based classifiers, the spectral variability within limited training samples is far from sufficient to represent diverse variations within testing ones. To handle this problem, a spectral variation augmented representation for hyperspectral imagery classification (SVARC) with few labeled samples is proposed in this article. Firstly, a novel class-independent and class-dependent components based linear representation model (CICD-LRM) is proposed to emphasize the representation of spectral variation. Secondly, depending on spatial and spectral correlation, the CICD-LRM guided global and local spectral variation extraction schemes are designed, and a fused spectral variation dictionary is constructed by concatenation. Finally, a classifier for hyperspectral images based on the CICD-LRM and spectral variation dictionary is proposed, and specifically three different spectral variation reconstruction strategies are designed. Similar to most of the representation based classifiers, residual-driven decision is also employed in the proposed classifier. Comparative experiments are conducted with eight classical and state-of-the-art methods using two benchmark datasets. The experimental results demonstrate that the proposed SVARC method significantly outperforms the compared ones in cases with few labeled samples.

Spectral unmixing expresses the mixed pixels existing in hyperspectral images as the product of endmembers and their corresponding fractional abundances, which has been widely used in hyperspectral imagery analysis. However, the endmember spectra even for pixels from the same material of an image may include variability due to the influence of lighting conditions and inherent properties of materials within different pixels. Though thein situspectral library has been used to accommodate such variability by using multiplein situspectra to represent each kind of material, the performance improvement may be restricted due to the limited number of endmembers for each material. Therefore, in this article, spectral variability is directly extracted from anin situendmember library and considered to be transferable among different endmembers for the first time. Furthermore, such a spectral variability is further used to augment sparse unmixing by synchronously performing endmember-based reconstruction and spectral variability-augmented reconstruction in the sparse unmixing model. By, respectively, imposing sparse and smoothness regularization over abundances and variability coefficients, a convex optimization-based spectral variability augmented sparse unmixing (SVASU) is finally proposed, and its convergence performance is also analyzed. Experiments conducted over synthetic and real-world datasets demonstrate that the proposed SVASU method not only significantly improves the unmixing performance of conventional spectral library-based unmixing but also outperforms several state-of-the-art sparse unmixing algorithms.

Video Summarization (VS) has become one of the most effective solutions for quickly understanding a large volume of video data. Dictionary selection with self representation and sparse regularization has demonstrated its promise for VS by formulating the VS problem as a sparse selection task on video frames. However, existing dictionary selection models are generally designed only for data reconstruction, which results in the neglect of the inherent structured information among video frames. In addition, the sparsity commonly constrained by$L_{2,1}$norm is not strong enough, which causes the redundancy of keyframes, i.e., similar keyframes are selected. Therefore, to address these two issues, in this paper we propose a general framework called graph convolutional dictionary selection with$L_{2,p}$($0< p\leq 1$) norm (GCDS$_{2,p}$) for both keyframe selection and skimming based summarization. Firstly, we incorporate graph embedding into dictionary selection to generate the graph embedding dictionary, which can take the structured information depicted in videos into account. Secondly, we propose to use$L_{2,p}$($0< p\leq 1$) norm constrained row sparsity, in which$p$can be flexibly set for two forms of video summarization. For keyframe selection,$0< p< 1$can be utilized to select diverse and representative keyframes; and for skimming,$p=1$can be utilized to select key shots. In addition, an efficient iterative algorithm is devised to optimize the proposed model, and the convergence is theoretically proved. Experimental results including both keyframe selection and skimming based summarization on four benchmark datasets demonstrate the effectiveness and superiority of the proposed method.

Hyperspectral remote sensing images are well-known for their abundant spectral characteristics to discriminate different object materials. However, due to the constraints of sensor limitations and exceedingly high acquisition costs, it is difficult to obtain high spatial resolution hyperspectral imagery. Though many methods have been focusing on the restoration of the spatial structure information, spectral information may be over-smoothed during such spatial super-resolution. In this paper, a novel self-calibrated attention residual network (SCARN) is proposed to increase spatial resolution of hyperspectral images while retain spectral consistency. In particular, a self-calibrated attention residual block (SCARB) is elaborately designed to fully exploit the spatial information and the correlation between the spectra of the hyperspectral data. Concretely, self-calibrated convolution, instead of standard convolution, is adopted to adaptively construct long-range spatial and spectral dependencies around each spatial location of hyperspectral imagery, and attention module is inserted to improve the representation ability of spectral information. Finally, global and local residual connections are designed to ease the network training difficulty and maintain a higher restoration accuracy. Experimental results over two benchmark hyperspectral datasets demonstrate the effectiveness and superiority of the proposed SCARN method against the state-of-the-art methods.

The deep learning based techniques have been widely applied to object tracking in color videos. When these techniques are applied to hyperspectral videos, how to fully explore unique spectral signatures of tracking objects is of crucial importance as well as simultaneously utilizing spatial and temporal information. Different with color videos, hyperspectral videos record continuous spectral reflectance of targets in light wavelength indexed band images and it is more difficult to explore unique spectral feature of tracking objects. Aiming to take advantage of existing object tracking techniques in color videos, a Siamese Multi-level Residual Attention Adaptive Network (SiamMRAAN) is designed to handle 3-band images by using the well-trained ResNet50 as backbone. By grouping hyperspectral videos into several 3-band-image subsets, the proposed SiamMRAAN can be used to explore high-dimensional spectral information. We design a loss function to fuse the tracking results over these subsets to improve the tracking performance. Finally, experiments over 75 hyperspectral videos confirmed that using spectral information is critical to improve the performance of object tracking in color videos, and also demonstrated that the proposed SiamMRAAN based strategy outperforms several compared networks for hyperspectral videos.

Animation production workflows centred around motion capture techniques often require animators to edit the motion for various artistic and technical reasons. This process generally uses a set of keyframes. Unsupervised keyframe selection methods for motion capture sequences are highly demanded to reduce the laborious annotations. However, most existing methods are optimization-based, which cause the issues of flexibility and efficiency and eventually constrains the interactions and controls with animators. To address these limitations, we propose a novel graph based deep reinforcement learning method for efficient unsupervised keyframe selection. First, a reward function is devised in terms of reconstruction difference by comparing the original sequence and the interpolated sequence produced by the keyframes. The reward complies with the requirements of the animation pipeline satisfying: 1) incremental reward to evaluate the interpolated keyframes immediately; 2) order insensitivity for consistent evaluation; and 3) non-diminishing return for comparable rewards between optimal and sub-optimal solutions. Then by representing each skeleton frame as a graph, a graph-based deep agent is guided to heuristically select keyframes to maximize the reward. During the inference it is no longer necessary to estimate the reconstruction difference, and the evaluation time can be reduced significantly. The experimental results on the CMU Mocap dataset demonstrate that our proposed method is able to select keyframes at a high efficiency without clearly compromising the quality in comparison with the state-of-the-art methods.

Video summarization (VS) is generally formulated as a subset selection problem where a set of representative keyframes or key segments is selected from an entire video frame set. Though many sparse subset selection based VS algorithms have been proposed in the past decade, most of them adopt linear sparse formulation in the explicit feature vector space of video frames, and don’t consider the local or global relationships among frames. In this paper, we first extend the conventional sparse subset selection for VS into kernel block sparse subset selection (KBS3) to utilize the advantage of kernel sparse coding and introduce a local inter-frame relationship through packing of frame blocks. Going a step further, we propose a similarity based block sparse subset selection (SB2S3) model by applying a specially designed transformation matrix on the KBS3 model in order to introduce a kind of global inter-frame relationship through the similarity. Finally, a greedy pursuit based algorithm is devised for the proposed NP-hard model optimization. The proposed SB2S3 has the following advantages: 1) through the similarity between each frame and any other frame, the global relationship among all frames can be considered; 2) through block sparse coding, the local relationship of adjacent frames is further considered; and 3) it has a wider application, since features can derive similarity, but not vice versa. It is believed that the effect of modeling such global and local relationships among frames in this paper, is similar to that of modeling the long-range and short-range dependencies among frames in deep learning based methods. Experimental results on three benchmark datasets have demonstrated that the proposed approach is superior to not only other sparse subset selection based VS methods but also most unsupervised deep-learning based VS methods.

Rotation-invariant features are of great importance for object detection and image classification in very-high-resolution (VHR) optical remote sensing images. Though multibranch convolutional neural network (mCNN) has been demonstrated to be very effective for rotation-invariant feature learning, how to effectively train such a network is still an open problem. In this article, a nested Siamese structure (NSS) is proposed for training the mCNN to learn effective rotation-invariant features, which consists of an inner Siamese structure to enhance intraclass cohesion and an outer Siamese structure to enlarge interclass margin. Moreover, a double center loss (DCL) function, in which training samples from the same class are mapped closer to each other while those from different classes are mapped far away to each other, is proposed to train the proposed NSS even with a small amount of training samples. Experimental results over three benchmark data sets demonstrate that the proposed NSS trained by DCL is very effective to encounter rotation varieties when learning features for image classification and outperforms several state-of-the-art rotation-invariant feature learning algorithms even when a small amount of training samples are available.

Laparoscopic videos have been increasingly acquired for various purposes including surgical training and quality assurance, due to the wide adoption of laparoscopy in minimally invasive surgeries. However, it is very time consuming to view a large amount of laparoscopic videos, which prevents the values of laparoscopic video archives from being well exploited. In this paper, a dictionary selection based video summarization method is proposed to effectively extract keyframes for fast access of laparoscopic videos. Firstly, unlike the low-level feature used in most existing summarization methods, deep features are extracted from a convolutional neural network to effectively represent video frames. Secondly, based on such a deep representation, laparoscopic video summarization is formulated as a diverse and weighted dictionary selection model, in which image quality is taken into account to select high quality keyframes, and a diversity regularization term is added to reduce redundancy among the selected keyframes. Finally, an iterative algorithm with a rapid convergence rate is designed for model optimization, and the convergence of the proposed method is also analyzed. Experimental results on a recently released laparoscopic dataset demonstrate the clear superiority of the proposed methods. The proposed method can facilitate the access of key information in surgeries, training of junior clinicians, explanations to patients, and archive of case files.

In recent years, sparse representation has been successfully utilized for video summarization (VS). However, most of the sparse representation based VS methods characterize each video frame with global features. As a result, some important local details could be neglected by global features, which may compromise the performance of summarization. In this paper, we propose to partition each video frame into a number of patches and characterize each patch with global features. Instead of concatenating the features of each patch and utilizing conventional sparse representation, we formulate the VS problem with such video frame representation as block sparse representation by considering each video frame as a block containing a number of patches. By taking the reconstruction constraint into account, we devise a simultaneous version of block-based OMP (Orthogonal Matching Pursuit) algorithm, namely SBOMP, to solve the proposed model. The proposed model is further extended to a neighborhood based model which considers temporally adjacent frames as a super block. This is one of the first sparse representation based VS methods taking both spatial and temporal contexts into account with blocks. Experimental results on two widely used VS datasets have demonstrated that our proposed methods present clear superiority over existing sparse representation based VS methods and are highly comparable to some deep learning ones requiring supervision information for extra model training.

In this letter, we propose a new sparse representation (SR)-based method for hyperspectral image (HSI) classification, namely SR with incremental dictionaries (SRID). Our SRID boosts existing SR-based HSI classification methods significantly, especially when used for the task with extremely limited training samples. Specifically, by exploiting unlabeled pixels with spatial information and multiple-feature-based SR classifiers, we select and add some of them to dictionaries in an iterative manner, such that the representation abilities of the dictionaries are progressively augmented, and likewise more discriminative representations. In addition, to deal with large-scale data sets, we use a certainty sampling strategy to control the sizes of the dictionaries, such that the computational complexity is well balanced. Experiments over two benchmark data sets show that our proposed method achieves higher classification accuracy than the state-of-the-art methods, i.e., the overall classification accuracy can improve more than 4%.

Many applications have benefited from the images with both high spatial and spectral resolution, such as mineralogy and surveillance. However, it is difficult to acquire such images due to the limitation of sensor technologies. Recently, super-resolution (SR) techniques have been proposed to improve the spatial or spectral resolution of images, e.g., improving the spatial resolution of hyperspectral images (HSIs) or improving spectral resolution of color images (reconstructing HSIs from RGB inputs). However, none of the researches attempted to improve both spatial and spectral resolution together. In this article, these two types of resolution are jointly improved using convolutional neural network (CNN). Specifically, two kinds of CNN-based SR are conducted, including a simultaneous spatial-spectral joint SR (SimSSJSR) that conducts SR in spectral and spatial domain simultaneously and a separated spatial-spectral joint SR (SepSSJSR) that considers spectral and spatial SR sequentially. In the proposed SimSSJSR, a full 3-D CNN is constructed to learn an end-to-end mapping between a low spatial-resolution mulitspectral image (LR-MSI) and the corresponding high spatial-resolution HSI (HR-HSI). In the proposed SepSSJSR, a spatial SR network and a spectral SR network are designed separately, and thus two different frameworks are proposed for SepSSJSR, namely SepSSJSR1 and SepSSJSR2, according to the order that spatial SR and spectral SR are applied. Furthermore, the least absolute deviation, instead of mean square error (MSE) in traditional SR networks, is chosen as the loss function for the proposed networks. Experimental results over simulated images from different sensors demonstrated that the proposed SepSSJSR1 is most effective to improve spatial and spectral resolution of MSIs sequentially by conducting spatial SR prior to spectral SR. In addition, validation on real Landsat images also indicates that the proposed SSJSR techniques can make full use of available MSIs for high-resolution-based analysis or applications.

Parkinson's disease significantly impacts the life quality of millions of people around the world. While freezing of gait (FoG) is one of the most common symptoms of the disease, it is time consuming and subjective to assess FoG for well-trained experts. Therefore, it is highly desirable to devise computer-aided FoG detection methods for the purpose of objective and time-efficient assessment. In this paper, in line with the gold standard of FoG clinical assessment, which requires video or direct observation, we propose one of the first vision-based methods for automatic FoG detection. To better characterize FoG patterns, instead of learning an overall representation of a video, we propose a novel architecture of graph convolution neural network and represent each video as a directed graph where FoG related candidate regions are the vertices. A weakly-supervised learning strategy and a weighted adjacency matrix estimation layer are proposed to eliminate the resource expensive data annotation required for fully supervised learning. As a result, the interference of visual information irrelevant to FoG, such as gait motion of supporting staff involved in clinical assessments, has been reduced to improve FoG detection performance by identifying the vertices contributing to FoG events. To further improve the performance, the global context of a clinical video is also considered and several fusion strategies with graph predictions are investigated. Experimental results on more than 100 videos collected from 45 patients during a clinical assessment demonstrated promising performance of our proposed method with an AUC of 0.887.

Three-dimensional (3D) convolutional networks have been proven to be able to explore spatial context and spectral information simultaneously for super-resolution (SR). However, such kind of network can't be practically designed very `deep' due to the long training time and GPU memory limitations involved in 3D convolution. Instead, in this paper, spatial context and spectral information in hyperspectral images (HSIs) are explored using Two-dimensional (2D) and One-dimenional (1D) convolution, separately. Therefore, a novel 2D-1D generative adversarial network architecture (2D-1D-HSRGAN) is proposed for SR of HSIs. Specifically, the generator network consists of a spatial network and a spectral network, in which spatial network is trained with the least absolute deviations loss function to explore spatial context by 2D convolution and spectral network is trained with the spectral angle mapper (SAM) loss function to extract spectral information by 1D convolution. Experimental results over two real HSIs demonstrate that the proposed 2D-1D-HSRGAN clearly outperforms several state-of-the-art algorithms.

Convolutional neural network (CNN) has been famous for its translation-invariant ability in feature learning. In order to further encounter rotation-invariant, data augmentation by rotation of training samples should be considered for multiple-branch based structure using maximum operator or average operator. In this paper, a novel Polar Coordinate CNN (PC-CNN) is proposed for rotation-invariant feature learning. Specifically, training samples are first input to a polar coordinate transform layer by which rotation-invariance is converted into translation-invariance. Consequently, rotation-invariance problem in feature learning can be easily encountered by traditional CNNs without the multiple-branch structure. Experimental results over two benchmark data sets demonstrate that the proposed polar transformation is very effective to encounter rotation-invariant into traditional CNNs and outperforms several state-of-the-art rotation-invariant CNNs.

Classification has been one of the most important applications of hyperspectral images (HSIs) in the past decade, because of the outstanding discrimination among different classes ensured by abundant and detailed spectral information enclosed in HSIs. While the classification accuracy must be guaranteed by plenty of training samples, which is difficult to be satisfied in many practical cases. Meanwhile, because of its comparatively low spatial resolution, mixed pixels are widely existed in HSIs which makes subpixel level classification techniques more preferable rather than traditional pixel-level ones. A novel super-resolution classification method is proposed in this paper to deal with the two above mentioned problems in HSI classification, that is, limited number of training samples and widely existed mixed pixels. Specifically, spectral variation is considered to construct spectral variation extended endmember library, with which the abundance fractions for each class within a mixed pixel are estimated using collaborative representation. And finally, the classification result with higher spatial resolution is obtained with subpixel spatial attraction model based subpixel mapping. Simulative experiments are employed for validation and comparison. Experimental results illustrate that the newly proposed method is capable of producing super-resolution classification map of low resolution HSI with less misclassification.

Reconstructing hyperspectral images from RGB inputs has gained great attention recently. In dictionary representation-based hyperspectral image reconstruction, dictionary representation is first carried out in RGB space and then dictionary reconstruction is conducted in hyperspectral space for per-pixel reconstruction. However, such work mainly focuses on spectral mapping from RGB space to hyperspectral space, ignoring physical distribution of objects in the image. In this paper, spatial context of pixels is used to improve the reconstruction performance. Specially, neighboring pixels are used to constrain the dictionary representation problem in RGB space, and the Simultaneous Orthogonal Matching Pursuit (SOMP) is used to improve the performance of hyperspectral reconstruction. Experimental results on two benchmark data sets demonstrate the superiority of the proposed technique.

Hyperspectral image (HSI) spatial super-resolution(SR) is a challenging task. Compared with a RGB images, the mapping between the low-high HSI pairs is more difficult since much more spectral bands are involved. In this paper, a novel dual 1D-2D spatial-spectral convolutional neural network (CNN) architecture is proposed for spatial SR of HSIs. Specifically, by differential treatment over redundancy in spectral and spatial domains of an HSI, the spectral and spatial context are first separately explored by 1D and 2D convolution. These two kinds of feature information are then fused using a novel hierarchical side connection, which impose the spectral information to the spatial path gradually. Experimental results over benchmark Pavia data set demonstrate that the proposed architecture clearly outperform state-of-the-art 3D CNN based works in terms of both visual quality and quantitative assessment.

Noises in hyperspectral image (HSI) degrades both spatial and spectral features of ground objects, and greately defects the following processing, such as classification, target detection and recognition. In this paper, a convolutional self-encoding network (DeCS-Net) is designed for HSI denoising, which integrates the superiority of convolutional neural network (CNN) and auto-encoder (AE) to learn multi-scale features. The noise in the observed HSI is estimated by residual learning strategy, and is removed from the observed HSI to obtain an estimation of the ideal HSI without noise. Experimental results on benchmark HSI data set illustrate that the proposed DeCS-Net is effective for HSI denoising and outperforms the state-of-the-art CNN based HSI denoising methods.

Convolutional neural networks (CNNs) have been demonstrated to be capable of learning effective spatial-spectral features for hyperspectral classification. However, traditional CNNs are mainly trained using classification errors in decision domain. In this paper, a metric learning based training strategy is proposed to further enhance feature separability by training CNNs in feature domain as well as decision domain. Specifically, a metric learning loss function is designed to train CNNs in the second last fully connected feature layer, instead of the last fully connected decision layer. As a result, both within-class feature similarity and between-class feature separability can be enhanced even with a small amount of training samples. Experimental results over two benchmark hyperspectral data sets demonstrate that the proposed metric learning strategy is very effective to explore more discriminative features and its performance obviously outperforms several state-of-art CNNs for classification of hyperspectral images.

Collaborative representation plays an increasingly important role in the field of hyperspectral imagery target detection, resulting in improving detection performance. It is known that, in hyperspectral imagery, both the sensor and external factors (such as weather, illumination and other environmental changes) will lead to the spectral variations within the same type of material, which may greatly affect the detection accuracy. To deal with this issue, a new target detection method using collaborative representation with spectral variation extended dictionary is proposed for hyperspectral imagery in this paper. In the proposed method, an extended dictionary is constructed by enclosing the spectral variation library into the original dictionary, and the following collaborative representation makes the atoms in both original dictionary and spectral variation library contribute to the residual estimation. Compared to the traditional collaborative representation based target detection method, the newly proposed one exhibits better detection performance.

High-resolution remote sensing image scene classification problem has attracted lots of attentions due to its crucial role in a wide range of applications. Recently, many convolutional neural network (CNN) based methods have significantly boosted the performance of image scene classification. However, most of these algorithms only make use of global features learned in the fully connected layers of a CNN, neglecting local features learned in the convolutional layers that is of crucial importance for some remote sensing scenes. Therefore, a sparse representation framework is proposed to make use of both local and global features learned in a CNN for classification of remote sensing scenes by balancing sparse representation based classifiers using these two kinds of features. Specially, in order to reduce redundancy of local features learned in a convolutional layer, the most effective local feature is generated from each convolutional layer using global average pooling and these selected local features of different convolutional layers are cascaded to form local feature representation of the scene. Finally, experimental results on UC-Merced and WHU-RS19 datasets demonstrate that fusing global and local features in a CNN using the proposed sparse representation framework can certainly improve the performance of classification using single kind of features. Moreover, the proposed global average pooling strategy is very effective to fuse local features select most representative features from convolutional layers of a CNN.

Hyperspectral unmixing has been widely used to decompose a mixed pixel into a collection of endmembers weighted by their corresponding fractional abundances, in which endmember extraction step is of crucial importance. Many classical endmember extraction algorithms mainly identify spectrally pure endmembers according to spectra of pixels, e.g., NFINDR and vertex component analysis (VCA), ignoring spatial distribution or structure information that has been demonstrated to be complemental for spectral information in hyperspectral image processing. In order to improve the performance of these classical endmember extraction algorithms, a novel spatial preprocessing method is proposed to explore spatial information prior to endmember extraction step. Specifically, pixels in hyperspectral images are modified using their sparse linear approximation by neighboring pixels, such that spectral variation within a local spatial neighbor-hood can be alleviated. Experimental results on both simulated and real data sets demonstrate that the proposed local sparse representation based spatial preprocessing algorithm is capable of producing better unmixing result compared to several state-of-the-art spatial preprocessing methods.

Spatial context has been demonstrated to be effective to constrain sparse unmixing (SU) of hyperspectral images. However, the existing algorithms employed simple spatial information without keeping spectral fidelity. By considering the fact that adjacent pixels own not only the endmembers with same variations but also approximated fractional abundances, in this paper, local spectral similarity preserving (LSSP) constraint is proposed to preserve spectral similarity in a local area during robust sparse unmixing (RSU). Specially, four LSSP constraints are constructed using different-norm-constrained pixel-level difference over abundance-level difference in a local area. Moreover, a convex optimization algorithm is proposed to solve the proposed LSSP-constrained RSU (LSSP-RSU). Experimental results on both synthetic and real hyperspectral data demonstrate that the developed algorithms yield better values of the signal-toreconstruction error (SRE). Especially, when using l2norm of pixel-level difference to weight the l1norm of abundance-level difference, the proposed LSSP-RSU algorithm can achieve superior unmixing performance.

Feature learning technologies using convolutional neural networks (CNNs) have shown superior performance over traditional hand-crafted feature extraction algorithms. However, a large number of labeled samples are generally required for CNN to learn effective features under classification task, which are hard to be obtained for hyperspectral remote sensing images. Therefore, in this paper, an unsupervised spatial-spectral feature learning strategy is proposed for hyperspectral images using 3-Dimensional (3D) convolutional autoencoder (3D-CAE). The proposed 3D-CAE consists of 3D or elementwise operations only, such as 3D convolution, 3D pooling, and 3D batch normalization, to maximally explore spatial-spectral structure information for feature extraction. A companion 3D convolutional decoder network is also designed to reconstruct the input patterns to the proposed 3D-CAE, by which all the parameters involved in the network can be trained without labeled training samples. As a result, effective features are learned in an unsupervised mode that label information of pixels is not required. Experimental results on several benchmark hyperspectral data sets have demonstrated that our proposed 3D-CAE is very effective in extracting spatial-spectral features and outperforms not only traditional unsupervised feature extraction algorithms but also many supervised feature extraction algorithms in classification application.

The recent explosive growth of multimedia data has posed a new set of challenges in computer vision, and video summarization (VS) techniques are increasingly important to automatically summarize a large amount of multimedia data in an effective and efficient manner. Recent years have witnessed the rise and developments of sparse representation based approaches for VS. While the existing methods select keyframes according to the information contained in the single frame, and such a selection based solely on single-frame information may not be robust. Therefore, in this paper, the information of the single frame's neighborhood is taken into consideration, and different weights are assigned to these neighbouring frames. We formulate the VS problem as a weighted neighborhood based representation model, and design a greedy pursuit algorithm to extract keyframes. Experimental results on a benchmark dataset demonstrate that the proposed method can outperform the state of the arts.

convolutional neural network (CNN) has been widely used for hyperspectral classification. Current researches of CN-N based hyperspectral image classification is mainly implemented on graphics processing unit (GPU) platform. However, GPU is not suitable for onboard processing due to the problem of space radiation and power supply on image acquiring platform. Therefore, in this paper, FPGA is selected to implement CNN based hyperspectral classification for further onboard processing. Specially, a hardware model is designed for the forward classification step of CNN using hardware description language, including computation structure for CNN, implementation of different layers, weight loading scheme, and data interfere. Simulation results over Pavia data set validate the proposed FPGA based implementation is coincide with that on GPU platform.

Pansharpening has been an important technique to increase the spatial resolution of the multispectral (MS) images provided by many earth observation satellites. Since the different spatial resolutions exist between the multispectral and panchromatic (PAN) images, pansharpening usually upsamples the MS images to the same size as the PAN image and then injects the spatial details into the upscaled MS ones. In this paper, we propose a novel pansharpening method focusing on the structure injection into the MS images through a joint Gaussian guided upsampling. The original spectral information is transferred to the joint upsampling outputs by using the hyperspherical color transformation (HCT). The experimental results show that our proposed method can obtain high-quality pansharpened results and outperforms some existing methods.

Convolutional neural network (CNN) has been widely used in many research areas due to its powerful ability of feature learning. In this paper, the powerful ability of feature learning in CNN is explored by constructing a novel convolutional network (ConvNet) for SAR image processing. The proposed ConvNet is firstly trained under classification task, in which effective features can be learned automatically from the training data. Specifically, data argument is adopted to overcome the small-sample-problem in SAR images. When well-trained, the proposed ConvNet can be directly used for feature extraction of other images, even though their classes maybe not used in the training. Experimental results on benchmark MSTAR data set demonstrate that the proposed ConvNet is effective for classification of SAR images, and the features learned from it are more effective than traditional hand-crafted features in SAR image processing.

Hyperspectral image compression by PCA and JPEG2000 can provide excellent rate distortion performance while preserving essential information for a successive application, e.g., classification tasks. However, for onboard applications, PCA suffers from high computational complexity and large memory requirements due to the eigen-analysis of high-dimensional covariance matrix. Therefore, a computationally more efficient analysis, namely Folded Principal Component Analysis (FPCA) is adopted to perform dimension reduction and combined with JPEG2000 for compression. In FPCA, the spectral vector of hyperspectral pixels is folded into a matrix to compute covariance matrix, by which the dimension of covariance matrix is highly reduced. As a result, both computational complexity and memory requirement in subsequent eigen-analysis is reduced. Experimental results demonstrate that the proposed FPCA+JPEG2000 based compression scheme outperforms existing PCA+JPEG2000 in terms of rate distortion and classification after de-compression.

The remote sensing images provided by satellites usually contain complex earth objects with different scales. In the fusion of such images, most of the existing filtering-based pansharpening methods often suffer from spectral and/or spatial information distortions due to the inaccuracy of the detail extraction. Motivated by this, we propose an effective and straightforward multi-scale structure perception pansharpening method, which uses the structure-preserving filter with great structure-aware ability to progressively perceive the structures and accurately extract the details. The experiment is carried out on GeoEye-1 satellite images. Visual and objective analysis show that our method can produce high-quality pansharpened results and outperform some existing methods.

Spatial context has shown to be very useful in hyperspectral image processing. Existing convolutional neural network (CNN)-based methods for hyperspectral classification explore spatial context by single-scale convolution kernels in 2D or 3D shapes. However, such single-scale convolution may not be capable to explore the complex spatial context in a hyperspectral image. In this paper, we propose a multi-scale CNN, MS-CNN to explore the spatial context in different extents, in which adaptive spatial neighborhood convolution kernels are used to simultaneously extract multiple spectral-spatial features from spatial context of pixels. These features obtained by different spatial kernels are then concatenated and fused for further feature extraction and classification. Experimental results show that the proposed adaptive spatial neighborhood convolution are more effective to explore spatial context than traditional single-scale spatial convolution and the performance of the proposed MS-CNN outperforms several state-of-art CNNs for classification of hyperspectral images.

This paper presents a new dictionary-based method to fuse hyperspectral image (HSI) and multispectral image (MSI) of the same observed scene. To incorporate spatial as well as spectral information and features from both source HSI and MSI, dual-source spatial-spectral dictionary pair is constructed. Furthermore, collaborative representation based image representation and reconstruction is employed for its outstanding representation performance and efficiency. Simulative experiments illustrate that the newly proposed fusion method is capable of producing better or comparable fusion result when compared to some state-of-the-art dictionary-based HSI and MSI fusion methods, with much less computational cost.

Arising from various environmental and atmos- pheric conditions and sensor interference, spectral variations are inevitable during hyperspectral remote sensing, which degrade the subsequent hyperspectral image analysis significantly. In this paper, we propose simultaneous spatial and spectral low-rank representation (S3LRR) that can effectively suppress the within-class spectral variations for classification purposes. The S3LRR recovers an intrinsic component with the same dimension as the original image, in which both spatial and spectral low-rank priors are adopted to regularize the intrinsic component simultaneously and compensate to each other, together with robust modeling of spectral variations. Compared with existing methods that explore only the spectral low-rank prior, the novel spatial low-rank prior (i.e., low-rank prior in band-wise) can take the spatial structure information of hyperspectral images into account, which has demonstrated to be very useful. Technically, we formulate S3LRR as a constrained convex optimization problem, and solve it using the efficient inexact augmented Lagrangian multiplier method. The resulting intrinsic component is less interfered by within-class spectral variations, and more discriminatory to offer higher classification accuracy. Comprehensive experiments on benchmark data sets demonstrate that the proposed S3LRR improves classification accuracy significantly, which outperforms state-of-the-art methods.

Dictionary selection based video summarization (VS) algorithms, in which keyframes are considered as a dictionary to reconstruct all the video frames, have been demonstrated to be effective and efficient for video summarization. It has been noticed that the feature representation of video plays a great impact of the performance of VS. In this paper, the influence of feature representation of video frames on the performance of dictionary selection-based VS is for the first time investigated. In addition to the traditional hand-crafted features used in VS, such as color histogram, the deep features learned through deep neural networks are firstly used to represent video frames for dictionary selection-based VS. The impact of dimensionality reduction to the high-dimensional deep learning features on VS is further discussed. Experimental results on a benchmark video dataset demonstrate that deep learning features are able to achieve better performance than traditional hand-crafted features for dictionary selection-based VS. Moreover, the dimensionality of deep learning features can be reduced to decrease the computational cost without the degradation of VS performance.

Due to the tradeoff between spatial and spectral resolution in remote sensing imaging, hyperspectral images are often acquired with a relative low spatial resolution, which limits their applications in many areas. Inspired by recent achievements in convolutional neural network (CNN) based super resolution (SR), a novel CNN based framework is constructed for SR of hyperspectral images by considering both spatial context and spectral correlation. As a result, the spectral distortion incurred by directly applying traditional SR algorithms to hyperspectral images is alleviated. Experimental results on several benchmark hyperspectral datasets have demonstrated that higher quality of reconstruction and spectral fidelity can be achieved, compared to band-wise manner based algorithms.

Sparse dictionary selection (SDS) has demonstrated to be an effective solution for keyframe based video summarization (VS), which generally assumes a linear relation among similar video frames. However, such a linear assumption is not always true for videos. In this paper, the nonlinearity among frames is taken into consideration and a nonlinear SDS model is formulated for VS, in which the nonlinearity is transformed to linearity by projecting a video to a high dimensional feature space induced by a kernel function. Moreover, a kernel simultaneous orthogonal matching pursuit (KSOMP) is proposed to solve the problem. In order to achieve an intuitive and flexible configuration of the VS process, an adaptive criterion is devised to produce video summaries with different lengths for different video content. Experimental results on benchmark video datasets demonstrate that the proposed algorithm outperforms several state-of-the-art VS algorithms.

Deep learning techniques have brought in revolutionary achievements for feature learning of images. In this paper, a novel structure of 3-Dimensional Convolutional AutoEncoder (3D-CAE) is proposed for hyperspectral spatial-spectral feature learning, in which the spatial context is considered by constructing a 3-Dimensional input using pixels in a spatial neighborhood. All the parameters involved in the 3D-CAE are trained without the need of labeled training samples such that feature learning is conducted in an unsupervised fashion. Such unsupervised spatial-spectral feature extraction is also extended to different images from the same sensor to learn sensor-specific features. As a result, spatial-spectral features of hyperspectral images are extracted for a specific sensor under an unsupervised manner. Experimental results on several benchmark hyperspectral datasets have demonstrated that our proposed 3D-CAE are very effective in extracting sensor-specific spatial-spectral features and outperform several state-of-the-art deep learning neural networks in classification application.

Deep learning techniques have been demonstrated to be a powerful tool to learn features of images automatically. In this paper, a novel deep learning structure, i.e., deep stacked neural network (DSNN), is constructed to extract different levels of deep features of hyperspectral images. Specifically, convolutional neural network (CNN) is used as basic units in the proposed DSNN for feature extraction of hyperspectral images. Then, different levels of deep features are concatenated to form a novel fused feature for classification with a typical classifier, e.g., SVM. Experimental results on two benchmark hyperspectral datasets show that the fusion of features extracted in DSNN can produce higher classification accuracy than state-of-the-art deep learning based methods, indicating its effectiveness in feature learning.

Convolutional neural network (CNN) is well known for its capability of feature learning and has made revolutionary achievements in many applications, such as scene recognition and target detection. In this paper, its capability of feature learning in hyperspectral images is explored by constructing a five-layer CNN for classification (C-CNN). The proposed C-CNN is constructed by including recent advances in deep learning area, such as batch normalization, dropout, and parametric rectified linear unit (PReLU) activation function. In addition, both spatial context and spectral information are elegantly integrated into the C-CNN such that spatial-spectral features are learned for hyperspectral images. A companion feature-learning CNN (FL-CNN) is constructed by extracting fully connected feature layers in this C-CNN. Both supervised and unsupervised modes are designed for the proposed FL-CNN to learn sensor-specific spatial-spectral features. Extensive experimental results on four benchmark data sets from two well-known hyperspectral sensors, namely airborne visible/infrared imaging spectrometer (AVIRIS) and reflective optics system imaging spectrometer (ROSIS) sensors, demonstrate that our proposed C-CNN outperforms the state-of-the-art CNN-based classification methods, and its corresponding FL-CNN is very effective to extract sensor-specific spatial-spectral features for hyperspectral applications under both supervised and unsupervised modes.

The performance of hyperspectral classification is affected by within-class spectral variation since different materials may present similar spectral signatures. In this paper, we investigate how to fully use the low-rank property of hyperspectral images to alleviate spectra variation. Particulary, two effective strategies that explore the low-rank property in local spectral and spatial space are proposed. According to experimental results, we conclude that exploring the low-rank property in local spectral-spatial space can help to alleviate spectral variation and improve the performance of classification obviously for all tested data, while exploring the low-rank property in spatial space is more effective for images presenting large homogeneous areas.

Deep convolutional neural networks (CNNs) have brought in achievements in image classification and target detection. In this paper, we propose a novel five-layer CNN for hyperspectral classification by encountering recent achievement in deep learning area, such as batch normalization, dropout, Parametric Rectified Linear Unit (PReLu) activation function. By taking advantage of the specific characteristics of hyperspectral images, spatial context and spectral information are elegantly integrated into the framework. Experimental results demonstrate that our proposed CNN out- performs the state-of-the-art methods.

Spectral variation is profound in remotely sensed images due to variable imaging conditions. The wide presence of such spectral variation degrades the performance of hyperspectral analysis, such as classification and spectral unmixing. In this letter, 11-based low-rank matrix approximation is proposed to alleviate spectral variation for hyperspectral image analysis. Specifically, hyperspectral image data are decomposed into a low-rank matrix and a sparse matrix, and it is assumed that intrinsic spectral features are represented by the low-rank matrix and spectral variation is accommodated by the sparse matrix. As a result, the performance of image data analysis can be improved by working on the low-rank matrix. Experiments on benchmark hyperspectral data sets demonstrate the performance of classification, and spectral unmixing can be clearly improved by the proposed approach.

Endmember extraction (EE) has been widely utilized to identify spectrally unique signatures of pure ground materials in hyperspectral images. Most of existing EE algorithms focus on spectral signature only, denoted as spectral EE (sEE) algorithms in this paper. In order to improve the performance of these sEE algorithms by considering spatial information, a novel spatial preprocessing (SPP) strategy based on Locally Linear Embedding (LLE) is proposed to alleviate the influence of spectral variation. Specifically, the LLE is adopted to revise pixels by smoothing spectral variation in their spatial neighborhood. Furthermore, anomalous pixels, which may be smoothed excessively by many current SPP algorithms, can be well retained by tuning off the spatial preprocessing if their signatures are revised unexpectively. As a result, the anomalous endmembers can be correctly identified by the proposed LLE based SPP algorithm. Experimental results on simulated benchmark dataset have demonstrated that the proposed LLE based SPP algorithm outperforms many state-of-the-art SPP algorithms.

The contradiction that imaging system can acquire huge amount of image data while communication system can deliver only a very small part of them has become a bottleneck for the small-satellite based earth observation. In this paper, a novel onboard image selection strategy is designed to select most informative images that are acquired by the imaging system for transmission. Specifically, the image that is worst reconstructed by previously transmitted images, instead of the instantly acquired image, is selected since it possesses most distinguishing information to previously transmitted images. Experiment on simulated image sequence has demonstrated the effectiveness of the proposed onboard image selection algorithm.

Hyperspectral (HS) remote sensing image with finer spectral information has great advantages in feature identification and classification. However, the spatial resolution of HS image is usually low due to practical limitations. In this paper, the low-spatial-resolution HS image is fused with the high-spatial-resolution multispectral (MS) image of the same observation scene to improve its spatial resolution. A novel spectral unmixing based HS and MS image fusion approach (VSC-CNMF) is proposed, in which CNMF with minimum endmember simplex volume and abundance sparsity constraints is employed for coupled unmixing of HS and MS images. Simulative experiments are employed for verification and comparison. The experimental results illustrate that the newly proposed VSC-CNMF based HS and MS fusion algorithm outperforms several state-of-the-art unmixing based fusion approaches in cases with moderate number of endmembers.

Video Summarization (VS) techniques have been widely utilized to produce a concise video content representation, such that the video content can be quickly explored and the complexity of video based analysis and retrieval applications can be highly reduced. However, little attention has been paid for on-line applications, especially for resource restricted applications, such as onboard VS. In this paper, our previous on-line Minimum Sparse Reconstruction (OnMSR) based VS algorithm is improved for resources restricted applications by confining the size of keyframes for reconstruction. Specially, an on-line reconstruction keyframe set update strategy is designed to meet the requirement of real-time resource restricted situation. Experimental results on various types of videos demonstrate the performance of OnMSR does not vary much by imposing resource constraint in the proposed resource restricted OnMSR (RR-onMSR) algorithm. As a result, the proposed RR-onMSR is very effective for real-time onboard VS applications.

Recent developments on sparse dictionary selection have demonstrated promising results for Video Summarization (VS). However, the convex relaxation based solution cannot ensure the sparsity of the dictionary directly. In this paper, a selection matrix is proposed to model the VS problem, according to which the L0norm of this selection matrix is imposed to ensure sparsity directly. As a result, a computational efficient Orthogonal Subspace Projection (OSP) based Iterative Keyframe Selection (IKS) algorithm is proposed for VS. In addition, a Percentage Of Reconstruction (POR) criterion is proposed to provide an intuitive and flexible control of the length of final video summaries even without prior knowledge of a given video. Experimental results on a popular benchmark dataset demonstrate that our proposed algorithm outperforms the state-of-the-art methods.

The ever increasing volume of video content has created profound challenges for developing efficient video summarization (VS) techniques to access the data. Recent developments on sparse dictionary selection have demonstrated promising results for VS, however, the convex relaxation based solution cannot ensure the sparsity of the dictionary directly and it selects keyframes in a local point of view. In this paper, an L2,0constrained sparse dictionary selection model is proposed to reformulate the problem of VS. In addition, a simultaneous orthogonal matching pursuit (SOMP) based method is proposed to obtain an approximate solution for the proposed model without smoothing the penalty function, and thus selects keyframes in a global point of view. In order to allow for intuitive and flexible configuration of VS process, a percentage of residuals (POR) criterion is also developed to produce video summaries in different lengths. Experimental results demonstrate that our proposed method outperforms the state-of-the-art.

The Hopfield neural network (HNN) has been demonstrated to be an effective tool for the spectral mixture unmixing of hyperspectral images. However, it is extremely time consuming for such per-pixel algorithm to be utilized in real-world applications. In this letter, the implementation of a multichannel structure of HNN (named as MHNN) on a graphics processing unit (GPU) platform is proposed. According to the unmixing procedure of MHNN, three levels of parallelism, including thread, block, and stream, are designed to explore the peak computing capacity of a GPU device. In addition, constant and texture memories are utilized to further improve its computational performance. Experiments on both synthetic and real hyperspectral images demonstrated that the proposed GPU-based implementation works on the peak computing ability of a GPU device and obtains several hundred times of acceleration versus the CPU-based implementation while its unmixing performance remains unchanged.

In this paper, a hyperspectral (HS) image resolution enhancement algorithm based on spectral unmixing is proposed for the fusion of the high-spatial-resolution multispectral (MS) image and the low-spatial-resolution HS image (HSI). As a result, a high-spatial-resolution HSI is reconstructed based on the high spectral features of the HSI represented by endmembers and the high spatial features of the MS image represented by abundances. Since the number of endmembers extracted from the MS image cannot exceed the number of bands in least-squares-based spectral unmixing algorithm, large reconstruction errors will occur for the HSI, which degrades the fusion performance of the enhanced HSI. Therefore, in this paper, a novel fusion framework is also proposed by dividing the whole image into several subimages, based on which the performance of the proposed spectral-unmixing-based fusion algorithm can be further improved. Finally, experiments on the Hyperspectral Digital Imagery Collection Experiment and Airborne Visible/Infrared Imaging Spectrometer data demonstrate that the proposed fusion algorithms outperform other famous fusion techniques in both spatial and spectral domains.

While most existing video summarization approaches aim to identify important frames of a video from either a global or local perspective, we propose a top-down approach consisting of scene identification and scene summarization. For scene identification, we represent each frame with global features and utilize a scalable clustering method. We then formulate scene summarization as choosing those frames that best cover a set of local descriptors with minimal redundancy. In addition, we develop a visual word-based approach to make our approach more computationally scalable. Experimental results on two benchmark datasets demonstrate that our proposed approach clearly outperforms the state-of-the-art.

The well-known difficulty in supervised hyperspectral image classification is the limited availability of training data, which are expensive, and quite difficult to access and to obtain in real remote sensing scenarios. The Support Vector Machine (SVM) technique has been proven to be well suited to classify hyperspectral data by using limited number of training samples. In this paper, modifications over Iterative Support Vector Machine algorithm have been proposed incorporating both spatial and spectral information and correcting the training samples at each iteration in order to increase the classification performance over SVM. In order to demonstrate the effectiveness of the proposed framework, experiments on AVIRIS data over Indian Pine Site (IPS) are conducted to compare the performance of the proposed classification approach against some existing classification techniques such as Linear-SVM, SVM-RBF, ISVM and K-NN. Experimental results demonstrate that the proposed method clearly outperform the well-known classification algorithms.

Nonnegative Matrix Factorization (NMF) has been successfully employed to address the mixed-pixel problem of hyperspectral remote sensing images. However, minimizing the representation error by NMF is not sufficient for SMA since the unmixing results of NMF are not unique. Therefore, in this paper, a neighborhood preserving regularization, which preserves the local structure of the hyperspectral data on a low-dimensional manifold, is proposed to constrain NMF for unique solution in SMA. As a result, a Neighborhood Preserving constrained NMF (NP-NMF) algorithm is proposed for SMA of highly mixed hyperspectral data. Finally, experimental results on AVIRIS data demonstrate the effectiveness of our proposed NP-NMF algorithm for SMA applications.

In order to enhance the spatial resolution of the hyperspectral images, a novel fast algorithm based on Spectral Mixture Analysis (SMA) techniques is proposed for the fusion of coarse-resolution hyperspectral (HS) image and high-resolution multispectral (MS) image. The high-resolution hyperspectral image is synthesized by integrating high-resolution spectral information of hyperspectral image represented by endmembers and high-resolution spatial information of multispectral image represented by abundance. As a result, a novel SMA based diagram is designed, in which Endmember Extraction (EE) is performed on hyperspectral images while Abundance Estimation is performed on multispectral images, and the unmixing process in these two images are matched by utilizing the spectral response matrix and the spatial spread transform matrix in the observation model. Finally, real HYDICE data experiments are utilized to demonstrate the effectiveness of the proposed fusion algorithm.

Spectral Mixture Analysis (SMA) has been widely utilized to address the mixed-pixel problem in the quantitative analysis of hyperspectral remote sensing images. Recently Nonnegative Matrix Factorization (NMF) has been successfully utilized to simultaneously perform endmember extraction (EE) and abundance estimation (AE). In this paper, we formulate the solution of NMF by performing EE and AE iteratively. Based on our previous Hopfield Neural Network (HNN) based AE algorithm, an HNN is also constructed for EE to solve the multiplicative updating problem of NMF for SMA. As a result, SMA is conducted in an unsupervised manner and our algorithm is able to extract virtual endmembers without assuming the presence of spectrally pure constituents in hyperspectral scenes. We further extend such strategy to solve the constrained NMF (cNMF) models for SMA, where extra constraints are imposed to better model the mixed-pixel problem. Experimental results on both synthetic and real hyperspectral images demonstrate the effectiveness of our proposed HNN based unsupervised SMA algorithms.

Nonnegative Matrix Factorization (NMF) and its extensions have gained lots of attentions in Spectral Mixture Analysis (SMA) since they can handle highly mixed hyperspectral pixels in an unsupervised way. In order to overcome the non-uniqueness problem in NMF, a minimum endmember-wise distance constraint (MewDC), which optimizes endmember spectra as compact as possible, is imposed for satisfying unmixing results. The proposed constraint works similar to minimum volume constraint (MVC). However, the dimension reduction step and numerical instability problems in MVC can be avoided. As a result, a minimum endmember-wise distance constrained NMF (MewDC-NMF) algorithm is proposed to extract endmembers and estimate their corresponding fractional abundance simultaneously. Both synthetic and real hyperspectral data experiments have demonstrate the effectiveness of the proposed MewDC-NMF algorithm.

In this paper, a wavelet-based Bayesian fusion framework is presented, in which a low spatial resolution hyperspectral (HS) image is fused with a high spatial resolution multi-spectral (MS) image by accounting for the joint statistics. Particularly, a zero-mean heavy-tailed model, Gaussian Scale Mixture (GSM) model, is employed as the prior, which is believed to be capable of modelling the distribution of wavelet coefficients more accurately than traditional Gaussian model. To keep the calculations feasible, a practical implementation scheme is presented. The proposed approach is validated by simulation experiments for both general HS and MS image fusion as well as the specific case of pansharpening. The experimental results of the proposed approach are also compared with its counterpart employing a Gaussian prior for performance evaluation.

Endmember extraction (EE) has been widely utilized to extract spectrally unique and singular spectral signatures for spectral mixture analysis of hyperspectral images. Recently, spatial–spectral EE (SSEE) algorithms have been proposed to achieve superior performance over spectral EE (SEE) algorithms by taking both spectral similarity and spatial context into account. However, these algorithms tend to neglect anomalous endmembers that are also of interest. Therefore, in this paper, an improved SSEE (iSSEE) algorithm is proposed to address such limitation of conventional SSEE algorithms by accounting for both anomalous and normal endmembers. By developing simplex projection and simplex complementary projection, all the hyperspectral pixels are projected into a simplex determined by the normal endmembers extracted in conventional SSEE algorithms. As a result, anomalous endmembers are identified iteratively by utilizing the$l_{2}^{\infty}$norm to find the maximum simplex complementary projection. In order to determine how many anomalous endmembers are to be extracted, a novel Residual-be-Noise Probability-based algorithm is also proposed by elegantly utilizing the spatial-purity map generated in the previous SSEE step. Experimental results on both synthetic and real datasets demonstrate that simplex projection errors can be significantly reduced by identifying both anomalous and normal endmembers in the proposed iSSEE algorithm. It is also confirmed that the performance of the proposed iSSEE algorithm clearly outperforms that of SEE algorithms since both spatial context and spectral similarity are utilized.

Due to the spatial-resolution limitation, mixed pixels containing energy reflected from more than one type of ground objects are widely present in remote sensing images, which often results in inefficient quantitative analysis. To effectively decompose such mixtures, a fully constrained linear unmixing algorithm based on a multichannel Hopfield neural network (MHNN) is proposed in this letter. The proposed MHNN algorithm is actually a Hopfield-based architecture which handles all the pixels in an image synchronously, instead of considering a per-pixel procedure. Due to the synchronous unmixing property of MHNN, a noise energy percentage (NEP) stopping criterion which utilizes the signal-to-noise ratio is proposed to obtain optimal results for different applications automatically. Experimental results demonstrate that the proposed multichannel structure makes the Hopfield-based mixture analysis feasible for real-world applications with acceptable time cost. It has also been observed that the proposed MHNN-based mixture-analysis algorithm outperforms the other two popular linear mixture-analysis algorithms and that the NEP stopping criterion can approach optimal unmixing results adaptively and accurately.

Spectral mixture analysis (SMA) has been widely utilized to address the mixed-pixel problem in the quantitative analysis of hyperspectral remote sensing images, in which endmember extraction (EE) plays an extremely important role. In this paper, a novel algorithm is proposed to integrate both spectral similarity and spatial context for EE. The spatial context is exploited from two aspects. At first, initial endmember candidates are identified by determining the spatial purity (SP) of pixels in their spatial neighborhoods (SNs). Several SP measurements are investigated at both intensity level and feature level. In order to alleviate local spectra variability, the average of the pixels in pure SNs are voted as endmember candidates. Then, the spatial connectivity is utilized to merge spatially related endmember candidates by finding connection paths in a graph so that the number of endmember candidates is further reduced, which results in computational efficiency and better performance in SMA by alleviating global spectral variability. Experimental results on both synthetic and real hyperspectral images demonstrate that the proposed SP based EE (SPEE) algorithm outperforms the other popular EE algorithms. It is also observed that feature-level SP measurements are more distinguishable than intensity-level SP measurements to discriminate pure SNs from mixed SNs.