Hyperspectral images (HSIs) typically have finer spectral resolution but coarser spatial resolution than multispectral images (MSIs). To obtain HSIs with enhanced spatial resolution, considerable emphasis has been placed on achieving hyperspectral super-resolution (SR) by fusing HSIs with MSIs in the same scene. However, most existing HSI-MSI fusion methods either rely on prior knowledge of degradation models or require sufficient training data, hindering their practicality and interpretability. This letter proposes a deep multistream network (DMSN) for HSI SR. Specifically, we introduce the Spa-DNet and the Spe-UNet modules to encode spatial and spectral transformations across resolutions. Furthermore, we design the Int-Net to achieve spatial and spectral information interaction, enhancing the model’s performance. Finally, the proposed approach enables high spatial and spectral resolution HSIs. Using the newly designed three-stage training strategy, the network parameters can exhibit the clear physical significance of the degradation process, thereby helping to ensure faithful reconstruction of the desired HSIs. Experimental results with real datasets demonstrate that the proposed DMSN performs better than other methods. The codes will be available athttps://github.com/yuanchaosu/dmsn-GRSL.

Unsupervised hyperspectral image super-resolution (HSI-SR) has recently emerged as a popular and active research topic in remote sensing data fusion. However, most methods neglect the non-local features of the data in representation learning, which limits their fusion performances. To overcome the issue, we propose a Graph Attention-based Fusion Network (FusGAT) in this letter. This approach first extracts local features from the input data using multi-scale convolutions, and then the graph attention mechanism is employed to model relationships between nodes in the spectral stream for deriving non-local features of the image and transferring them to the spatial stream. FusGAT will iteratively update the node connections and refine node embedding, facilitating the extraction of non-local features and enabling effective information flow between the streams. We conducted several experiments on two datasets to prove the effectiveness of the proposed method. The source code will be available at: https://github.com/yuanchaosu/FusGAT-GRSL.

Most existing methods for predicting drug-drug interactions (DDI) predominantly concentrate on capturing the explicit relationships among drugs, overlooking the valuable implicit correlations present between drug pairs (DPs), which leads to weak predictions. To address this issue, this paper introduces a hierarchical multi-relational graph representation learning (HMGRL) approach. Within the framework of HMGRL, we leverage a wealth of drug-related heterogeneous data sources to construct heterogeneous graphs, where nodes represent drugs and edges denote clear and various associations. The relational graph convolutional network (RGCN) is employed to capture diverse explicit relationships between drugs from these heterogeneous graphs. Additionally, a multi-view differentiable spectral clustering (MVDSC) module is developed to capture multiple valuable implicit correlations between DPs. Within the MVDSC, we utilize multiple DP features to construct graphs, where nodes represent DPs and edges denote different implicit correlations. Subsequently, multiple DP representations are generated through graph cutting, each emphasizing distinct implicit correlations. The graph-cutting strategy enables our HMGRL to identify strongly connected communities of graphs, thereby reducing the fusion of irrelevant features. By combining every representation view of a DP, we create high-level DP representations for predicting DDIs. Two genuine datasets spanning three distinct tasks are adopted to gauge the efficacy of our HMGRL. Experimental outcomes unequivocally indicate that HMGRL surpasses several leading-edge methods in performance.

Multimodal fusion widely uses convolutional layers to capture local correlations and adjust feature dimensions. However, the progressive expansion of the receptive field in convolutional layers often compromises spatial context retention, leading to the loss of fine details. Furthermore, the fixed-size kernels typically used in standard convolution restrict the network’s ability to capture multiscale contextual details. To address this limitation, this paper develops a dilated transformation-guided unsupervised multimodal learning (DTUML) method to fuse a high-resolution multispectral image (HR-MSI) and a low-resolution hyperspectral image (LR-HSI), thereby generating a high-resolution hyperspectral image (HR-HSI). Our DTUML adopts a dual-stream encoder architecture to conduct multimodal data, where one stream focuses on preserving spectral information from LR-HSIs, while the other emphasizes the acquisition of spatial details from HR-MSIs. These complementary features are subsequently integrated to ensure spectral fidelity and retain spatial detail. Then, a convolutional layer restores dimensional consistency and outputs an HR-HSI. Extensive experiments demonstrate the effectiveness of DTUML, showing superior performance and strong competitiveness compared to state-of-the-art methods. Code:https://github.com/yuanchaosu/TGRS-DTUML.

Convolution Neural Networks (CNNs) have demonstrated strong feature extraction capabilities in Euclidean spaces, achieving remarkable success in hyperspectral image (HSI) classification tasks. Meanwhile, Graph convolution networks (GCNs) effectively capture spatial-contextual characteristics by leveraging correlations in non-Euclidean spaces, uncovering hidden relationships to enhance the performance of HSI classification (HSIC). Methods combining GCNs with CNNs have achieved excellent results. However, existing GCN methods primarily rely on single-scale graph structures, limiting their ability to extract features across different spatial ranges. To address this issue, this paper proposes a multiscale segmentation-guided fusion network (MS2FN) for HSIC. This method constructs pixel-level graph structures based on multiscale segmentation data, enabling the GCN to extract features across various spatial ranges. Moreover, effectively utilizing features extracted from different spatial scales is crucial for improving classification performance. This paper adopts distinct processing strategies for different feature types to enhance feature representation. Comparative experiments demonstrate that the proposed method outperforms several state-of-the-art (SOTA) approaches in accuracy. The source code will be released at https://github.com/shengrunhua/MS2FN.

Vision transformer (ViT) has recently been a popular topic in the foundation model field, taking advantage of its strong scalability and outstanding representation capabilities. As a deep model, ViT introduces a new architecture for achieving hyperspectral image (HSI) unmixing. However, traditional ViTs overlook pixel-level spatial continuity by partitioning the input image into nonoverlapping fixed-size patches. This approach disrupts local structural relationships and hinders the model's ability to capture fine-grained spatial dependencies, resulting in suboptimal feature representation for dense prediction tasks in unmixing. To address these challenges, this article proposes the development of a self-supervised relation-aware ViT (SRViT). SRViT incorporates a self-embedded module comprising encoders, a pixel-level position encoder (PLPE), a self-supervised contrastive mechanism (SCM), and a decoder. The self-embedded module and PLPE preserve local correlations in HSI across different views, facilitating cross-view learning through SCM to ensure generalization. In addition, the decoder incorporates Kronecker-factored approximate curvature (K-FAC) to capture the local geometric structure of spectral information. Ultimately, SRViT learns endmembers and fractional abundance as the unmixing result. The effectiveness and competitiveness of SRViT have been systematically validated through comparative experiments, demonstrating its superior performance. The source code is available at the following link: https://github.com/yuanchaosu/TNNLS-SRViT.

Hyperspectral super-resolution (SR) can effectively enhance the spatial resolution of hyperspectral images, holding significant application value. Nevertheless, existing methods tend to overlook global information and some detailed aspects of hyperspectral images, resulting in limitations in feature extraction. In addressing this issue, we propose a Dilated Attention With Multi-Stream Learning (DAMS) network to facilitate the fusion of hyperspectral and multispectral images. The network comprises three autoencoders, incorporating dilated residual multipath feature extraction for high-resolution multispectral images and a dense convolutional neural network for low-resolution hyperspectral images. Notably, no prior knowledge of point spread function (PSF) and spectral response function (SRF) is required. Experimental results with DAMS showcase its advantages over other super-resolution fusion methods, demonstrating robust performance across diverse datasets with varying PSF and SRF.

In recent years, deep learning has been proposed for hyperspectral images(HSIs) super-resolution, and many fusion models for HSI and multispectral images(MSIs) have been developed. However, these networks are constrained to the structure of convolutional neural networks(CNNs), and more attention needs to be paid to the disadvantage of the restricted receptive field of CNNs, such that some of the distal information needs to be included in the process of acquiring features. This approach involves capturing large-scale spatial features through multi-head spatial attention and spectral features of MSI and HSI through multi-head spectral attention. Subsequently, the features are processed by convolution kernels of different scales compactly. The effectiveness and competitiveness of MTSANet are evaluated by comparing it with some state-of-the-art (SOTA) methods.

Hyperspectral unmixing can handle the mixed pixels in hyperspectral images (HSIs). Shadows of objects in observed areas are recorded by sensors, resulting in an HSI contaminated by shadows. Therefore, shadow pollution is a grievous obstacle for unmixing applications. Although shadow pollution occurs frequently in HSIs, previous unmixing studies have never considered the interference caused by shadows. Hence, mitigating shadow interference for unmixing will be significant for further acquiring subpixel information. In this letter, we employ a generative adversarial autoencoder (GAA) to develop a supervised unmixing method that can substantially reduce the impacts of shadow for unmixing. Specifically, we adopt the GAA to establish an anti-shadow unmixing network (GAA-AS), where the encoder block is used to feature reinforcement, and the decoder serves for abundance estimation. Moreover, we adopt a spectral-aware loss (SAL) as the loss function of adversarial training, which makes the discriminator better capture the difference between pixels. Finally, a softmax layer is adopted for the abundance sum-to-one constraint (ASC). Several experiments verify the effectiveness and advantages of our GAA-AS. In the experiment with shadow-polluted data, the proposed GAA-AS improves accuracies by approximately 70% compared to SOTA approaches in the quantitative experiment with synthetic data, and the impacts of shadow pollution are also significantly alleviated in the experiment with real shadow-polluted HSIs. Additionally, note that the proposed GAA-AS is competitive even when no shadow exists in HSIs, verified by the experiment with shadowless data.

Transformer holds significance in deep learning (DL) research. Node embedding (NE) and positional encoding (PE) are usually two indispensable components in a Transformer. The former can excavate hidden correlations from the data, while the latter can store locational relationships between nodes. Recently, the Transformer has been applied for hyperspectral image (HSI) classification because the model can capture long-range dependencies to aggregate global features for representation learning. In an HSI, adjacent pixels tend to be homogeneous, while the NE does not identify the positional information of pixels. Therefore, PE is crucial for Transformers to understand locational relationships between pixels. However, in this area, most Transformer-based methods randomly generate PEs without considering their physical meaning, which leads to weak representations. This article proposes a new graph generative structure-aware Transformer (GraphGST) to solve the above-mentioned PE problem when implementing HSI classification. In our GraphGST, a new absolute PE (APE) is established to acquire pixels’ absolute positional sequences (APSs) and is integrated into the Transformer architecture. Moreover, a generative mechanism with self-supervised learning is developed to achieve cross-view contrastive learning (CL), aiming to enhance the representation learning of the Transformer. The proposed GraphGST model can capture local-to-global correlations, and the extracted APSs can complement the spectral features of pixels to assist in NE. Several experiments with real HSIs are conducted to evaluate the effectiveness of our GraphGST. The proposed method demonstrates very competitive performance compared with other state-of-the-art (SOTA) approaches. Our source codes will be provided in the following linkhttps://github.com/yuanchaosu/TGRS-graphGST.

In recent years, deep learning (DL) has accelerated the development of hyperspectral image (HSI) processing, expanding the range of applications further. As a typical model of unsupervised DL, the autoencoder framework has been extensively applied for spectral unmixing due to its strong representation ability and scalability. Nowadays, most DL-based unmixing approaches adopt the linear mixture model (LMM) to estimate pure spectral signatures (endmembers) and their corresponding abundance fractions. However, since sunlight scattering is an inevitable physical phenomenon, the spectral mixture problem is inherently nonlinear. Moreover, most existing nonlinear unmixing approaches focus exclusively on spectral information, neglecting the spatial distribution of materials and the intrinsic correlation between pixels, making it challenging to explore latent features. To address these issues, this article develops a new deep autoencoder-based augmented network (DAAN). The proposed DAAN employs the multilinear mixture model (MLMM) to handle the nonlinear influence caused by multiple scattering. Meanwhile, the proposed DAAN constraints homogenous smoothing in the autoencoder architecture, enabling the aggregation of intrinsic correlations by means of spatial relationships to enhance the performance of abundance estimation. We achieve unsupervised nonlinear hyperspectral unmixing by combining spectral and spatial information. The effectiveness and advantages of DAAN are confirmed by several experiments with synthetic and real HSI datasets. The results indicate that the proposed method outperforms other DL-based unmixing approaches. The source codes of the proposed DAAN will be provided in the following linkhttps://github.com/yuanchaosu/TGRS-daan.

This paper presents CellFusion, a system designed for high-quality, real-time video streaming from vehicles to the cloud. It leverages an innovative blend of multipath QUIC transport and network coding. Surpassing the limitations of individual cellular carriers, CellFusion uses a unique last-mile overlay that integrates multiple cellular networks into a single, unified cloud connection. This integration is made possible through the use of in-vehicle Customer Premises Equipment (CPEs) and edge-cloud proxy servers.

Spatial-spectral classification (SSC) has become a trend for hyperspectral image (HSI) classification. However, most SSC methods mainly consider local information, so that some correlations may not be effectively discovered when they appear in regions that are not contiguous. Although many SSC methods can acquire spatial-contextual characteristics via spatial filtering, they lack the ability to consider correlations in non-Euclidean spaces. To address the aforementioned issues, we develop a new semisupervised HSI classification approach based on normalized spectral clustering with kernel-based learning (NSCKL), which can aggregate local-to-global correlations to achieve a distinguishable embedding to improve HSI classification performance. In this work, we propose a normalized spectral clustering (NSC) scheme that can learn new features under a manifold assumption. Specifically, we first design a kernel-based iterative filter (KIF) to establish vertices of the undirected graph, aiming to assign initial connections to the nodes associated with pixels. The NSC first gathers local correlations in the Euclidean space and then captures global correlations in the manifold. Even though homogeneous pixels are distributed in noncontiguous regions, our NSC can still aggregate correlations to generate new (clustered) features. Finally, the clustered features and a kernel-based extreme learning machine (KELM) are employed to achieve the semisupervised classification. The effectiveness of our NSCKL is evaluated by using several HSIs. When compared with other state-of-the-art (SOTA) classification approaches, our newly proposed NSCKL demonstrates very competitive performance. The codes will be available at https://github.com/yuanchaosu/TCYB-nsckl.

Deep learning (DL) has brought many new trends for hyperspectral image classification (HIC). Graph neural networks (GNNs) are models that fuse DL and structured data. Although GNN-based methods have focused on modeling relations, most of them are susceptible to noise, being adverse to capturing hidden correlations from data. Moreover, existing related approaches typically adopt changeless graph structures, which might lead to poor generalization. To solve the problems mentioned above, this paper develops an adaptive cuckoo refinement-based graph transfer network (ACGT-Net) that introduces a meta-heuristic optimization strategy to refine the graph structure. Specifically, we first pre-train a graph convolutional network (GCN) to learn transferable weight parameters. In the undirected graph, nodes are associated with pixels, and edges correspond to similarities between nodes. Afterward, we integrate a cuckoo search strategy (CSS) into the trained GCN to adaptively refine the graph structure. The graph structure refinement (GSR) with the CSS can pay more attention to significant channels by global optimization to improve the generalization of the GNN. Several experiments with real datasets verify the effectiveness and competitiveness of our ACGT-Net compared with other state-of-the-art (SOTA) methods.

Transformer is a powerful tool for capturing long-range dependencies and has shown impressive performance in hyperspectral image (HSI) classification. However, such power comes with a heavy memory footprint and huge computation burden. In this paper, we propose two types of lightweight self-attention modules (a channel lightweight multi-head self-attention module and a position lightweight multi-head self-attention module) to reduce both memory and computation while associating each pixel or channel with global information. Moreover, we discover that transformers are ineffective in explicitly extracting local and multi-scale features due to the fixed input size and tend to overfit when dealing with a small number of training samples. Therefore, a lightweight transformer (LiT) network, built with the proposed lightweight self-attention modules, is presented. LiT adopts convolutional blocks to explicitly extract local information in early layers and employs transformers to capture long-range dependencies in deep layers. Furthermore, we design a controlled multi-class stratified sampling strategy to generate appropriately sized input data, ensure balanced sampling, and reduce the overlap of feature extraction regions between training and test samples. With appropriate training data, convolutional tokenization, and lightweight transformers, LiT mitigates overfitting and enjoys both high computational efficiency and good performance. Experimental results on several HSI datasets verify the effectiveness of our design.

This work proposes a new hyperspectral image classification method based on chaotic cuckoos (CC) search with graph convolution network (CC-GCN). Although GCNs can extract inherent features by node embeddings, most models neglect fragmented relations in the spectral domain. The proposed CC-GCN can refine the graph structure and reduce the redundancy information of spectral dimension, further improving the classification accuracy of hyperspectral images. The experiment results demonstrate the effectiveness of the proposed CC-GCN.

Dimensionality reduction (DR) is a common preprocessing technology for hyperspectral images (HSIs). Recently, many neural networks can implement DR to remove the re-dundant information by node embedding. However, numer-ous hidden-layer parameters limit the generalization ability of the node embedding. In this paper, we develop a graph-cut-based node embedding (GCNE) that can be used for DR of HSIs. The embedding can refine correlations by a graph-cut strategy, and it can avoid numerous parameters when using graph models. Moreover, we combine the graph-cut strategy and extreme learning machine (ELM) to achieve HSI classi-fication. The effectiveness of the proposed method is verified by using real HSIs. Compared with other state-of-the-art DR and classification methods, the proposed approach demon-strates very competitive performance.

Node embedding (NE) is conducive to aggregating correlations and relieving the influence of the Hughes phenomenon when processing high-dimensional data. Although some graph neural networks can capture correlations during achieving NE, the application of NE still faces two rigorous challenges: numerous model parameters and poor generalization. In this letter, we propose a new approach for hyperspectral image (HSI) classification, called the graph-cut-based collaborative NEs (GCCNE). Specifically, we develop a graph-cut-based NE (GCNE) to achieve low-dimensional feature representation, which avoids numerous model parameters when using a graph structure. Considering that the graph-cut in a low-dimensional space does not need to set anchors to decrease the calculation amount, we adopt an ensemble framework based on random subspaces (RSs) to implement the GCNE to obtain the collaborative feature sets, enhancing the generalization of feature representation. Afterward, the collaborative feature sets are input in several kernel-based extreme learning machines (KELMs), respectively, classifying pixels. The number of RSs is the same as the number of KELMs. Finally, we acquire an ensemble result associated with each class. The effectiveness and competitiveness of the proposed method are evaluated by using real HSI datasets.

Hyperspectral unmixing is an important problem for remotely sensed data interpretation. It amounts at estimating the spectral signatures of the pure spectral constituents in the scene (endmembers) and their corresponding subpixel fractional abundances. Although the unmixing problem is inherently nonlinear (due to multiple scattering), the nonlinear unmixing of hyperspectral data has been a very challenging problem. This is because nonlinear models require detailed knowledge about the physical interactions between the sunlight scattered by multiple materials. In turn, bilinear mixture models (BMMs) can reach good accuracy with a relatively simple model for scattering. In this article, we develop a new BMM and a corresponding unsupervised unmixing approach which consists of two main steps. In the first step, a deep autoencoder is used to linearly estimate the endmember signatures and their associated abundance fractions. The second step refines the initial (linear) estimates using a bilinear model, in which another deep autoencoder (with a low-rank assumption) is adapted to model second-order scattering interactions. It should be noted that in our developed BMM model, the two deep autoencoders are trained in a mutually interdependent manner under the multitask learning framework, and the relative reconstruction error is used as the stopping criterion. The effectiveness of the proposed method is evaluated using both synthetic and real hyperspectral data sets. Our experimental results indicate that the proposed approach can reasonably estimate the nature of nonlinear interactions in real scenarios. Compared with other state-of-the-art unmixing algorithms, the proposed approach demonstrates very competitive performance.

In this paper, we propose a hyperspectral anomaly detection approach based iForest (Isolation Forest) with band clustering. Instead of random selecting a feature from all bands in iForest algorithm, the proposed approach designed the following three main steps. Firstly, all bands were divided into several groups by use of the correlation among bands and optimal clustering. Then, one of the groups was randomly selected as a candidate. Finally, a band was randomly selected from the candidate group as an attribute for tree node splitting. Compared with other anomaly detection methods, our approach for attributes selection can not only handle high dimensional problem, but also reduce the probability that important information was ignored. The experiments demonstrate its robustness and competitive performance.

In this letter, we propose a new hyperspectral image (HSI) classification approach, called the random subspace ensemble with enhanced feature (RSE-EF), which trains several individual classifiers with enhanced spatial information. The proposed approach aims to address two common issues: the curses of the imbalanced training samples and high feature-to-instance ratio. Specifically, we first propose a similar-neighboring-sample-search (SNSS) method to address the issue of imbalanced training samples. Afterward, we generate the enhanced random subspaces (ERSs) that possess relatively lower dimensionality and more distinctive information compared with the original random subspaces (RSs) so as to alleviate the curse of high feature-to-instance ratio more effectively. Furthermore, a shallow neural network kernel-based extreme learning machine (KELM) is applied to the RSE-EF to classify image pixels. Experimental results on two public hyperspectral data sets illustrate that the proposed RSE-EF approach outperforms the state-of-the-art HSI classification counterparts.

Nonlinear unmixing of hyperspectral images has been a very challenging research problem, as it needs to consider the physical interactions between the sunlight scattered by multiple materials. In this paper, we propose a new approach for nonlinear unmixing which is based on multi-task learning (MTL) with low-rank matrix factorization (LRMF). The proposed approach establishes two tasks to conduct the unmixing problem under a nonlinear mixing model. In the first task, we employ LRMF to obtain endmember signatures and their corresponding abundance fractions simultaneously. Then, the second task uses LRMF to solve interactions from multiple scattering. The effectiveness of the proposed method is verified by using real hyperspectral data. Compared with other state-of-the-art nonlinear unmixing algorithms, the proposed approach demonstrates very competitive performance.

Spectral unmixing is a technique for remotely sensed image interpretation that expresses each (possibly mixed) pixel as a combination of pure spectral signatures (endmembers) and their fractional abundances. In this paper, we develop a new technique for unsupervised unmixing which is based on a deep autoencoder network (DAEN). Our newly developed DAEN consists of two parts. The first part of the network adopts stacked autoencoders (SAEs) to learn spectral signatures, so as to generate a good initialization for the unmixing process. In the second part of the network, a variational autoencoder (VAE) is employed to perform blind source separation, aimed at obtaining the endmember signatures and abundance fractions simultaneously. By taking advantage from the SAEs, the robustness of the proposed approach is remarkable as it can unmix data sets with outliers and low signal-to-noise ratio. Moreover, the multihidden layers of the VAE ensure the required constraints (nonnegativity and sum-to-one) when estimating the abundances. The effectiveness of the proposed method is evaluated using both synthetic and real hyperspectral data. When compared with other unmixing methods, the proposed approach demonstrates very competitive performance.

In this paper, we propose a deep auto-encoder network for the unmixing for hyperspectral data with outliers and low signal to noise ratio. The proposed deep auto-encoder network composes of two parts. The first part of the network adopts stacked non-negative sparse auto-encoder to learn the spectral signatures such that to generate a good initialization for the network. In the second part of the network, a variational auto-encoder is employed to perform unmixing, aiming at the endmember signatures and abundance fractions. The effectiveness of the proposed method is verified by using a synthetic data set. In our comparison with other state-of-the-art unmixing methods, the proposed approach demonstrates highly competitive performance.

As an unsupervised learning tool, autoencoder has been widely applied in many fields. In this letter, we propose a new robust unmixing algorithm that is based on stacked nonnegative sparse autoencoders (NNSAEs) for hyperspectral data with outliers and low signal-to-noise ratio. The proposed stacked autoencoders network contains two main steps. In the first step, a series of NNSAE is used to detect the outliers in the data. In the second step, a final autoencoder is performed for unmixing to achieve the endmember signatures and abundance fractions. By taking advantage from nonnegative sparse autoencoding, the proposed approach can well tackle problems with outliers and low noise-signal ratio. The effectiveness of the proposed method is evaluated on both synthetic and real hyperspectral data. In comparison with other unmixing methods, the proposed approach demonstrates competitive performance.

Endmember extraction is a fundamental task in spectral unmixing of remotely sensed hyperspectral images. In this work, we develop a new robust algorithm for endmember extraction which is based on a nonnegative sparse autoencoder. The proposed approach is based on two main steps. First, it uses an automatic sampler approach with local outlier factor and affinity propagation to intelligently gather a set of training samples. Then, a set of endmember signatures are extracted from the selected training samples by the nonnegative sparse autoencoder. Taking advantage from both automatic sampling and nonnegative sparse autoencoding, the proposed method can tackle problems with outliers. The effectiveness of the proposed method is verified by using simulated data. In our comparison with other state-of-the-art endmember extraction methods, the proposed approach demonstrates highly competitive performance.