Detail injection model-based methods are the mainstream pan-sharpening techniques for multispectral (MS) images. In recent years, the research on this type of method mainly focuses on optimizing the extraction and injection of panchromatic (PAN) image details, while paying less attention to the adaptive enhancement of MS image details. Due to the differences in spectral responses of different sources, it is difficult to effectively enhance or recover the multispectral details in the fused image. In this article, we analyze the limitations of the existing interpolation enhancement work from the perspective of model derivation, and propose an extended model of pan-sharpening detail injection based on “plug-and-play” (PnP) residual optimization. The model not only focuses on the flexibility in the choice of optimization routes and interpolation enhancement methods, but also emphasizes the universality of interpolation enhancement schemes across detail injection models. Our main contributions include the proposed residual interpolation optimization-based pan-sharpening extension model for detail injection oriented to additive and multiplicative rules, which successfully solves the problem of ineffective optimization in earlier related studies, especially for important multiresolution analysis (MRA) methods such as generalized Laplacian pyramid (GLP), and achieves universally effective optimization. In addition, through large-scale adaptive experiments, we selected 17 PnP methods including optimization model (OM) and deep learning (DL) methods for optimization tests, and comprehensively evaluated the applicability and effectiveness of the model. Comprehensive tests on 12 sets of images from five types of sensors on two public datasets show that our methods can achieve significant improvements in the main evaluation metrics, and the average ERGAS, Q2n, and HQNR metrics can reach 8.9%, 2.3%, and 6.3%, respectively. The source code of the proposed method can be downloaded fromhttps://github.com/JZ-Tao/EXP-PnP/.

Obtaining fine land surface change information from multitemporal hyperspectral images (HSIs) is a key goal pursued in remote sensing image processing. Recently, HSI change detection (HSI-CD) methods based on convolutional neural networks (CNNs) have achieved surprising detection results. One of the reasons is the support of large-scale labeled samples for network learning. However, the existence of mixed pixels greatly increases the difficulty of HSI interpretation, resulting in accurate pixel-level labeling work with a heavy burden and unable to meet the needs of time-sensitive applications. For this reason, achieving stable and high-precision CD with fewer samples is a difficult issue in this field. To address the above problems, a composite consistency-constrained self-ensemble framework (C3SelF) for HSI-CD is proposed, to alleviate the problems of low detection accuracy and instability caused by small samples. The framework mainly comprises two lightweight networks with the same structure aiming at accelerating the model inference process and thus improving the processing timeliness. The composite learning mode implements patch-wise classification loss, class-wise consistency loss on labeled samples, and patch-wise consistency loss on unlabeled samples under a multilevel noise perturbation strategy, which improves the classification results and reduces the labeling cost. Moreover, to exploit the multidimensional features contained in HSIs, a lightweight selective spatial-spectral feature joint network (S3Net) is designed to overcome over-fitting, and to deeply mine the discriminative information in unlabeled samples, a new sample screening strategy is designed to ensure the stability of the network during training unlabeled samples. Extensive experiments prove that the proposed C3SelF outperforms the state-of-the-art (SOTA) methods at a sampling rate of 0.1%, reaching 93.44% Kappa and 97.26% overall accuracy (OA) on the Farmland dataset. The source code of the proposed framework will be released athttps://github.com/zxylnnu/C3SelF.

Multisource remote sensing (RS) image fusion leverages data from various sensors to enhance the accuracy and comprehensiveness of Earth observation. Notably, the fusion of hyperspectral (HS) images and light detection and ranging (LiDAR) data has garnered significant attention due to their complementary features. However, current methods predominantly rely on simplistic techniques such as weight sharing, feature superposition, or feature products, which often fall short of achieving true feature fusion. These methods primarily focus on feature accumulation rather than integrative fusion. The transformer framework, with its self-attention mechanisms, offers potential for effective multimodal data fusion. However, simple linear transformations used in feature extraction may not adequately capture all relevant information. To address these challenges, we propose a novel multimodal contrastive fusion transformer (MCFT). Our approach employs convolutional neural networks (CNNs) for feature extraction from different modalities and leverages transformer networks for advanced fusion. We have modified the basic transformer architecture and propose a double position embedding mode to make it more suitable for RS image processing tasks. We introduce two novel modules: feature alignment module and feature matching module, designed to exploit both paired and unpaired samples. These modules facilitate more effective cross-modal learning by emphasizing the commonalities within the same features and the differences between features from distinct modalities. Experimental evaluations on several publicly available HS-LiDAR datasets demonstrate that proposed method consistently outperforms existing advanced methods. The source code for our approach is available at:https://github.com/SYFYN0317/MCFT.

In recent years, the significant success of deep learning (DL) in computer vision has contributed to its continuous development in the field of hyperspectral image (HSI) anomaly detection (AD). However, in practical applications, HSI-AD based on DL faces many challenges due to the inability to effectively acquire training samples and predict the types of anomaly targets. This makes it a challenging task, especially for AD in complex scenes. In this article, we propose an unsupervised DL framework for HSI-AD based on the multivariate probability distribution autoencoder (MPDA). First, to explore the distribution characteristics of high-dimensional data, we use the probability density histogram to statistically distribute the frequencies of each interval adaptively, dividing the HSI and obtaining an AD-guided image through the designed grid structure. Second, we propose an unsupervised multilayer autoencoder network based on energy-weighted skip connections. By coupling the detection-guided image in the network, we achieve reverse-guided module reconstruction, weakening the feature representation of anomalous in the reconstructed information and enhancing the separability of targets. Finally, we model the reconstructed error images using the multivariate skewed t-distribution based on data distribution characteristics to obtain the final AD map. Through the comparative experiments with other innovative AD algorithms on authentic HSI datasets captured in five different scenarios, the proposed algorithm demonstrates strong generalization and detection capabilities. The source code of the MPDA will be public athttps://github.com/muzhenhuam/MPDA/tree/master.

The hyper-sharpening technique fuses a high-resolution auxiliary image with a low spatial resolution hyperspectral (HS) image, aiming to enhance the spatial details of the HS image while preserving its spectral integrity. Although it extends from multispectral (MS) sharpening, the unique properties of HS data and the diversity of auxiliary images bring challenges to the application of MS sharpening methods. In this paper, a subspace dynamic combined sparse hyper-sharpening method for diverse auxiliary images is proposed. First, based on the dynamic total variation of MS sharpening, effective data reduction, and spectral coordination are realized by seeking reasonable subspace transformation paths and adopting band-matching processing for auxiliary images. Secondly, by associating the derivation of the relevant a priori with the residual representation, an idea of generalized “subspace + dynamic" regularization term is proposed. On this basis, a combination of regularization terms corresponding to dynamic gradient domain sparsity and dynamic nonlocal transform domain sparsity in subspace is explored. Finally, the alternating direction multiplier method and the improved closed-form solution are used for optimization. The general effectiveness of the proposed method is verified in three types of experiments (HS-PAN, HS-MS, HS-HS) on four public datasets. In the HS-HS class of experiments, where the inter-source correlation is low, the proposed method can overcome the spectral degradation assumption failure problem, with a PSNR improvement of 5.14% to 131.79% and an ERGAS improvement of 21.87% to 99.39% compared to the other nine methods. The code is at https://github.com/JZ-Tao/SDCS.

With the maturity and popularization of high-performance sensor technology, it is now possible to acquire huge amounts of very high-resolution (VHR) remote sensing images. The change detection (CD) for VHR images is currently receiving special attention for remote sensing earth observation applications, however, as a hot research field, it needs to be studied in depth to improve the detection accuracy of fine changes. To this end, a full-scale gated message passing framework (GaMPF) based on collaborative estimation for VHR remote sensing image change detection is proposed in this paper. On one hand, the key embedding representation is generated for each feature map by means of the collaborative estimation (CE) strategy; On the other hand, grounded in timing analysis, bitemporal features are sent selectively on dual paths according to the full-scale gated (FsG) mechanism. Specifically, this framework consists of the following four components: 1) Taking shared-weights Siamese network as an encoder to extract multi-scale features; 2) Generate a set of shared compact bases under the CE strategy and infer the key embedding representations on the basis of the shared bases for feature maps at the same level, considering the representations as the gated switches; 3) FsG mechanism is used as the mode of message passing between bitemporal images, which guides the information can be transmitted simultaneously on both within-and cross-temporal paths. 4) Creating a stepwise dense fusion module (DFM) as a decoder for predicting the change map. Experimental results show that the GaMPF proposed in this paper outperforms existing SOTA methods, and is particularly good at detecting edges and small objects. The source code will be released at https://github.com/zxylnnu/GaMPF.

Current remote sensing hardware technology is not yet able to acquire multiband remote sensing images with both high spatial and spectral resolution. As an important tool to compensate for the lack of spatial information acquisition of multispectral (MS) images, pan-sharpening has been an important and continuously active research area in remote sensing image processing. Although many methods have emerged, the problem of how to obtain high spatial resolution while effectively maintaining the spectral information of MS images has not been well solved. Many aspects still need further research. In this article, we first investigate the essential properties and rationality of two common framework types in the multiresolution analysis (MRA) sharpening method of pan-sharpening from the source perspective—the identical-resolution framework (IRF) derived from the generalized fusion application and the different-resolution framework (DRF) exclusive to the sharpening application, and show that the core difference between the two frameworks lies in the different ideas of utilizing the multiscale transformation, i.e., they tend to expand the scale space and model the spatially blurred degradation relationship between the sources, respectively. Both of them have their own advantages and disadvantages in handling detailed information, and neither of them can effectively deal with the “detail exclusivity” problem. Based on this, the idea of “entropy level matching” (ELM) of pan-sharpening is presented, and a comprehensive framework that can combine the advantages of the two types of frameworks is constructed, namely, the multiscale ELM framework. Furthermore, as an application of this framework, we propose a sharpening method shearlet transform-based entropy matching (STEM) built on the nonsubsampled shearlet as a multiscale transformation method. According to the difference in detail injection mode in it, it can be further divided into two sharpening methods based on additive mode and substitutive mode. The comparison experiments with 11 popular methods show that the proposed two sharpening methods can effectively improve the spatial resolution of MS images while keeping the spectral information well, and the comprehensive performance advantage is obvious. The source code of the proposed method can be downloaded fromhttps://github.com/JZ-Tao/STEM/.

The application of hyperspectral image (HSI) is more and more extensive, but the lower spatial resolution seriously affects its application effect. Using low-resolution hyperspectral image (LR-HSI) and high-resolution multispectral image (MSI) fusion technology to achieve super-resolution reconstruction of HSI has become a mainstream method. However, most of the existing fusion methods do not make full use of the large-scale range of remote sensing images, and neglect the preservation of spatial-spectral information in the fusion process. Considering that the spectral information in fused high-resolution hyperspectral image (HR-HSI) mainly depends on HSI, and the spatial information mainly depends on MSI, this paper proposes a full-scale linked Unet with spatial-spectral joint perceptual attention for hyperspectral and multispectral image fusion (FSL-Unet). The FSL-Unet consists of two modules, the first is spatial-spectral attention extraction module (SSAE), which is used to calculate the spectral attention of LR-HSI and the spatial attention of HR-MSI at different scales. The second is the full-scale link U-shaped fusion module (FLUF), which adopts a multi-level feature extraction strategy, using denser full-scale skip connections to explore feature information in a finer-grained range, enabling flexible combination of multi-scale and multi-path features. At the same time, we propose spatial-spectral joint peceptual attention (SSJPA) on the encoder side of FLUF. SSJPA can make full use of the attention maps computed by the SSAE, and then effectively embed spatial and spectral information into the fused image, enabling uninterrupted information transfer and aggregation. To demonstrate the effectiveness of FSL-Unet, we selected five public hyperspectral datasets for experiments. Compared with other eight state-of-the-art fusion methods, the experimental results show that the FSL-Unet achieves competitive results. The source code for FSL-Unet can be downloaded from https://github.com/wxy11-27/FSL-Unet.

The high spectral resolution of hyperspectral (HS) images provides the possibility of omnidirectional feature identification of objects. However, the high-dimensional features and the high redundancy information properties make data processing and the application of HS images extremely challenging. Thus, effectively expressing and correlating the intrinsic correlations of HS images by establishing a statistical model is of great significance. This article proposes a nonsubsampled shearlet transform hidden Markov forest (NSST-HMF) model. This new approach has three key characteristics: 1) the statistical properties of the NSST coefficients are studied in the spatial and spectral directions, respectively, and the “clustering” and “aggregation” properties are observed in both directions; 2) the HMF structure is proposed to depict the multidimensional collaborative correlation of the HS image NSST coefficients, and the proposed method considers the multidirectional transfer relationships among the Markov structure of HS images NSST coefficient for the first time, which significantly improves the prediction ability of the model; and 3) a novel HS-pansharpening approach based on the NSST-HMF model and amplitude modulation of large state probability in the high-frequency subband direction region is proposed. Experimental results show that our method can efficiently improve the spatial resolution of HS images while simultaneously preserving their spectral features. The HMF structure is first proposed in this article, which provides a way to depict the collaborative correlation of multichannel images.

Previous video summarization studies focused on monocular videos, and the results would not be good if they were applied to multi-view videos directly, due to problems such as the redundancy in multiple views. In this paper, we present a method for summarizing multi-view videos. We construct a spatio-temporal shot graph and formulate the summarization problem as a graph labeling task. The spatio-temporal shot graph is derived from a hypergraph, which encodes the correlations with different attributes among multi-view video shots in hyperedges. We then partition the shot graph and identify clusters of event-centered shots with similar contents via random walks. The summarization result is generated through solving a multi-objective optimization problem based on shot importance evaluated using a Gaussian entropy fusion scheme. Different summarization objectives, such as minimum summary length and maximum information coverage, can be accomplished in the framework. Moreover, multi-level summarization can be achieved easily by configuring the optimization parameters. We also propose the multi-view storyboard and event board for presenting multi-view summaries. The storyboard naturally reflects correlations among multi-view summarized shots that describe the same important event. The event-board serially assembles event-centered multi-view shots in temporal order. Single video summary which facilitates quick browsing of the summarized multi-view video can be easily generated based on the event board representation.

Summary form only given. Shift-variance and computational complexity are bottleneck of existing wavelet-based motion estimation (ME). Moreover, to the best of our knowledge, few works have been reported on wavelet-domain ME of video objects (VOs). In this paper, we present an efficient wavelet-domain approach to ME of arbitrarily shaped VOs.