Semi-supervised domain adaptation aims to transfer knowledge from a labeled source domain to a scarcely labeled target domain, despite distribution shifts. The challenge becomes greater when source and target data differ in acquisition modality, as in remote sensing where variations in sensor type (e.g., optical vs. radar), spectral properties (e.g., RGB vs. multispectral), or spatial resolution are common. This challenging scenario, known as Semi-Supervised Heterogeneous Domain Adaptation (SSHDA), requires learning across modalities with limited target labels. In this work, we propose SAHARA (Semi-supervised Adaptation in Heterogeneous domains via conditional Adversarial Representation disentanglement and Adaptive pseudo-labeling), a new method for SSHDA that combines conditional adversarial feature adaptation with dynamic pseudo-labeling to learn domain-invariant features and handle extremely scarce target annotations. Experiments on two heterogeneous remote sensing benchmarks for scene classification, conducted with both convolutional and transformer-based backbones, demonstrate that SAHARA consistently outperforms existing SSHDA and semi-supervised methods. The code is available at https: //TO-BE-DISCLOSED-UPON-ACCEPTANCE.

The goal of a deep learning-based general image fusion method is to solve multiple image fusion tasks with a single model, thereby facilitating the deployment of models in practical applications. However, existing methods fail to provide an efficient and comprehensive solution from both model training and network design perspectives. Regarding model training, current approaches cannot effectively leverage complementary information across different tasks. In terms of network design, they rely on experience-based network designs. To address these issues, we propose a comprehensive framework for general image fusion using the newly proposed gradient transfer learning and fusion rule unfolding. To leverage complementary information across different tasks during training, we propose a sequential gradient-transfer framework based on the idea that different image fusion tasks often exhibit complementary structural details and that image gradients effectively capture these details. To move beyond heuristic-based network design, we evolved a fundamental image fusion rule and integrated it into a deep equilibrium model, resulting in a more efficient and versatile image fusion network capable of uniformly handling various fusion tasks. Considering three different image fusion tasks, i.e., multi-focus image fusion, multi-exposure image fusion, and infrared and visible image fusion, our method not only produces images with richer structural information but also achieves highly competitive objective metrics. Furthermore, the results of generalization experiments on previously unseen image fusion tasks, i.e., medical image fusion, demonstrate that our method significantly outperforms competing approaches.

The problem of robust matrix completion-the recovery of a low-rank matrix and a sparse matrix from a sampling of their superposition-has been addressed extensively in prior literature. Yet, much of this work has focused exclusively on the case in which the matrix sampling is done at random, as this scenario is amenable to theoretical analysis. In contrast, sampling with an arbitrary deterministic pattern is often more accommodating to hardware implementation; consequently, the problem of robust matrix completion under deterministic sampling is considered. To this end, a restricted approximate isometry property is proposed and used, along with a modified golfing scheme and a slightly strengthened incoherence condition, to prove that the latent low-rank and sparse matrices are uniquely recoverable via convex optimization with asymptotically high probability, providing the first exact-recovery theory for robust matrix completion with arbitrary deterministic sampling. A corresponding convex-optimization algorithm, driven by a traditional nuclear norm, is developed and then subsequently generalized by substituting a convolutional nuclear norm in order to cover a broader range of application scenarios. Empirical experiments on synthetic data verify the proposed theory while a battery of results on real-world images demonstrate the practical efficacy of the generalized algorithm for robust matrix recovery.

Recently, tensor network (TN) decompositions have gained prominence in computer vision and contributed promising results to tensor recovery for their capability of compactly and efficiently representing high-order tensors. However, current TN topologies are rather being developed towards more intricate structures to pursue incremental improvements, resulting in a drastically increased number of TN ranks, which requires laborious hyper-parameter selection, especially for higher-order cases. In this paper, we propose a novel TN decomposition, dubbed tensor wheel (TW) decomposition, in which a high-order tensor is represented by a set of latent factors mapped into a specific wheel topology. Such a decomposition is constructed starting from analyzing the graph structure, aiming to more accurately characterize the complex interactions inside objectives while maintaining a lower hyper-parameter scale, theoretically alleviating the above deficiencies. The comprehensive analysis of the mathematical properties fully demonstrates that TW decomposition can be more potential in representation capabilities and more flexible in controlling both parameter storage and computational costs. To compute the TW-format decomposition, the sequential singular value decomposition (SVD)-based and the alternating least squares (ALS)-based learning algorithms are developed. Furthermore, to investigate the validity of TW decomposition, we provide its one numerical application, i.e., tensor completion (TC), yet develop an efficient proximal alternating minimization-based solving algorithm with guaranteed convergence. Experimental results on both synthetic and real-world data reveal that TW decomposition significantly outperforms other state-of-the-art tensor decompositions for incomplete-tensor inference, especially under solely few observations, thus substantiating the superiority and reliability of TW decomposition.

Multimodal image registration aims to spatially align images from different modalities at the pixel level. However, due to the nonlinear relationship of radiation intensities caused by different imaging modalities, achieving high accuracy in multimodal image registration presents a significant challenge. Additionally, the presence of both global transformations (i.e., large-scale rigid affine transformations) and local distortions (i.e., small-scale nonrigid deformations) between paired images further complicates the registration process. This article addressed the challenge resulting from modality differences through modality distillation. Specifically, a teacher (i.e., a homomodal image registration model) is trained to guide the student (i.e., a multimodal image registration model). Besides, this article simultaneously aligned large-scale rigid and small-scale nonrigid deformations by predicting deformation flow from both global and local features, thereby achieving high-precision registration. Furthermore, this proposed method incorporated a deformation mask during training to mitigate the negative impact of black edges in the obtained registration results on model performance. Experimental results demonstrate that the proposed method delivers state-of-the-art registration accuracy across various multimodal datasets, with ablation studies confirming the effectiveness of each component. The codes will be available at https://github.com/2351056918/Multimodality-Image-Registration-with-Modailty-Distillation.

SAR colorization aims to enrich gray-scale SAR images with color while ensuring the preservation of original radiometric and spatial details. However, researchers often limit themselves to using only the red, green, and blue bands of a multispectral image as the source of color information, coupled with a single-polarization channel from the SAR image. This approach neglects the intrinsic characteristics of remote sensing data and thus fails to fully leverage available information. To overcome this limitation, this research attempts to explore inclusion of all available bands from multispectral images along with dual-polarization channels from SAR imagery in the colorization process. Furthermore, we present a new colorization method called improved conditional generative adversarial network for SAR colorization (IcGAN4ColSAR). This method tries to include the spectral angle mapper index within its loss function. Sufficient experiments show that our explorations in the number of data channels and the loss function are helpful in improving the colorization performance of the SAR image.

Image fusion aims to merge image pairs collected by different sensors over the same scene, preserving their distinct features. Recent works have often focused on designing various image fusion losses, developing different network architectures, and leveraging downstream tasks (e.g., object detection) for image fusion. However, a few studies have explored how language and semantic masks can serve as guidance to aid image fusion. In this paper, we investigate how the combination of language and masks can guide image fusion tasks, discarding the previously complex frameworks, which rely on downstream tasks, GAN-based cycle training, diffusion models, or deep image priors. Additionally, we exploit a recurrent neural network-like architecture to build a lightweight network that avoids the quadratic-cost of traditional attention mechanisms. To adapt the receptance weighted key value (RWKV) model to an image modality, we modify it into a bidirectional version using an efficient scanning strategy (ESS). To guide image fusion by language and mask features, we introduce a multi-modal fusion module (MFM) to facilitate information exchange. Comprehensive experiments show that the proposed framework achieved state-of-the-art results in various image fusion tasks (i.e., visible-infrared image fusion, multi-focus image fusion, multi-exposure image fusion, medical image fusion, hyperspectral and multispectral image fusion, and pansharpening).

Multi-source image fusion combines the information coming from multiple images into one data, thus improving imaging quality. This topic has aroused great interest in the community. How to integrate information from different sources is still a big challenge, although the existing self-attention based transformer methods can capture spatial and channel similarities. In this paper, we first discuss the mathematical concepts behind the proposed generalized self-attention mechanism, where the existing self-attentions are considered basic forms. The proposed mechanism employs multilinear algebra to drive the development of a novel fully-connected self-attention (FCSA) method to fully exploit local and non-local domain-specific correlations among multi-source images. Moreover, we propose a multi-source image representation embedding it into the FCSA framework as a non-local prior within an optimization problem. Some different fusion problems are unfolded into the proposed fully-connected transformer fusion network (FC-Former). More specifically, the concept of generalized self-attention can promote the potential development of self-attention. Hence, the FC-Former can be viewed as a network model unifying different fusion tasks. Compared with state-of-the-art methods, the proposed FC-Former method exhibits robust and superior performance, showing its capability of faithfully preserving information.

Spectral and spatial fidelity remains a longstanding challenge in the field of pansharpening, which aims to generate high-resolution multispectral (HRMS) images by integrating high-resolution panchromatic (PAN) images with low-resolution multispectral (LRMS) images. This study proposes a high-fidelity pansharpening network that utilizes bidirectional trigeminal pyramid decoding of features encoded by a CNN-Transformer architecture. Specifically, local and global features at multiple scales are initially extracted using a CNN-Transformer encoder to facilitate multi-scale feature fusion. Subsequently, we design a decoder based on bidirectional trigeminal pyramids to achieve a high-fidelity fusion output. One reverse decoding pyramid decodes the fused features of LRMS and PAN images from the encoder. One spectral feature pyramid is employed to enhance the spectral information of the reverse decoding pyramid, while the last spatial feature pyramid is utilized to enrich the spatial information, thereby improving the overall spectral and spatial fidelity of the fused output. Furthermore, content-guided attention (CGA) is incorporated to adaptively integrate the spectral and spatial feature pyramids into the reverse decoding pyramid. Extensive experiments demonstrate that our network surpasses the comparative state-of-the-art (SOTA) methods in both qualitative and quantitative evaluations. The code is available at https://github.com/songvvvv/pansharpening.

Nowadays, various graph convolutional networks (GCNs) to process graph-structured data have been proposed for hyperspectral image (HSI) classification. Nevertheless, most GCN-based HSI classification methods emphasize graph node feature aggregation instead of graph pooling, resulting in them being shallow networks and unable to extract deep discriminative features. Besides, to obtain the new graph after the pooling layer, current graph pooling methods used for HSI classification just consider node feature information to select important nodes and directly discard unselected nodes, which could be a subjective process and may cause information loss. To solve this issue, we propose a novel graph U-Net with topology-feature awareness pooling (the so-called TFAP graph U-Net) for HSI classification considering a deep network to extract compelling features and automatically selecting nodes beneficial to classification. More specifically, to establish a more precise pooled graph, the graph’s topology structure and node feature information are taken into account, making the node selection process more convincing and objective. Furthermore, to allow that graph nodes preserve more useful graph information, our method aggregates node features from neighboring nodes that may not be selected, which can alleviate the loss of information during the pooling process. Moreover, a cross-attention module (CAM) is used to filter out irrelevant or noisy features. Finally, we evaluate the proposed method on three public HSI datasets, i.e., Indian Pines, University of Pavia (PaviaU), and University of Houston. The experimental results demonstrate the superiority of the proposed approach compared with other state-of-the-art methods.

The semantic segmentation of remote sensing images is crucial for Earth observation. The semi-supervised semantic segmentation method can effectively reduce the dependence of the training process on labeled data. Among them, the semi-supervised semantic segmentation method based on the teacher-student paradigm is currently one of the most mainstream methods. However, the issue of weight coupling has constrained further performance improvements. This article proposes an alternating guidance method that combines cross-view learning to improve the teacher-student paradigm and enhance the semantic segmentation performance of remote sensing images. The student model is designed by using two decoders with the same architecture but independently updated parameters. Two decoders process the input obtained after image and feature level perturbations. This allows the student model to generate unique feature representations and enhances its learning capability. The teacher model uses two decoders to construct an alternating supervision mechanism. The two decoders of the teacher model take turns outputting pseudo-labels to guide the training process of the student model. This alternating supervision strategy can provide richer supervision signals for student model while helping to alleviate weight coupling between teacher and student. The experiments on two remote sensing image datasets show that compared with the state-of-the-art (SOTA) semi-supervised semantic segmentation methods, the method proposed demonstrates excellent competitiveness.

Hyperspectral pansharpening has received much attention in recent years due to technological and methodological advances that open the door to new application scenarios. However, research on this topic is only now gaining momentum. The most popular methods are still borrowed from the more mature field of multispectral pansharpening and often overlook the unique challenges posed by hyperspectral data fusion, such asi)the very large number of bands,ii)the overwhelming noise in selected spectral ranges,iii)the significant spectral mismatch between panchromatic and hyperspectral components,iv)a typically high resolution ratio. Imprecise data modeling especially affects spectral fidelity. Even state-of-the-art methods perform well in certain spectral ranges and much worse in others, failing to ensure consistent quality across all bands, with the risk of generating unreliable results. Here, we propose a hyperspectral pansharpening method that explicitly addresses this problem and ensures uniform spectral quality. To this end, a single lightweight neural network is used, with weights that adapt on the fly to each band. During fine-tuning, the spatial loss is turned on and off to ensure a fast convergence of the spectral loss to the desired level, according to a hysteresis-like dynamic. Furthermore, the spatial loss itself is appropriately redefined to account for nonlinear dependencies between panchromatic and spectral bands. Overall, the proposed method is fully unsupervised, with no prior training on external data, flexible, and low-complexity. Experiments on a recently published benchmarking toolbox show that it ensures excellent sharpening quality, competitive with the state-of-the-art, consistently across all bands. The software code and the full set of results are shared online on https://github.com/giu-guarino/rho-PNN.

Monitoring particulate matter (PM) is of critical importance due to its significant impact on human health. Ground stations provide highly accurate measurements of various pollutants on a local scale. However, the limited distribution of these stations makes achieving global coverage challenging. To address this limitation, satellite imagery serves as a valuable resource, offering wide-area PM estimates in near real-time through abundant data and frequent revisit intervals. In contrast to other studies, this work introduces deep learning (DL) models to estimate ground-level PM concentration maps over Europe. These models rely exclusively on radiance data from the Sentinel-5P satellite, forgoing auxiliary information, such as meteorological data, which are commonly incorporated in similar studies. The proposed approach has demonstrated both robust estimation accuracy and effective generalization capabilities. Furthermore, the estimated PM concentration maps have been validated against ground-based measurements, showing superior performance with respect to widely used models and datasets that consider meteorological inputs. The dataset and the code are available here:https://github.com/antoniomazza88/PMUnet.

In recent years, tensor decomposition-based approaches forhyperspectral anomaly detection(HAD) have gained significant attention in the field of remote sensing. However, existing methods often fail to flexibly and effectively extract both the global correlations and local smoothness of the background components inhyperspectral images(HSIs). To mitigate this critical issue, we put forward a novel HAD method named HAD-EUNTRFR, which incorporates an enhanced unified nonconvex tensor ring (TR) factors regularization. In the HAD-EUNTRFR framework, the raw HSIs are first decomposed into background and anomaly components using the idea of tensor robust principal component analysis. The TR decomposition is then employed to capture the spatial-spectral correlations within the background component. Additionally, we introduce a unified and efficient nonconvex regularizer, induced bytensor singular value decomposition(T-SVD), to simultaneously encode the low-rankness and sparsity of the 3-D gradient TR factors into a unique concise form. The above characterization scheme enables the interpretable gradient TR factors to inherit the low-rankness and smoothness of the original background. To further enhance anomaly detection, we design a generalized nonconvex regularization term to exploit the group sparsity of the anomaly component. Based upon the above, we ultimately propose a scalable and reliable nonconvex HAD model. To solve the resulting doubly nonconvex model, we develop a highly efficient optimization algorithm based on thealternating direction method of multipliers(ADMM) framework. Theoretical results on convergence analysis for the proposed algorithm are derived. Experimental results on several benchmark datasets demonstrate that our proposed method outperforms existingstate-of-the-art(SOTA) approaches in terms of detection accuracy.

Lightweight remote sensing image super-resolution (RSISR) methods aim to reconstruct remote sensing images (RSIs) while reducing computational complexity. Previous lightweight model development has primarily focused on the design of convolutional neural networks (CNNs). While CNNs excel at capturing local features, they are limited in establishing long-range dependencies. Mamba, as a model for long-range modeling, has linear computational complexity, making it a viable option for lightweight models. Based on these considerations, this paper proposes an efficient mamba-attention network (EMAN) that can efficiently capture the intricate details and broader semantic information in RSIs. Specifically, we designed a multi-scale detail extraction unit (MDEU) and a multi-dimensional mamba-attention (MDMA). In MDEU, we introduced a multi-scale mechanism and local variance to focus on structural information in RSIs. In MDMA, we integrated spatial expansion and an atrous-based selective scan mechanism to design an efficient scanning method. This method ensures the lightweight nature of the model while establishing global correlations. Additionally, MDMA establishes inter-channel correlations to enhance information exchange. We conducted a comprehensive evaluation of the proposed method on two remote sensing datasets and five benchmark super-resolution (SR) datasets. Extensive experiments demonstrate that our method can achieve superior performance while maintaining a model complexity similar to other lightweight models.

Underwater imaging is essential for marine remote sensing tasks, such as environmental monitoring, resource exploration, and autonomous navigation. However, images captured in underwater environments often suffer from complex degradations, including wavelength-dependent color distortion, contrast attenuation, and structural detail loss. To address these challenges, we propose a Cross-Scale Style-Guided Network (CSG-Net) for robust underwater image enhancement. CSG-Net employs a dual-stage collaborative framework that decouples global degradation modeling from local detail refinement. In the first stage, a Style Extraction Network (SE-Net) extracts multi-scale degradation-aware style priors that implicitly encode large-scale physical degradation patterns, such as red-channel attenuation and spectral imbalance. In the second stage, a Style-Guided Enhancement Network (SG-Net) leverages these style features to guide spatially adaptive enhancement, enabling consistent color correction and fine-grained detail recovery. To alleviate semantic degradation during scale transitions, CSG-Net introduces the proposed multi-resolution feature-preserving cross-scale interaction (MFPCSI) module, which enhances the preservation and integration of hierarchical features. Combined with the Multi-Stream Information Fusion (MSIF) module, this design enables the effective fusion of semantic and structural information across spatial scales. The proposed components enable the preservation of fine-grained details while adaptively integrating semantic and structural cues across multiple scales. Comprehensive experiments conducted on diverse and challenging underwater image datasets demonstrate that CSG-Net consistently surpasses state-of-the-art approaches in terms of PSNR, SSIM, and UIQM. Furthermore, the model exhibits strong cross-domain generalization and delivers high-fidelity visual results, underscoring its suitability for deployment in practical vision systems operating under complex, real-world environments.

The spatial resolution of TROPOMI, the sensor mounted on board of the satellite Sentinel-5P to monitor air quality, is much higher than its predecessors. Yet, the high variability of pollutants limits the use of the resulting maps in practical applications. Super-resolution approaches can improve the precision of estimates, but their use is heavily reliant on the ability to precisely tune the parameters of the algorithms. For this reason, the employment of a specific image acquisition model is essential for both learning-based and traditional methods. This contribution leverages real full-scale images for validation of a recently proposed model for the degradation introduced by the TROPOMI instrument, which is applied to both classical and learning-based techniques. The model's validity can be evaluated by analysing the quantitative data and visually inspecting the images that have been generated. This contribution proves that the degradation model is an essential basis for the development of novel approaches as well as for the application of all already available techniques.

In this paper, a representative set of state-of-the-art methods for hyperspectral pansharpening, comprising both model- and deep learning-based ones, are reviewed and compared on four datasets from the PRISMA mission. The experimental analysis has been carried out using the most credited pansharpening quality indexes, complemented by a subjective visual inspection of sample results. The obtained outcomes have provided us a preview of the strengths and weaknesses of the latest solutions to the problem at hand, paving the way for future research lines from both the methodological and quality assessment perspectives.

This work proposes a hybrid approach to address the pansharpening of hyperspectral images, which mixes the use of a recently proposed CNN-based solution and a classical solution such as the Gram Schmidt Adaptive method (GSA). The hyperspectral datacube is split in two sets of bands, those falling in the visible range and the remaining ones. The first set is pansharpened using the GSA approach which has proven to grant very high quality results in this range. The remaining bands, whose relationship with the panchromatic band is much weaker, undergo a fusion process based on a recently proposed hyperspectral pansharpening method known as Rolling hyperspectral Pansharpening Neural Network (R-PNN). By doing so, we are able to take the best features from both solutions, getting higher quality results compared to the marginal use of any of the two methods.

This work deals with the estimation of the tropospheric vertical column density of nitrogen dioxide from Sentinel-5P radiance data using convolutional neural networks. The current processing chain to retrieve this information from Sentinel-5P data requires a complex, computationally demanding, physical modeling that involves the use of additional side information such as meteorological variables, which are not always available. Therefore, in this proof-of-concept study, we explored the feasibility of an estimation exclusively using radiance data from Sentinel-5P, leveraging on the powerful representational capacity of deep neural networks. Preliminary results are very promising encouraging further investigation.

This paper proposes a bi-level tensor decomposition (BLTD), properly exploiting the characterization advantages of tensor subspace representation (TSR) and tensor ring decomposition (TRD). More specifically, the first level is related to the decomposition of a third-order tensor into its TSR using the tensor-tensor product (t-product), and the second level performs TRD on the coefficient tensor obtained by TSR. Leveraging the proposed BLTD, we design a bi-level tensor nuclear norm (BLTNN)-based model for hyperspectral images (HSIs) denoising. To solve the model, we develop an efficient alternating direction method of multipliers (ADMM)-based algorithm. Experimental results demonstrate the superior performance of our method compared to existing methods.

Sentinel-5P is a valuable resource for academics and policymakers. The ability of the satellite’s equipment to span the electromagnetic spectrum from ultraviolet (UV) to short-wave infrared (SWIR) frequencies is vital in determining the distribution of important gaseous pollutants on a global scale, a significant turning point for air quality monitoring. In technical terms, Sentinel-5P provides an excellent balance between spatial and spectral resolutions; however, physical limitations keep hindering the quality of its products. S5Net is the first deep-learning-based (DL-based) approach designed to super-resolve Sentinel-5P radiance images. Despite its simplicity, this neural network has showed excellent performance when applied to monochromatic images, particularly when compared to more complex deep neural networks. Yet, this groundbreaking study has a significant limitation: the computational inefficiency of the fine-tuning employed, which must be adequately extended to numerous channels. We hence propose a novel dynamic multidirectional cascade fine-tuning procedure, whose routine is fully governed by the correlation between consecutive spectral channels. Our study is accordingly successful in striking a remarkable balance between spectral coherence and spatial resolution improvement, as well as substantially optimizing computing efficiency. The code is available athttps://github.com/alcarbone/S5P_SISR_Toolbox.

Sentinel-5P provides excellent spatial information, but its resolution is insufficient to characterise the complex distribution of air contaminants within limited areas. As physical constraints prevent significant advances beyond its nominal resolution, employing processing techniques like single-image super-resolution can notably contribute to both research and air quality monitoring applications. This study presents the very first use of such methodologies on Sentinel-5P data. We demonstrate that superior results may be obtained if the degrading filter used to simulate pairs of low- and high-resolution images is tailored to the acquisition technology at hand, an issue frequently ignored in the scientific literature on the subject. Because of this, as well as the fact that these data have never been deployed in any previous studies, most of the work theoretical contribution is the estimation of the degradation model of TROPOMI, the sensor mounted on Sentinel-5P. Leveraging this model—which is essential for applications involving super-resolution—we additionally improve a well-known deconvolution-based strategy and present a brand-new neural network that outperforms both traditional super-resolution techniques and well-established neural networks in the field. The findings of this study, that are supported by experimental tests on real Sentinel-5P radiance images, using both full-scale and reduced-scale protocols, offer a baseline for enhancing algorithms that are driven by the understanding of the imaging model and provide an efficient way of evaluating innovative approaches on all the available images. The code is available at https://github.com/alcarbone/S5P_SISR_Toolbox.

Hyperspectral pansharpening is receiving a growing interest since the last few years as testified by a large number of research papers and challenges. It consists in a pixel-level fusion between a lower-resolution hyperspectral datacube and a higher-resolution single-band image, the panchromatic image, with the goal of providing a hyperspectral datacube at panchromatic resolution. Thanks to their powerful representational capabilities, deep learning models have succeeded to provide unprecedented results on many general purpose image processing tasks. However, when moving to domain specific problems, as in this case, the advantages with respect to traditional model-based approaches are much lesser clear-cut due to several contextual reasons. Scarcity of training data, lack of ground-truth, data shape variability, are some such factors that limit the generalization capacity of the state-of-the-art deep learning networks for hyperspectral pansharpening. To cope with these limitations, in this work we propose a new deep learning method which inherits a simple single-band unsupervised pansharpening model nested in a sequential band-wise adaptive scheme, where each band is pansharpened refining the model tuned on the preceding one. By doing so, a simple model is propagated along the wavelength dimension, adaptively and flexibly, with no need to have a fixed number of spectral bands, and, with no need to dispose of large, expensive and labeled training datasets. The proposed method achieves very good results on our datasets, outperforming both traditional and deep learning reference methods. The implementation of the proposed method can be found on https://github.com/giu-guarino/R-PNN.

Hyperspectral image super-resolution (HSI SR) aims to combine the detailed spectral information of hyperspectral images with the spatial resolution of multispectral images, thus enhancing the ability to extract valuable insights across various applications. Recently, the tensor singular value decomposition (t-SVD) has emerged as a powerful tool and has been introduced into the HSI SR field for exploring low-rank prior information. For t-SVD, the domain transform is crucial to acquiring more low-rank data characteristics. Nevertheless, previous efforts on domain transform have only involved the single transformed domain (i.e., single domain), while ignoring the potential pursuing the lower rankness in multiple successional transformed domains, termed cross-domain (CD). In this article, we propose a novel CD-based t-SVD and define the corresponding tensor CD rank based on a pivotal observation, i.e., the low-rank behavior of HSI in CD is more significant than that in single domain. More specifically, we first define a successional linear transform (SLT) to establish the CD concept, then develop a novel CD-based t-SVD and tensor CD rank, and theoretically deduce a new tensor CD-nuclear norm as the convex approximation of CD rank. Equipped with such a CD rank, we thus formulate a CD-rank-constrained minimization model for the HSI SR task, called CroDoSR, which is effectively solved by the alternating direction method of multipliers (ADMMs). Comprehensive experiments on several widely used datasets evidently demonstrate the superiority of the proposed CroDoSR method.

Segment anything model (SAM) has been widely applied to various downstream tasks for its excellent performance and generalization capability. However, SAM exhibits three limitations related to remote sensing semantic segmentation task: 1) the image encoders excessively lose high-frequency information, such as object boundaries and textures, resulting in rough segmentation masks; 2) due to being trained on natural images, SAM faces difficulty in accurately recognizing objects with large-scale variations and uneven distribution in remote sensing images; 3) the output tokens used for mask prediction are trained on natural images and not applicable to remote sensing image segmentation. In this paper, we explore an efficient paradigm for applying SAM to the semantic segmentation of remote sensing images. Furthermore, we propose MeSAM, a new SAM fine-tuning method more suitable for remote sensing images to adapt it to semantic segmentation tasks. Our method first introduces an inception mixer into the image encoder to effectively preserve high-frequency features. Secondly, by designing a mask decoder with remote-sensing correction and incorporating multiscale connections, we make up the difference in SAM from natural images to remote sensing images. Experimental results demonstrated that our method significantly improves the segmentation accuracy of SAM for remote sensing images, outperforming some state-of-the-art methods. The code will be available at https://github.com/Magic-lem/MeSAM.

Water-related optics images are often degraded by absorption and scattering effects. Current underwater image enhancement (UIE) methods improve image quality but neglect the constraints of underwater imaging environments. To address this issue, we propose a double-teacher knowledge distilling network (DTKD-Net), which uses a dynamic teaching strategy within a dual-teacher framework to enhance knowledge distillation (KD), improving the student network’s ability to capture complex underwater features. Specifically, DTKD-Net focuses on clear-to-clear and blurry-to-clear image learning to enhance underwater images. It aims to preserve details in clear images and restore blurred ones. The dual-teacher network uses an intermediate layer with the middle layer of the student network to compute feature differences for feature guidance. The network uses a dynamic strategy where a Teacher-Sub stops guidance when its output matches the student’s, which helps with contrastive learning and improves the network’s ability to handle complex underwater scenes. Extensive experiments and visual comparisons show that DTKD-Net reduces the model size, demonstrating superior efficiency and effectiveness in enhancing underwater images.

Convolutional neural networks (CNNs) have recently achieved outstanding performance for hyperspectral (HS) and multispectral (MS) image fusion. However, CNNs cannot explore the long-range dependence for HS and MS image fusion because of their local receptive fields. To overcome this limitation, a transformer is proposed to leverage the long-range dependence from the network inputs. Because of the ability of long-range modeling, the transformer overcomes the sole CNN on many tasks, whereas its use for HS and MS image fusion is still unexplored. In this article, we propose a spectral-spatial transformer (SST) to show the potentiality of transformers for HS and MS image fusion. We devise first two branches to extract spectral and spatial features in the HS and MS images by SST blocks, which can explore the spectral and spatial long-range dependence, respectively. Afterward, spectral and spatial features are fused feeding the result back to spectral and spatial branches for information interaction. Finally, the high-resolution (HR) HS image is reconstructed by dense links from all the fused features to make full use of them. The experimental analysis demonstrates the high performance of the proposed approach compared with some state-of-the-art (SOTA) methods.

This work proposes a simple yet effective method to adapt unsupervised convolutional neural networks for pansharpening of multispectral images to the problem of hyperspectral image pansharpening, i.e., the fusion of a single high-resolution panchromatic band with a low-resolution hyperspectral data cube. This is achieved by means of a PCA transformation which allows to compact the most of the HS image energy in a few bands, which are then suitably super-resolved using a pansharpening network designed for few spectral bands. Our experiments show very encouraging results which compare favorably against the state-of-the-art methods.

In this paper, we propose a novel dynamic fusion model (DFM) with joint variational and deep priors for the task of hyperspectral image super-resolution (HISR). The given model can benefit from both the advantages of traditional modeling and deep learning methods, thus achieving significant improvements based on existing deep pre-trained models. Specifically, the given model mainly contains two new designed terms, i.e., the weighted spatial fidelity (WSF) term and the deep fusion (DF) term. The WSF term focuses on the spatial recovery of the low-resolution hyperspectral image through the high-resolution multispectral image without the knowledge of the spectral response matrix, thus the proposed DFM can be viewed as a semi-blind model for HISR. Moreover, the DF term relied upon deep fusion with a designed adaptive weight matrix, which can effectively inject the deep priors into the traditional minimization model. Besides, the proposed DFM can be quickly and effectively solved using the alternating direction method of multipliers. Experimental results on widely used datasets demonstrate the superiority of our approach compared with state-of-the-art HISR methods.

This work proposes a simple yet effective method to adapt unsupervised convolutional neural networks from multispectral to hyperspectral pansharpening. Thus, it focuses on the fusion of a single high-resolution panchromatic band with a low-resolution hyperspectral data cube. This is achieved by means of a decorrelation transform, following the principal component analysis approach, which enables the compression of a significant portion of the hyperspectral image energy into a few bands. Afterwards, a suitably adapted pansharpening network designed for four spectral bands is used to super-resolve only the principal components. Experiments demonstrate high performance in both quantitative and qualitative evaluations, favorably comparing against state-of-the-art methods.

Hyperspectral image super-resolution (HISR) is about fusing a low-resolution hyperspectral image (LR-HSI) and a high-resolution multispectral image (HR-MSI) to generate a high-resolution hyperspectral image (HR-HSI). Recently, convolutional neural network (CNN)-based techniques have been extensively investigated for HISR yielding competitive outcomes. However, existing CNN-based methods often require a huge amount of network parameters leading to a heavy computational burden, thus, limiting the generalization ability. In this article, we fully consider the characteristic of the HISR, proposing a general CNN fusion framework with high-resolution guidance, called GuidedNet. This framework consists of two branches, including 1) the high-resolution guidance branch (HGB) that can decompose the high-resolution guidance image into several scales and 2) the feature reconstruction branch (FRB) that takes the low-resolution image and the multiscaled high-resolution guidance images from the HGB to reconstruct the high-resolution fused image. GuidedNet can effectively predict the high-resolution residual details that are added to the upsampled HSI to simultaneously improve spatial quality and preserve spectral information. The proposed framework is implemented using recursive and progressive strategies, which can promote high performance with a significant network parameter reduction, even ensuring network stability by supervising several intermediate outputs. Additionally, the proposed approach is also suitable for other resolution enhancement tasks, such as remote sensing pansharpening and single-image super-resolution (SISR). Extensive experiments on simulated and real datasets demonstrate that the proposed framework generates state-of-the-art outcomes for several applications (i.e., HISR, pansharpening, and SISR). Finally, an ablation study and more discussions assessing, for example, the network generalization, the low computational cost, and the fewer network parameters, are provided to the readers. The code link is: https://github.com/Evangelion09/GuidedNet.

Graph convolutional network (GCN) has recently received increasing attention in hyperspectral image (HSI) classification, benefiting from its superiority in conducting shape adaptive convolutions on arbitrary non-Euclidean structure data. However, the performance of GCN heavily depends on the quality of the initial graph. Conventional GCN-based methods only adopt spectral-spatial similarity to build the initial graph without extracting other contextual information from neighboring nodes. In addition, most GCN-based methods use shallow layers, which cannot extract deep discriminative features from HSIs under the limited number of training samples. To solve these issues, we propose a superpixel feature learning via offset graph U-Net for HSI classification, which can learn deep discriminative features from HSIs. Multiple strategies of measuring similarity among superpixels are utilized to build the initial graph, including spectral information, spatial information and context-aware information among nodes, making the initial graph more accurate. Furthermore, the graph U-Net structure, containing the graph pooling layer and the graph unpooling layer, is helpful in constructing deep GCN layers and learning multi-scale features, which can alleviate the oversmoothing problem. Moreover, an offset module is introduced to emphasize the local spectral-spatial information. Finally, we comprehensively evaluate the proposed method on three public data sets. The experimental results demonstrate the superiority of the proposed approach compared with other state-of-the-art methods.

Deep learning (DL) methods have achieved impressive performance for pansharpening in recent years. However, because of poor generalization, most DL methods achieve unsatisfactory performance for data acquired by sensors not considered during the training phase and decreased performance for samples at full resolution. To solve this issue, we propose a data augmentation framework for pansharpening neural networks. Specifically, we introduce first a random spatial degradation based on anisotropic Gaussian-shaped modulation transfer functions (MTFs) to increase the generalization with respect to different spatial models and sensors. Then, considering that various sensors have different ground sampling distances (GSDs), we randomly rescale the GSD of the training samples to improve the generalization with respect to spatial resolution. Thanks to this module, the generalization to tests from different sensors and samples at full resolution can easily be achieved. Experimental results demonstrate the effectiveness of the proposed approach with better performance when data for training are decoupled with the ones for testing and comparable performance when training and testing are coupled (i.e., data acquired by the same sensor are considered in the two phases). Besides, performance at full resolution for pansharpening neural networks is improved by the proposed approach. The proposed approach has been integrated into existing pansharpening neural networks showing satisfactory performance for widely used sensors, including, GaoFen-1, QuickBird, WorldView-2, WorldView-3, IKONOS, Spot-7, GeoEye, and PHR1A.

A Transformer has received a lot of attention in computer vision. Because of global self-attention, the computational complexity of Transformer is quadratic with the number of tokens, leading to limitations for practical applications. Hence, the computational complexity issue can be efficiently resolved by computing the self-attention in groups of smaller fixed-size windows. In this article, we propose a novel pyramid Shuffle-and-Reshuffle Transformer (PSRT) for the task of multispectral and hyperspectral image fusion (MHIF). Considering the strong correlation among different patches in remote sensing images and complementary information among patches with high similarity, we design Shuffle-and-Reshuffle (SaR) modules to consider the information interaction among global patches in an efficient manner. Besides, using pyramid structures based on window self-attention, the detail extraction is supported. Extensive experiments on four widely used benchmark datasets demonstrate the superiority of the proposed PSRT with a few parameters compared with several state-of-the-art approaches. The related code is available athttps://github.com/Deng-shangqi/PSRT.

Pansharpening refers to the fusion of a panchromatic (PAN) image with high spatial resolution and a multispectral (LRMS) image with low spatial resolution to obtain a high spatial resolution multispectral (HRMS) image, which is beneficial to visual display and geographic research. Recently, many deep learning (DL) methods have been proposed to address the pansharpening problem, but still a few examples of DL-based techniques are designed from the perspective of a better receptive field while the scale of features greatly varies among different ground objects. In this paper, we mainly focus on designing a cascadic multi-receptive learning module (CML-resblock) relying on the ResNet block, which can efficiently extract multi-scale features from both the PAN and LRMS images. Moreover, we propose a novel multiplication network preserving a physical significance, which uses deep neural networks (DNNs) to learn the coefficients of the pixel-wise restoration mapping and multiplies the up-sampled LRMS image with the learned coefficients to get the HRMS image. The two parts mentioned above constitute our cascadic multi-receptive learning network (CMLNet). Extensive experiments on both reduced-resolution and full-resolution images acquired by the WorldView-3, GaoFen-2, and QuickBird satellites show that the proposed approach outperforms state-of-the-art methods. Furthermore, additional experiments have been conducted to prove the generality of the CML-resblock and multiplication network. The code is available at: https://github.com/wajuda/CML.

The purpose of panchromatic sharpening, i.e., pansharpening, is to fuse a low spatial resolution multispectral (LRMS) image with a high spatial resolution panchromatic (PAN) image, aiming to obtain a high spatial resolution multispectral (HRMS) image. Pansharpening models based on variational optimization consist of a spectral fidelity term, a spatial fidelity term, and a regularization term. Most of the methods assume that the existing PAN image and the homologous HRMS image satisfy the global or local linear relationship, which could be far from the real case, thus causing sub-optimal performance. Inspired by the nonlinear mapping ability of machine learning (ML) techniques, we propose a novel spatial fidelity term with learnable nonlinear mapping (LNM-SF), which trains an implicit functional operator via a specifically designed convolutional neural network (CNN) and efficiently constructs the nonlinear relationship between the known PAN and the latent HRMS images. Relying upon the above description of the spatial fidelity term, a new variational model with a learnable nonlinear mapping in the spatial fidelity term for pansharpening, named LNM-PS, is simply integrated by the conventional spectral fidelity term into the proposed LNM-SF. To effectively solve the resulting optimization problem, we develop an alternating direction method of multipliers (ADMM)-based algorithm with the fast iterative shrinkage-thresholding algorithm (FISTA) as inner solver. Extensive numerical experiments on different datasets, assessing the performance both at reduced-resolution and full-resolution, show the superiority of the proposed LNM-PS method. The code is available at https://github.com/liangjiandeng/-LNM-PS.

Pansharpening (which stands for panchromatic sharpening) involves the fusion between a multispectral (MS) image with a higher spectral content than a fine spatial resolution panchromatic (PAN) image to generate a high spatial resolution multispectral (HRMS) image. A widely-used concept is the construction of the relationship between PAN and HRMS images by designing pixel-based coefficients. Previous pixel-based methods compute the coefficients pixel-by-pixel while suffering from inaccuracies in some areas leading to spatial distortion. However, we found that the coefficients inherit the spatial properties of the HRMS image, e.g., the local smoothness and nonlocal self-similarity, and the spatial correlation between the coefficients and the HRMS image can increase the accuracy of the estimation process. In this article, we propose a novel spatial fidelity with nonlocal regression (SFNLR) to describe the relationship between PAN and HRMS images. Unlike from the pixel-based perspective, the SFNLR can jointly utilize the local smoothness and nonlocal self-similarity of the coefficients for preserving spatial information. Besides, the SFNLR is integrated with a widely-used spectral fidelity to formulate a new variational model for the pansharpening problem. An effective algorithm based on the alternating direction method of multiplier (ADMM) framework is designed to solve the proposed model. Qualitative and quantitative assessments on reduced and full resolution datasets from different satellites demonstrate that the proposed approach outperforms several state-of-the-art methods. The code is available at: https://github.com/Jin-liangXiao/SFNLR.

Pansharpening refers to the fusion of a low spatial-resolution multispectral image with a high spatial-resolution panchromatic image. In this paper, we propose a novel low-rank tensor completion (LRTC)-based framework with some regularizers for multispectral image pansharpening, called LRTCFPan. The tensor completion technique is commonly used for image recovery, but it cannot directly perform the pansharpening or, more generally, the super-resolution problem because of the formulation gap. Different from previous variational methods, we first formulate a pioneering image super-resolution (ISR) degradation model, which equivalently removes the downsampling operator and transforms the tensor completion framework. Under such a framework, the original pansharpening problem is realized by the LRTC-based technique with some deblurring regularizers. From the perspective of regularizer, we further explore a local-similarity-based dynamic detail mapping (DDM) term to more accurately capture the spatial content of the panchromatic image. Moreover, the low-tubal-rank property of multispectral images is investigated, and the low-tubal-rank prior is introduced for better completion and global characterization. To solve the proposed LRTCFPan model, we develop an alternating direction method of multipliers (ADMM)-based algorithm. Comprehensive experiments at reduced-resolution (i.e., simulated) and full-resolution (i.e., real) data exhibit that the LRTCFPan method significantly outperforms other state-of-the-art pansharpening methods. The code is publicly available at: https://github.com/zhongchengwu/code_LRTCFPan.

Pansharpening refers to the fusion of a panchromatic (PAN) image with a high spatial resolution and a multispectral (MS) image with a low spatial resolution, aiming to obtain a high spatial resolution MS (HRMS) image. In this article, we propose a novel deep neural network architecture with level-domain-based loss function for pansharpening by taking into account the following double-type structures, i.e., double-level, double-branch, and double-direction, called as triple-double network (TDNet). By using the structure of TDNet, the spatial details of the PAN image can be fully exploited and utilized to progressively inject into the low spatial resolution MS (LRMS) image, thus yielding the high spatial resolution output. The specific network design is motivated by the physical formula of the traditional multi-resolution analysis (MRA) methods. Hence, an effective MRA fusion module is also integrated into the TDNet. Besides, we adopt a few ResNet blocks and some multi-scale convolution kernels to deepen and widen the network to effectively enhance the feature extraction and the robustness of the proposed TDNet. Extensive experiments on reduced- and full-resolution datasets acquired by WorldView-3, QuickBird, and GaoFen-2 sensors demonstrate the superiority of the proposed TDNet compared with some recent state-of-the-art pansharpening approaches. An ablation study has also corroborated the effectiveness of the proposed approach. The code is available at https://github.com/liangjiandeng/TDNet.

Pansharpening is a critical yet challenging low-level vision task that aims to obtain a higher-resolution image by fusing a multispectral (MS) image and a panchromatic (PAN) image. While most pansharpening methods are based on convolutional neural network (CNN) architectures with standard convolution operations, few attempts have been made with context-adaptive/dynamic convolution, which delivers impressive results on high-level vision tasks. In this paper, we propose a novel strategy to generate local-context adaptive (LCA) convolution kernels and introduce a new global harmonic (GH) bias mechanism, exploiting image local specificity as well as integrating global information, dubbed LAGConv. The proposed LAGConv can replace the standard convolution that is context-agnostic to fully perceive the particularity of each pixel for the task of remote sensing pansharpening. Furthermore, by applying the LAGConv, we provide an image fusion network architecture, which is more effective than conventional CNN-based pansharpening approaches. The superiority of the proposed method is demonstrated by extensive experiments implemented on a wide range of datasets compared with state-of-the-art pansharpening methods. Besides, more discussions testify that the proposed LAGConv outperforms recent adaptive convolution techniques for pansharpening.

Convolutional neural networks (CNNs) have achieved impressive performance for hyperspectral (HS) and multispectral (MS) image fusion in recent years. They extract features by local filters, which is limited to explore long-range dependency in input images. However, long-range dependence is an import cue for HS and MS image fusion, as it contributes to exploration of spatial self-similarity and spectral dependence. To take advantage of long-range dependence, we propose a spectral-spatial transformer (SST) for MS and HS image fusion. The experimental results demonstrate the high performance of the proposed approach compared to some state-of-the-art methods.

Hyperspectral image super-resolution (HISR) is to fuse a low-resolution hyperspectral image (LR-HSI) and a high-resolution multispectral image (HR-MSI), aiming to obtain a high-resolution hyperspectral image (HR-HSI). Recently, various convolution neural network (CNN) based techniques have been successfully applied to address the HISR problem. However, these methods generally only consider the relation of a local neighborhood by convolution kernels with a limited receptive field, thus ignoring the global relationship in a feature map. In this paper, we design a transformer-based architecture (called Fusformer) for the HISR problem, which is the first attempt to apply the transformer architecture to this task to the best of our knowledge. Thanks to the excellent ability of feature representations, especially by the self-attention in the transformer, our approach can globally explore the intrinsic relationship within features. Considering the specific HISR problem, since the LR-HSI holds the primary spectral information, our method estimates the spatial residual between the upsampled LR-MSI and the desired HR-HSI, reducing the burden of training the whole data in a smaller mapping space. Various experiments show that our approach outperforms current state-of-the-art HISR methods. The code is available at https://github.com/J-FHu/Fusformer.

Model-based approaches to pansharpening still constitute a class of widely employed methods, thanks to their straightforward applicability to many problems, dispensing the user from time-consuming training phases. The injection scheme based on an accurate estimation (exploiting regression) of the relationship between the details contained in the PAN image and those required for the enhancement of the MS image represents the most updated approach to this problem, being characterized by both theoretical and practical optimality. We elaborated on this scheme by designing a procedure for estimating the key parameters required for the optimal setting of such regression-based approach. We tested this approach on several datasets acquired by the WorldView satellites comparing the proposed approach with a benchmark consisting of some state-of-the-art pansharpening methods.

Although the performance of pansharpening has been significantly improved by advanced deep-learning (DL) technologies in recent years, most DL-based methods fail to process multispectral (MS) images with arbitrary numbers of bands by a single model. Consequently, it is inevitable to train separate models for MS images with different numbers of bands, which is time- and storage-consuming as well as inefficient in practice. To tackle the above problem, we propose a bidirectional recurrent pansharpening network (named ArbRPN) for MS images with arbitrary numbers of bands. Our ArbRPN can dynamically reconstruct high-resolution (HR) MS images with different numbers of bands by adaptively changing the number of recurrence to the number of bands of the low-resolution (LR) MS images. Leveraging on the ability of the ArbRPN to process MS images with any number of bands, one can even customize the bands to be pansharpened. Moreover, to achieve superior performance, spectral discrepancy and dependence are considered in the ArbRPN. Details from the panchromatic (PAN) image are adaptively injected into the fused product according to the captured spectral dependence. Furthermore, training strategies of existing DL-based pansharpening methods can only group MS images with a constant number of bands into mini-batches. Therefore, we present a mask-based training method (called mask-training) to solve this problem. Benefiting from the mask-training, our ArbRPN can achieve superior performance and robustness during pansharpening. Extensive experiments show the superior performance of our ArbRPN with respect to the state-of-the-art (SOTA) methods applied to MS images with different numbers of bands. The code of our ArbRPN is available onhttps://github.com/Lihui-Chen/ArbRPN.git.

Pansharpening refers to a spatio-spectral fusion of a lower spatial resolution multispectral (MS) image with a high spatial resolution panchromatic image, aiming at obtaining an image with a corresponding high resolution both in the domains. In this article, we propose a generic fusion framework that is able to weightedly combine variational optimization (VO) with deep learning (DL) for the task of pansharpening, where these crucial weights directly determining the relative contribution of DL to each pixel are estimated adaptively. This framework can benefit from both VO and DL approaches, e.g., the good modeling explanation and data generalization of a VO approach with the high accuracy of a DL technique thanks to massive data training. The proposed method can be divided into three parts: 1) for the VO modeling, a general details injection term inspired by the classical multiresolution analysis is proposed as a spatial fidelity term and a spectral fidelity employing the MS sensor’s modulation transfer functions is also incorporated; 2) for the DL injection, a weighted regularization term is designed to introduce deep learning into the variational model; and 3) the final convex optimization problem is efficiently solved by the designed alternating direction method of multipliers. Extensive experiments both at reduced and full-resolution demonstrate that the proposed method outperforms recent state-of-the-art pansharpening methods, especially showing a higher accuracy and a significant generalization ability.

Pansharpening is related to the fusion of a low spatial resolution multispectral (MS) image retaining an abundant spectral content and a high spatial resolution panchromatic (PAN) image to obtain a product with both the abundant spectral content of the former and the high spatial resolution of the latter. Many previous studies are only focused on the global or local relationship between the PAN image and the corresponding high-resolution multispectral (HRMS) image. However, we found that the relationship between PAN and HRMS images in the gradient domain can be better explored through the image context. In this article, we propose context-aware details injection fidelity (CDIF) with adaptive coefficients estimation, which can fully explore the complicated relationship between the PAN image and the HRMS image in the gradient domain. More specifically, we apply a clustering method to divide the pixels of an image into different context-based regions. Afterward, the adaptive coefficients are estimated by using a regression-based method for each region. The CDIF is effective in extracting the main features from the two inputs to be fused. In addition, we integrate the CDIF with a conventional fidelity term and a total variation regularization to formulate a novel variational pansharpening model that is solved by designing an algorithm based on the alternating direction method of multiplier (ADMM) framework. Qualitative and quantitative assessments on different datasets support the effectiveness and robustness of the proposed method. The code is available athttps://github.com/liangjiandeng/CDIF.

Hyperspectral images (HSIs) are of crucial importance in order to better understand features from a large number of spectral channels. Restricted by its inner imaging mechanism, the spatial resolution is often limited for HSIs. To alleviate this issue, in this work, we propose a simple and efficient architecture of deep convolutional neural networks to fuse a low-resolution HSI (LR-HSI) and a high-resolution multispectral image (HR-MSI), yielding a high-resolution HSI (HR-HSI). The network is designed to preserve both spatial and spectral information thanks to a new architecture based on: 1) the use of the LR-HSI at the HR-MSI's scale to get an output with satisfied spectral preservation and 2) the application of the attention and pixelShuffle modules to extract information, aiming to output high-quality spatial details. Finally, a plain mean squared error loss function is used to measure the performance during the training. Extensive experiments demonstrate that the proposed network architecture achieves the best performance (both qualitatively and quantitatively) compared with recent state-of-the-art HSI super-resolution approaches. Moreover, other significant advantages can be pointed out by the use of the proposed approach, such as a better network generalization ability, a limited computational burden, and the robustness with respect to the number of training samples. Please find the source code and pretrained models from https://liangjiandeng.github.io/Projects_Res/HSRnet_2021tnnls.html.

The fusion of high spatial resolution panchromatic (PAN) data with simultaneously acquired multispectral (MS) data with the lower spatial resolution is a hot topic, which is often called pansharpening. In this article, we exploit the combination of machine learning techniques and fusion schemes introduced to address the pansharpening problem. In particular, deep convolutional neural networks (DCNNs) are proposed to solve this issue. The latter is combined first with the traditional component substitution and multiresolution analysis fusion schemes in order to estimate the nonlinear injection models that rule the combination of the upsampled low-resolution MS image with the extracted details exploiting the two philosophies. Furthermore, inspired by these two approaches, we also developed another DCNN for pansharpening. This is fed by the direct difference between the PAN image and the upsampled low-resolution MS image. Extensive experiments conducted both at reduced and full resolutions demonstrate that this latter convolutional neural network outperforms both the other detail injection-based proposals and several state-of-the-art pansharpening methods.

The use of hyperspectral (HS) data is growing over the years, thanks to the very high spectral resolution. However, HS data are still characterized by a spatial resolution that is too low for several applications, thus motivating the design of fusion techniques aimed to sharpen HS images with high spatial resolution data. To reach a significant resolution enhancement, high-resolution images should be acquired by different satellite platforms. In this article, we highlight the pros and cons of employing real multiplatform data, using the EO-1 satellite as an exemplary case. The spatial resolution of the HS data collected by the Hyperion sensor is improved by exploiting both the ALI panchromatic image collected from the same platform and acquisitions from the WorldView-3 and the QuickBird satellites. Furthermore, we tackle the problem of assessing the final quality of the fused product at the nominal resolution, which presents further difficulties in this general environment. Useful indications for the design of an effective sharpening method in this case are finally outlined.

Pansharpening techniques allow a detailed reproduction of the Earth surface by fusing a multispectral (MS) and a panchromatic (PAN) image acquired over the same area. Classical pansharpening methods consist in the extraction of the details from the PAN image and their subsequent injection into the MS image through a linear function. In this letter, we propose to apply a nonlinear injection procedure that implements the detail injection through a polynomial function. Optimal polynomial coefficients in the least squares sense can be easily obtained in the closed form, and the consequent pansharpening algorithm is shown to obtain superior performance with respect to the existing linear approaches, especially for MS bands with a reduced wavelength overlap with the PAN channel.

Pansharpening refers to the combination of panchromatic (PAN) and multispectral (MS) images, designed to obtain a fused product retaining the fine spatial resolution of the former and the high spectral content of the latter. One of the most popular and successful approaches to pansharpening is the method known as context-based generalized Laplacian pyramid, which requires as a key ingredient for the estimation of the so-called injection coefficients. In this article, we propose the adoption of robust techniques for the estimation of the injection coefficients and detection strategies to select the clusters for which robust regression is needed, providing a suitable balancing between fusion performance and computational burden. Experimental results conducted on five real data sets acquired by the sensors QuickBird, WorldView-3, and WorldView-4, show the superiority of the proposed method with respect to current state-of-the-art pansharpening techniques.

Image fusion is growing interest in recent years, thanks to the huge amount of data acquired everyday by sensors on board of satellite platforms. The enhancement of the spatial resolution of a multispectral (MS) image through the use of a panchromatic (PAN) image, usually called pansharpening, is getting more and more relevant. In this work, we focus on the problem of the estimation of the injection coefficients that rule the enhancement of the spatial resolution of the MS image by properly adding the PAN details. In particular, a statistical analysis of the residuals coming from the linear multivariate regression between details extracted from the PAN image and the MS image is performed. A novel hybrid model is introduced for accurately describing the statistical distribution of these residuals, together with a procedure for efficiently estimating both the parameters of the residual distribution and the injection coefficients. The improvements achieved by the proposed approach are assessed using two very high resolution datasets acquired by the WorldView-3 and Worldview-4 satellites. The benefits of the proposed approach are particularly clear when vegetated areas are involved in the fusion process.

Pansharpening refers to the problem of fusing a multispectral (MS) image and a panchromatic image in order to get an MS image at a finer spatial resolution than the one of the original MS image. Due to the lack of a reference image, the assessment of the quality of a pansharpened product is a challenging task. Typical solutions are related to the reduced resolution assessment by exploiting Wald's protocol or to indexes without reference. In this letter, an efficient approach based on the combination of the two above-mentioned solutions is proposed. The performance is assessed using real data acquired by sensors with very different features mounted on-board of the Pléiades, the WorldView-3, and the WorldView-4 satellite platforms.

The use of cutting-edge geospatial technologies to monitor ecosystems and the development of tailored tools for assessing such natural areas is a fundamental task. In this context, the growing availability of hyperspectral (HS) imagery from satellite and aerial platforms can provide valuable information for the sustainable management of ecosystems. However, in some cases, the spectral richness provided by HS sensors is at the expense of spatial quality. To alleviate this inconvenience, which can be critical to monitor some heterogeneous and mixed natural areas, a number of HS sharpening techniques have been developed to increase the spatial resolution while trying to preserve the spectral content. This image processing field has attracted the interest of the scientific community, and many research studies have been conducted to assess the performance of different HS sharpening algorithms. In the last decade, however, many comparative studies rely upon simulated data. In this work, the challenging application of sharpening methods in real situations using multiplatform or multisensor data is also addressed. Thus, experiments with real data have been conducted, in addition to a thorough assessment of HS sharpening techniques using simulated imagery in scenarios with different spatial resolution ratios and registration errors. In particular, airborne and satellite HS imageries have been pansharpened with drone, orthophotos, and satellite high spatial resolution data evaluating 11 fusion algorithms. After a comprehensive analysis, considering different visual and quantitative quality indicators, the algorithm characteristics have been summarized and the methods with higher performance and robustness have been identified.

Pansharpening refers to the fusion of a multispectral (MS) image with a finer spectral resolution but coarser spatial resolution than a panchromatic (PAN) image. The classical pansharpening problem can be dealt with component substitution or multiresolution analysis techniques. One of the most notable approaches in the former class is the band-dependent spatial-detail (BDSD) method. It has been shown state-of-the-art performance, in particular, when the fusion of four band data sets is addressed. However, new sensors, such as the WorldView-2/-3 ones, usually acquire MS images with more than four spectral bands to be fused with the PAN image. The BDSD method has shown limitations in performance in these cases. Thus, in this paper, several BDSD-based approaches are provided to solve this issue getting a robustness of the BDSD with respect to the spectral bands to be fused. The experimental results conducted both at reduced and at full resolutions on four real data sets acquired by the IKONOS, the QuickBird, the WorldView-2, and the WorldView-3 sensors demonstrate the validity of the proposed approaches against the benchmark.

The combination of a multispectral (MS) image and a panchromatic (PAN) image, the so-called pansharpening, allows to produce very appealing images that are useful both for visual interpretation and for feature extraction. The state-of-the-art multiresolution analysis pansharpening algorithms are based on the extraction of spatial details from the PAN image through image filters matched with the MS sensors' modulation transfer function. However, this knowledge is often poor due to measurement inaccuracies and/or its aging. Thus, deconvolution algorithms have been proposed to overcome this limitation. In this paper, we propose a multiband filter estimation (FE) approach to improve the solutions in the literature. The main idea in this paper is to exploit a preliminary pansharpened image to estimate the spatial filter used for detail extraction associated with each spectral band. We demonstrate that the proposed method outperforms the state-of-the-art FE approaches by employing data sets acquired by the IKONOS, the Quickbird, and the WorldView-3 sensors.

Pansharpening usually refers to the fusion of a high spatial resolution panchromatic image with a low spatial resolution multispectral image. One of the most debated issue in this research field regards the quality assessment of fused products. The two exploited quality assessments are at reduced resolution and at full resolution. The former is an accurate procedure, but the main drawback is that it works on synthetic (with lower spatial resolutions) products. The latter is able to work at full resolution paying it with a reduced accuracy due to the absence of a ground-truth. In this work, we will focus on the assessment at full resolution by introducing a new measure of spatial consistency based on multivariate linear regression of the panchromatic image towards the multispectral bands. Simulations with an IKONOS dataset and six fusion methods show that the proposed spatial index is the ideal counterpart of Khan's spectral consistency index.

Pansharpening regards the fusion of a high spatial resolution but low spectral resolution (panchromatic) image with a high spectral resolution but low spatial resolution (multispectral) image. The estimation, at reduced resolution, of injection coefficients through regression is a widespread and powerful approach. In this work, the problem of the estimation of the injection coefficients at full resolution for regression-based pansharpening approaches is studied. Multiple approaches (based on guess images or an iterative method) are proposed. These are assessed at reduced resolution by exploiting a real dataset acquired by the IKONOS sensor. The quantitative results clearly demonstrate the superiority of the proposed iterative method.

Pansharpening usually refers to the fusion of a high spatial resolution panchromatic (PAN) image with a higher spectral resolution but coarser spatial resolution multispectral (MS) image. Owing to the wide applicability of related products, the literature has been populated by many papers proposing several approaches and studies about this issue. Many solutions require a preliminary spectral matching phase wherein the PAN image is matched with the MS bands. In this paper, we propose and properly justify a new approach for performing this step, demonstrating that it yields state-of-the-art performance. The comparison with existing spectral matching procedures is performed by employing four data sets, concerning different kinds of landscapes, acquired by the Pléiades, WorldView-2, and GeoEye-1 sensors.

Pansharpening regards the fusion of a high-spatial resolution panchromatic image with a low-spatial resolution multispectral image. One of the most debated topics about pansharpening is related to the quality assessment of fused products. Two main assessment procedures are usually exploited in the literature: the reduced resolution validation and the full-resolution (FR) validation. The former has the advantage to be accurate, but the hypothesis of invariance among scales has to be assumed. The latter overcomes this limitation but paying it with a lower accuracy. In this paper, we will focus on the FR assessment proposing an approach for estimating an overall quality index at FR by using multiscale FR measurements. The problem is recast into the sequential Bayesian framework exploiting a Kalman filter to find its solution. The proposed procedure for quality evaluation has been tested on four real data sets acquired by the Pléiades, the GeoEye-1, the WorldView-3, and the WorldView-4 sensors assessing the quality of 19 pansharpened methods. The proposed approach has demonstrated its superiority with respect to the benchmark consisting of state-of-the-art quality assessment procedures.

Pansharpening is an important application in remote sensing image processing. It can increase the spatial-resolution of a multispectral image by fusing it with a high spatial-resolution panchromatic image in the same scene, which brings great favor for subsequent processing such as recognition, detection, etc. In this paper, we propose a continuous modeling and sparse optimization based method for the fusion of a panchromatic image and a multispectral image. The proposed model is mainly based on reproducing kernel Hilbert space (RKHS) and approximated Heaviside function (AHF). In addition, we also propose a Toeplitz sparse term for representing the correlation of adjacent bands. The model is convex and solved by the alternating direction method of multipliers which guarantees the convergence of the proposed method. Extensive experiments on many real datasets collected by different sensors demonstrate the effectiveness of the proposed technique as compared with several state-of-the-art pansharpening approaches.

Pansharpening is usually related to the fusion of a high spatial resolution but low spectral resolution (panchromatic) image with a high spectral resolution but low spatial resolution (multispectral) image. The calculation of injection coefficients through regression is a very popular and powerful approach. These coefficients are usually estimated at reduced resolution. In this paper, the estimation of the injection coefficients at full resolution for regression-based pansharpening approaches is proposed. To this aim, an iterative algorithm is proposed and studied. Its convergence, whatever the initial guess, is demonstrated in all the practical cases and the reached asymptotic value is analytically calculated. The performance is assessed both at reduced resolution and at full resolution on four data sets acquired by the IKONOS sensor and the WorldView-3 sensor. The proposed full scale approach always shows the best performance with respect to the benchmark consisting of state-of-the-art pansharpening methods.

Inpainting in hyperspectral imagery is a challenging research area and several methods have been recently developed to deal with this kind of data. In this paper we address missing data restoration via a convex optimization technique with regularization term based on Collaborative Total Variation (CTV). In particular we evaluate the effectiveness of several instances of CTV in conjunction with different dimensionality reduction algorithms.

In this paper, we propose a continuous modeling and sparse optimization based method for the fusion of a panchromatic (PAN) image and a multispectral (MS) image. The proposed model is mainly based on reproducing kernel Hilbert space (RKHS) and approximated Heaviside function (AHF). In addition, we also design an iterative strategy to recover more image details. The final model is a convex one and solved by the designed alternating direction method of multipliers (ADMM) which guarantees the convergence of the proposed method. Experimental results on two real datasets corresponding to different sensors and different resolutions demonstrate the effectiveness of the proposed approach as compared with several state-of-the-art pansharpening approaches.

Variational methods are widely used in image processing for problems ranging from denoising to data fusion. In this paper we focus on a recent regularization method, called Collaborative Total Variation, applied to the hyperspectral pansharpening, which deals with the fusion of low resolution hyperspectral and high resolution panchromatic images. The effectiveness of this novel approach is evaluated for different Collaborative Norms and the assessment is performed on the Pavia University dataset.

In this letter, we show that pansharpening of visible/near-infrared (VNIR) bands takes advantage from a correction of the path-radiance term introduced by the atmosphere during the fusion process. This holds whenever the fusion mechanism emulates the radiative transfer model ruling the acquisition of the Earth's surface from space, that is, for methods exploiting a contrast-based injection model of spatial details extracted from the panchromatic (Pan) image into the interpolated multispectral (MS) bands. Such methods are high-pass modulation (HPM), Brovey transform, synthetic variable ratio (SVR), University of New Brunswick pansharp, smoothing filter-based intensity modulation, and spectral distortion minimization. The path radiance should be estimated and subtracted from each band before the product by Pan is accomplished and added back after. Both empirical and model-based estimation techniques of MS path radiances are compared within the framework of optimized SVR and HPM algorithms. Simulations carried out on QuickBird and IKONOS data highlight that haze correction of MS before fusion is always beneficial, especially on vegetated areas and in terms of spectral quality.

Classical pansharpening algorithms constitute a class of image fusion methods that have been widely investigated in the literature. They have been developed for combining a single- and a multichannel image (panchromatic (PAN) and multispectral (MS), respectively), but can be adapted to the sharpening of hyperspectral (HS) data, both through companion PAN and MS images. We focus in this letter on the HS/MS fusion, showing that the assignation of the MS channel to each HS band is a key step, and investigate several alternatives to make this choice. The assignment algorithms are tested in conjunction with both component substitution and multiresolution analysis pansharpening methods and assessed on images acquired by the Hyperion and ALI sensors. The numerical evaluation shows that the best results can be obtained by optimizing the spectral angle mapper metric confirming that classical methods represent a reliable basis for the development of novel sharpening algorithms.

In this paper, the authors investigate the statistical matching of the panchromatic (Pan) image to the multispectral (MS) bands, also known as the histogram matching, for the two main classes of pansharpening methods, i.e., those based on component substitution (CS) or spectral methods and those based on multiresolution analysis (MRA) or spatial methods. Also, hybrid methods combining CS with MRA, like the widespread additive wavelet luminance proportional (AWLP), are investigated. It is shown that all spectral, spatial, and hybrid methods must perform a dynamics matching of the enhancing Pan to the individual MS bands for MRA or a combination of them (the component that shall be substituted) for CS. For hybrid methods, the problem is more complex and both types of histogram matching may be suitable. Such an intersensor balance may be either explicit or implicitly performed by the detail-injection model, e.g., the popular projective and multiplicative injection models. An experimental setup exploiting IKONOS and WorldView-2 data sets demonstrates that a correct histogram matching is the key to attain extra performance from established methods. As a first result of this paper, the AWLP method has been revisited and its performance significantly improved by simply performing the histogram matching of Pan to the individual MS bands, rather than to the intensity component, thereby losing the original proportionality feature.

Pansharpened images are widely used synthetic representations of the Earth surface characterized by both a high spatial resolution and a high spectral diversity. They are usually generated by extracting spatial details from a high-resolution PANchromatic image and by injecting them into a low spatial resolution multispectral image. The details injection is performed through injection coefficients, whose values can be either uniform for the whole image (global methods) or spatially variant (context-adaptive (CA) approaches). In this paper, we propose a CA approach in which the injection coefficients are estimated over image segments achieved through a binary partition tree segmentation algorithm. The approach is applied to two credited pansharpening algorithms based on the Gram-Schmidt orthogonalization procedure and the generalized Laplacian pyramid technique. The performance assessment is performed using two different data sets acquired by the QuickBird and the WorldView-3 satellites. The validation procedure, both at full and at reduced resolution, shows the suitability of the proposed approach, which reaches a good tradeoff between accuracy and computational burden.

Ship traffic monitoring is a foundation for many maritime security domains, and monitoring system specifications underscore the necessity to track vessels beyond territorial waters. However, vessels in open seas are seldom continuously observed. Thus, the problem of long-term vessel prediction becomes crucial. This paper focuses attention on the performance assessment of the Ornstein-Uhlenbeck (OU) model for long-term vessel prediction, compared with usual and well-established nearly constant velocity (NCV) model. Heterogeneous data, such as automatic identification system (AIS) data, high-frequency surface wave radar data, and synthetic aperture radar data, are exploited to this aim. Two different association procedures are also presented to cue dwells in case of gaps in the transmission of AIS messages. Suitable metrics have been introduced for the assessment. Considerable advantages of the OU model are pointed out with respect to the NCV model.

Thermal Sharpening (TS) is usually referred to techniques widely used in several Earth Observation applications in order to increase the spatial resolution of thermal images. Profiting from the particular design of the Visible Infrared Imaging Radiometer Suite (VIIRS) sensor mounted on board of the Suomi National Polar-orbiting Partnership (NPP) satellite, we propose here a new approach for obtaining synthetic thermal data with increased spatial resolution and spectral diversity. The method exploits classical Pansharpening algorithms, which are very popular in the field of Visible and Near-InfraRed (VNIR) image fusion, for combining the VIIRS thermal bands with partially overlapping spectral responses. We evaluate the effectiveness of several algorithms by performing a Reduced Resolution (RR) assessment on VIIRS real data, showing the importance of an adequate knowledge of the sensor characteristics.

Accurate representations of the Earth surface in both spatial and spectral domains are highly desirable in many applications using remotely sensed data. An effective solution is achieved by combining hyperspectral data, which are characterized by a high spectral diversity, with high spatial resolution images, collected by multispectral or panchromatic sensors. In this work, we compare the outcomes provided by fusing single-platform or multi-platform data. We demonstrate that the optimal choice depends on the target spatial resolution to be achieved. To this aim, real images collected by the Hyperion sensor are combined with data acquired by the ALI sensor or the QuickBird sensor assessing the fused outcomes at reduced resolution.

Radar imaging is typically based on linear models of the electromagnetic scattering phenomenon. These models are robust and computationally efficient, but do not account for mutual interactions among targets in the scene and between the targets and the surrounding environment. As a result, the radar images are characterized by spurious targets, i.e., multipath ghosts, which appear at positions where no physical targets exist. In this letter, we compare two key approaches for clutter suppression. The first approach applies multiplicative fusion of the images corresponding to subapertures of the deployed array, whereas the second approach is based on coherence factor filtering, which enhances the image quality by suppressing low-coherence features. We assess the performance of these two methods in terms of imaging and detection capabilities. Numerical results based on synthetic data are reported to support the comparative analysis.

Nonlinear decomposition schemes constitute an alternative to classical approaches for facing the problem of data fusion. In this paper, we discuss the application of this methodology to a popular remote sensing application called pansharpening, which consists in the fusion of a low resolution multispectral image and a high-resolution panchromatic image. We design a complete pansharpening scheme based on the use of morphological half gradient operators and demonstrate the suitability of this algorithm through the comparison with the state-of-the-art approaches. Four data sets acquired by the Pleiades, Worldview-2, Ikonos, and Geoeye-1 satellites are employed for the performance assessment, testifying the effectiveness of the proposed approach in producing top-class images with a setting independent of the specific sensor.

Obtaining radiometric surface temperature information with both high acquisition rate and high spatial resolution is still not possible through a single sensor. However, in several earth observation applications, the fusion of data acquired by different sensors is a viable solution for so called image sharpening. A related issue is the presence of clouds, which may impair the performance of the data fusion algorithms. In this paper we propose a robustified setup for the sharpening of thermal images in a non real-time scenario, capable to deal with missing thermal data due to cloudy pixels, and robust with respect to cloud mask misclassifications. The effectiveness of the presented technique is assessed via numerical simulations based on SEVIRI data.

Recent developments in high resolution sensors have encouraged the use of Extended Target Tracking (ETT) algorithms specifically designed to deal with targets that generate more than one detection per frame. At the same time, the availability of more powerful computational resources enable the use of soft computing techniques that yield a target probability, instead of a hard decision. This paper proposes a realistic target model feasible for an Extended Target - Track before Detect framework. Physical phenomena related to the acquisition of high resolution X-band marine radar data are considered. Real radar data is used to assess the superior performance of the featured model with respect to previous approaches. Results show that the realistic model provides better estimations of the target velocity and size.

Pansharpening algorithms enable to produce synthetic data with high spatial details and spectral diversity by combining a panchromatic image with multispectral or hyperspectral data. In classical approaches the details extracted from the panchromatic image are introduced into the original multichannel image through injection gains, which can be spatially variant on the image. In this paper we analyze several methods for partitioning an image into regions in which the pixels will share the same injection coefficients. Gram-Schmidt pansharpening methods are used as paradigmatic examples for assessing the performance of global and local gain estimation strategies, using hyperspectral data acquired by sensors mounted on one (Earth Observing-1) or multiple (PROBA and Quick-bird) satellite platforms.

X-band Marine radar systems are low-cost tools for monitoring multiple targets in a surveillance area. Although they may suffer from several sources of interference, high resolution measurements in both space and time can be provided. Such features offer the opportunity to get accurate information not only about the targets' kinematics, as other conventional sensors, but also about the targets' extent. In this paper, a signal processing chain composed by a detector and a joint probabilistic data association tracker is proposed to address the problem of tracking using X-band Marine radar data. Estimations of both the targets' kinematics, i.e. positions and velocities, and length and width, are provided. The performance assessment, conducted on real data acquired by an X-band Marine radar located in the Gulf of La Spezia, Italy, demonstrates the ability of the processing chain to obtain high tracking performance with a limited computational burden.

In recent years, low-power high-frequency surface-wave radars have received significant attention thanks to their over-the-horizon coverage capability and the continuous-time operation mode. These radars have become effective long-range early-warning tools for maritime situational awareness applications. In this paper a knowledge-based multi-target tracking algorithm is described. The advantages in using a prior information on ship traffic are assessed exploiting real data acquired by two high-frequency surface-wave radars. The outcomes confirm the ability of the proposed approach to better follow targets with a time-on-target increment up to 30% with respect to existing methods. A reduction of the track fragmentation up to 20% is also observed.

Pansharpening consists of fusing a multispectral (MS) image together with a panchromatic (PAN) image with the aim of jointly preserving the spectral diversity of the former and the geometric richness of the latter. A crucial step in pansharpening algorithms is the detail extraction. This problem is usually addressed by the means of 2D Gaussian filters matched with the MS sensor's modulation transfer function (MTF). Nevertheless, several issues can affect this characterization (e.g. the MTF's gains at the Nyquist frequency could be not available or unreliable). Thus, in this paper we propose a technique based on blind image deblurring in order to estimate band-dependent spatial detail extraction filters by taking into consideration the possible variability of the MS spatial features along bands. The validation is carried out exploiting two real datasets acquired by the IKONOS and the QuickBird sensors.

The application of sparse representation (SR) theory to the fusion of multispectral (MS) and panchromatic images is giving a large impulse to this topic, which is recast as a signal reconstruction problem from a reduced number of measurements. This letter presents an effective implementation of this technique, in which the application of SR is limited to the estimation of missing details that are injected in the available MS image to enhance its spatial features. We propose an algorithm exploiting the details self-similarity through the scales and compare it with classical and recent pansharpening methods, both at reduced and full resolution. Two different data sets, acquired by the WorldView-2 and IKONOS sensors, are employed for validation, achieving remarkable results in terms of spectral and spatial quality of the fused product.

Tracking moving targets hidden behind visually opaque structures as building walls is a crucial issue in many surveillance, rescue, and security applications. The electromagnetic waves at the low microwave frequency range penetrate into common building materials and thereby enable the radar to expose behind the wall scene. However, due to complexity of the scattering scenario, the radar signal undergoes multipath propagation phenomena. These typically manifest themselves as environmental clutter which may impair detection and tracking of true targets. In this paper, a signal processing strategy is proposed to track multiple extended targets in a scene by means of a wide-band monostatic through-wall radar. The system collects data sets at regular time steps which are first processed by a microwave tomographic technique. Then, a detection/tracking stage is implemented in order to track the position and dynamics of targets in real time. An extended target-tracking approach is applied to properly exploit at the tracking stage the information related to extended nature of targets. The effectiveness of the proposed signal processing chain is assessed by numerical tests based on full-wave data pertaining to an indoor scenario.

Pansharpening aims at fusing a multispectral and a panchromatic image, featuring the result of the processing with the spectral resolution of the former and the spatial resolution of the latter. In the last decades, many algorithms addressing this task have been presented in the literature. However, the lack of universally recognized evaluation criteria, available image data sets for benchmarking, and standardized implementations of the algorithms makes a thorough evaluation and comparison of the different pansharpening techniques difficult to achieve. In this paper, the authors attempt to fill this gap by providing a critical description and extensive comparisons of some of the main state-of-the-art pansharpening methods. In greater details, several pansharpening algorithms belonging to the component substitution or multiresolution analysis families are considered. Such techniques are evaluated through the two main protocols for the assessment of pansharpening results, i.e., based on the full- and reduced-resolution validations. Five data sets acquired by different satellites allow for a detailed comparison of the algorithms, characterization of their performances with respect to the different instruments, and consistency of the two validation procedures. In addition, the implementation of all the pansharpening techniques considered in this paper and the framework used for running the simulations, comprising the two validation procedures and the main assessment indexes, are collected in a MATLAB toolbox that is made available to the community.

These last decades spawned a great interest toward low-power high-frequency (HF) surface-wave (SW) radars for ocean remote sensing. By virtue of their over-the-horizon coverage capability and continuous-time mode of operation, these sensors are also effective long-range early warning tools in maritime situational awareness applications providing an additional source of information for target detection and tracking. Unfortunately, they also exhibit many shortcomings that need to be taken into account, and proper algorithms need to be exploited to overcome their limitations. In this paper, we develop a knowledge-based (KB) multitarget tracking methodology that takes advantage of a priori information on the ship traffic. This a priori information is given by the ship sea lanes and by their related motion models, which together constitute the basic building blocks of a variable structure interactive multiple model procedure. False alarms and missed detections are dealt with using a joint probabilistic data association rule and nonlinearities are handled by means of the unscented Kalman filter. The KB-tracking procedure is validated using real data acquired during an HF-radar experiment in the Ligurian Sea (Mediterranean Sea). Two HFSW radar systems were operated to develop and test target detection and tracking algorithms. The overall performance is defined in terms of time-on-target, false-alarm rate (FAR), track fragmentation (TF), and accuracy. A full statistical characterization is provided using one month of data. A significant improvement of the KB-tracking procedure, in terms of system performance, is demonstrated in comparison with a standard joint probabilistic data association tracker recently proposed in the literature to track HFSW radar data. The main improvement of our approach is the better capability of following targets without increasing the FAR. This increment is much more evident in the region of low FAR, where it can be over the 30% for both the HFSW radar systems. The KB-tracking exhibits on average a reduction of the TF of about the 20% and the 13% of the utilized HFSW-radar systems.

Many powerful pansharpening approaches exploit the functional relation between the fusion of PANchromatic (PAN) and MultiSpectral (MS) images. To this purpose, the modulation transfer function of the MS sensor is typically used, being easily approximated as a Gaussian filter whose analytic expression is fully specified by the sensor gain at the Nyquist frequency. However, this characterization is often inadequate in practice. In this paper, we develop an algorithm for estimating the relation between PAN and MS images directly from the available data through an efficient optimization procedure. The effectiveness of the approach is validated both on a reduced scale data set generated by degrading images acquired by the IKONOS sensor and on full-scale data consisting of images collected by the QuickBird sensor. In the first case, the proposed method achieves performances very similar to that of the algorithm that relies upon the full knowledge of the degrading filter. In the second, it is shown to outperform several very credited state-of-the-art approaches for the extraction of the details used in the current literature.

Pansharpening is a successful application of data fusion to remotely sensed data. It aims at obtaining a detailed representation of an Earth's zone both in terms of spatial and spectral resolution. This is done through the fusion of a panchromatic and a multispectral image (having complementary spatial and spectral resolutions) that are acquired simultaneously by several optical satellites. The result of the fusion is commonly achieved by introducing the spatial details, modulated opportunely by gains, in the multispectral one. The injection gains can be estimated globally over the image, or locally, thus obtaining spatially variant values. The latter approach has been proven to achieve better results and it is based on windowing the analyzed image in squared blocks. In this paper we propose a more elaborated concept of locality, as it is based on an opportune segmentation of the target scene. In greater details, we propose to estimate the local injection gains on regions composed of pixel with similar spectral characteristic, as defined by a segmentation. Such local approach is compared to the global one and to the conventional local estimation based on overlapping and non-overlapping blocks. The performances have been assessed by using three real datasets, the first acquired by WorldView-2 and the other two by Pléiades. The analysis evidences the appreciable improvements of the performances with respect to classical schemes.

In several human activities, such as agriculture and forest management, the monitoring of radiometric surface temperature is key. In particular both high spatial resolution and high acquisition rate are desirable but, due to the hardware limitations, these two characteristics are not met by the same sensor. The fusion of remotely sensed data acquired by sensors with different spatial and temporal resolution is a profitable choice to face this issue. When the real-time requirement is relaxed, the data sequence can be processed as a whole, allowing to improve the final result. Within this framework, we propose a novel batch sharpening strategy, relying on interpolation, data fusion and Bayesian smoothing techniques, and we assess its effectiveness on SEVIRI and MODIS thermal data.

The design of a pansharpening algorithm for enriching a MultiSpectral image with the spatial details of a Panchromatic image should preserve the characteristics of the original dataset. A widely employed quality check consists in verifying the consistency of the fused product, namely the similarity of the original image and a reduced resolution version of the sharpened product. We propose to improve this feature by applying an Iterative Back-Projection algorithm after the fusion procedure. The approach is validated on two datasets, acquired by the Ikonos and WorldView-2 sensors, showing remarkable improvements, especially in conjunction with Component Substitution pansharpening methods.

In this paper state-of-the-art and advanced methods for multispectral pansharpening are reviewed and evaluated on two very high resolution datasets acquired by IKONOS-2 (four bands) and WorldView-2 (eight bands). The experimental analysis allows us to highlight the performances of the two main pansharpening approaches (i.e. component substitution and multiresolution analysis).

Images with high spatial and spectral resolutions are desirable for remote sensing applications. Unfortunately, due to sensor physical constraints, this result cannot be obtained by a single sensor. To overcome these limitations, a great number of data fusion approaches have been developed in the last years. The fusion of panchromatic and multispectral images, also known as Pansharpening, is capturing a lot of attention in the literature. In this paper, we extend and analyze the use of some classical pansharpening techniques, belonging to the MultiResolution Analysis and Component Substitution families, for fusing hyperspectral data instead of multispectral ones. The experimental results, conducted on two real datasets acquired by the Hyperion/ALI and CHRIS-Proba/QuickBird sensors, point out the greater suitability of the algorithms into the MRA class thanks to a better spectral consistency of the final products, which is a desirable feature when the number of bands to fuse increases.

The pansharpening process has the purpose of building a high-resolution multispectral image by fusing low spatial resolution multispectral and high-resolution panchromatic observations. A very credited method to pursue this goal relies upon the injection of details extracted from the panchromatic image into an upsampled version of the low-resolution multispectral image. In this letter, we compare two different injection methodologies and motivate the superiority of contrast-based methods both by physical consideration and by numerical tests carried out on remotely sensed data acquired by IKONOS and Quickbird sensors.

A recurrent concern in cloud detection approaches is the high misclassification rate for pixels close to cloud edges. We tackle this problem by introducing a novel penalty term within the classical maximum a posteriori probability-Markov random field (MAP-MRF) approach. To improve the classification rate, such term, for which we suggest two different functional forms, accounts for the predictable motion of cloud volumes across images. Two mass tracking techniques are proposed. The first one is an effective and efficient implementation of the probability hypothesis density (PHD) filter, which is based on Gaussian mixtures (GMs) and relies on finite set statistics (FISST). The second one is a region matching procedure based on a maximum cross-correlation (MCC) that is characterized by low computational load. Through extensive tests on simulated images and real data, acquired by the SEVIRI sensor, both methods show a clear performance gain in comparison with classical spatial MRF-based algorithms.

The continuous time monitoring of the radiometric surface temperature by means of high spatial resolution images is desirable in agricoltural applications, such as irrigation management. Since the requirement of high spatial and temporal resolutions can hardly be met by a single sensor, we resort to a fusion strategy of data from multiple sensors. Specifically we consider the Interacting Sequential Bayesian Estimation strategy, as it is able to deal with the sudden changes observed in the temperature dynamics. The method has been validated on SEVIRI TIR real data, properly spatially degraded in order to mimic sensors with different characteristics.

The design of multitarget tracking procedures includes, as the most time consuming steps, the definition of the objective class and the formulation of the detection criteria. In this paper we investigate a solution toward an intuitive way for implementing a detector for any ad-hoc application. We capitalize on the OSPA metric to discriminate between the semantic object class of interest and other look-alike classes starting from a short number of unlabeled markers. We propose an illustrative algorithm with a toy example, then we apply it to two real images, the first acquired by SEVIRI, the second by MERIS. In the first case we discriminate between lakes, sea and look-alike clouds, in the other between ground and sea ice. We show how semantic classes with very similar spectral properties can be separated even in the presence of uncertainties or errors in the ground truth.

Pansharpening algorithms aim to enhance low resolution multi-spectral images by means of high resolution panchromatic ones. Several approaches are based on the MultiResolution Analysis (MRA) achieved through the pyramidal decomposition of images. We focus here on the implementation based on Morphological Filters (MF) that are optimized through Genetic Algorithms (GA). The effectiveness of this algorithm is compared with other techniques, among which those based on Wavelet operators, through several quality indices on two different real scenarios.

Temporal correlation has been recently taken into consideration to improve the performances of cloud detection algorithms. We exploit this concept within the Maximum A Posteriori Markov Random Field MAP-MRF framework by adding a penalization term which is determined according to the hystory of cloud masses. Multi Target Tracking of clouds is accomplished by methods of Finite Set Statistics (FISS) and several particle-based implementations are compared among them and with other previous methods both on simulated and real data.