Establishing point-to-point correspondences across multiple 3D shapes is a fundamental problem in computer vision and graphics. In this paper, we introduce DcMatch, a novel unsupervised learning framework for non-rigid multi-shape matching. Unlike existing methods that learn a canonical embedding from a single shape, our approach leverages a shape graph attention network to capture the underlying manifold structure of the entire shape collection. This enables the construction of a more expressive and robust shared latent space, leading to more consistent shape-to-universe correspondences via a universe predictor. Simultaneously, we represent these correspondences in both the spatial and spectral domains and enforce their alignment in the shared universe space through a novel cycle consistency loss. This dual-level consistency fosters more accurate and coherent mappings. Extensive experiments on several challenging benchmarks demonstrate that our method consistently outperforms previous state-of-the-art approaches across diverse multi-shape matching scenarios.

The differences in imaging devices cause multimodal images to have modal differences and geometric distortions, complicating the matching task. Deep learning-based matching methods struggle with multimodal images due to the lack of large annotated multimodal datasets. To address these challenges, we propose XCP-Match based on cross-modality completion pre-training. XCP-Match has two phases. (1) Self-supervised cross-modality completion pre-training based on real multimodal image dataset. We develop a novel pre-training model to learn cross-modal semantic features. The pre-training uses masked image modeling method for cross-modality completion, and introduces an attention-weighted contrastive loss to emphasize matching in overlapping areas. (2) Supervised fine-tuning for multimodal image matching based on the augmented MegaDepth dataset. XCP-Match constructs a complete matching framework to overcome geometric distortions and achieve precise matching. Two-phase training encourages the model to learn deep cross-modal semantic information, improving adaptation to modal differences without needing large annotated datasets. Experiments demonstrate that XCP-Match outperforms existing algorithms on public datasets.

Due to the high-luminance (HL) background clutter in infrared (IR) images, the existing IR small target detection methods struggle to achieve a good balance between efficiency and performance. Addressing the issue of HL clutter, which is difficult to suppress, leading to a high false alarm rate, this letter proposes an IR small target detection method based on local-global feature fusion (LGFF). We develop a fast and efficient local feature extraction operator and utilize global rarity to characterize the global feature of small targets, effectively suppressing a significant amount of HL clutter. By integrating local and global features, we achieve further enhancement of the targets and robust suppression of the clutter. Experimental results demonstrate that the proposed method outperforms existing methods in terms of target enhancement, clutter removal, and real-time performance.

Non-blind rotary motion deblurring (RMD) aims to restore a latent image from its blurred image. Since the integration path of rotary motion blurring (RMB) is a circle, RMD is modelled as a typical motion deblurring in the polar coordinate system (PCS). However, existing PCS-based methods use hand-designed image priors and are limited by transformation errors, including Cartesian-to-polar transformation (CPT) error and polar-to-Cartesian transformation (PCT) error. In this paper, we analyze the impact of transformation errors on the restored image and propose a novel end-to-end network which introduces a convolutional neural network (CNN) to learn image priors. Specifically, considering the CPT error, we construct a degradation model and solve it in an unrolling way, effectively reducing the ringing artifacts. For the PCT error, we develop a PCT error correction module (PCM) to reconstruct the lost details and textures. Experiments show our method performs against state-of-the-art (SOTA) approaches on synthetic and real-world rotary motion blur datasets by a large margin. The code and model are available athttps://github.com/Jinhui-Qin/RMD_PCS.

Infrared small target detection (IRSTD) has significantly benefited from UNet-based neural models in recent years. However, current methodologies face challenges in achieving optimal compromise between missed detections and false alarms. To overcome this limitation, we rethink the role of each structural component within UNet-based architectures applied for IRSTD. Accordingly, we conceptualize the UNet’s encoder as specializing in feature extraction, the skip connections in feature selection, and the decoder in fusion-based reconstruction. Building upon these conceptualizations, we propose the DSTransNet. Within the feature extraction stage, the edge shape receptive field (ESR) module enhances edge and shape feature extraction and expands the receptive field via multiple convolutional branches, thereby reducing missed detections. At the feature selection stage, the reliable dynamic selection filtering (RDSF) module employs dynamic feature selection, leveraging encoder-based self-attention and decoder-based cross-attention of the Transformer to suppress background features resembling small targets and mitigate false alarms. During the feature fusion-based reconstruction stage, the cross-attention of spaces and channels (CSCE) module emphasizes small target features via spatial and channel cross-attention, reconstructing more accurate multi-scale detection masks. Extensive experiments on the SIRST, NUDT-SIRST, and SIRST-Aug datasets demonstrate that the proposed DSTransNet method outperforms state-of-the-art IRSTD approaches. The code is available at https://github.com/RuiminHuang/DSTransNet.

Infrared small target detection (IRSTD) faces significant challenges due to the small scale and low intensity of targets, which are characterized by extremely sparse features. Most existing methods primarily concentrate on spatial features while neglecting the significant cluttered interference inherent in complex backgrounds. Such an oversight poses substantial challenges in distinguishing targets from background noise, thereby limiting detection performance. Drawing inspiration from the frequency characteristics that differentiate targets from backgrounds in infrared images, we introduce an innovative detection network that leverages high- and low-frequency partitioning and interaction. Specifically, we introduce a patch-wise fast Fourier transform (PFFT), which divides the input image into patches and applies the Fourier transform to each patch. Subsequently, we employ convolutional neural networks (CNNs) for learnable high- and low-frequency partitioning and propose a learnable frequency augmentation module (FAM) to enhance the interfrequency and intrafrequency feature. This methodology effectively harnesses the spatial information inherent in both high and low frequencies to suppress background clutter and accurately extract sparse target features. Furthermore, to further integrate frequency information with spatial information, we propose a frequency spatial fusion module (FSFM) to merge features from frequency and spatial domains. Experimental results show that our method surpasses state-of-the-art techniques on four publicly available datasets.

Denoising Diffusion Probabilistic Model (DDPM) has demonstrated exceptional performance in low-light enhancement task. However, the dependency on paired training datas has left the generality of DDPM in low-light enhancement largely untapped. Therefore, this paper proposes a mutually reinforcing learning framework of decoupled degradation and diffusion enhancement, named MRLIE, which leverages style guidance from unpaired low-light images to generate pseudo-image pairs that are consistent with the target domain, thereby optimizing the latter diffusion enhancement network in a supervised manner. During the degradation process, the diffusion loss of fixed enhancement network serves as a evaluation metric for structure consistency and is combined with adversarial style loss to form the optimization objective for degradation network. Such loss design ensures that scene structure information is retained during the degradation process. During the enhancement process, the degradation network with frozen parameters continuously generates pseudo-paired low-/normal-light image pairs as training datas, thus the diffusion enhancement network could be progressively optimized. On the whole, the two processes are interdependent and could achieve cooperative improvement in terms of degradation realism and enhancement quality through iterative optimization. Additionally, we propose the Retinex-based decoupled degradation strategy for simulating the complex degradation in real low-light imaging, which ensures the color correction and noise suppression capabilities of latter diffusion enhancement network. Extensive experiments show that MRLIE can achieve promising results and better generality across various datasets.

RGB-Thermal semantic segmentation provides reliable support for intelligent traffic perception systems, such as road safety monitoring and autonomous driving perception, by fusing visible and thermal imaging modalities under adverse weather conditions and low-light environments at night. However, the scarcity of multimodal data and the high cost of annotations severely limit the generalization capability of traditional models. To address the core demands of urban scene segmentation, we propose a Hierarchical Matching Guided Feature Enhancement Network (HMFENet) tailored for few-shot learning. It tackles two major challenges: 1) scale diversity of traffic objects (e.g., vehicles and pedestrians) under limited labeled data, which significantly degrades segmentation accuracy; 2) information redundancy across multimodal features, which undermines the enhancement effect of the thermal modality on traffic object segmentation. HMFENet employs a hierarchical dense matching mechanism to establish multi-scale and multi-level feature alignment between query images and support samples. Additionally, it incorporates a mutual information minimization constraint to optimize cross-modal complementarity, thereby enhancing segmentation robustness in complex urban scenes. Experiments on the urban scene dataset, Tokyo Multi-Spectral-$4^{i}$demonstrate that the proposed method achieves state-of-the-art results: an improvement of 5.9% and 9.4% in mean mIoU for critical traffic objects under 1-shot and 5-shot settings, respectively, compared to baseline models. Furthermore, the complementary effect of the thermal modality contributes to a 2.5% improvement under the 1-shot setting. The proposed method provides a feasible solution for deploying multimodal traffic perception systems with low annotation costs. The source code is available athttps://github.com/Zhou-xy99/HMFENet.

Infrared image nonuniformity correction aims to remove the column-wise stripe noise. Most existing methods just consider stripe noise whereas failing to handle real captured nonuniformity, as directional characteristic of stripe is severely disrupted by random Gaussian noise. Moreover, deep learning-based methods proposed in recent years are blocked by limited receptive field thus cannot accurately distinguish vertical structure and vertical stripes. To address these issues, we propose a universal infrared image nonuniformity correction method based on stripe-aware attention network. We seek to improve the performance of our algorithm by first restoring the damaged stripe directional characteristics, then maximizing the utilization of the prior characteristics. On the one hand, we construct the two-stage framework, in which denoising network is firstly applied to eliminate Gaussian noise and preserve stripes as scene information. As a result, the prior directional characteristics are restored, thereby enhancing the ability of subsequent sub-network to perceive stripe noise. On the other hand, due to the distinct long-range pixel correlations of vertical structures and vertical textures, we introduce a column-wise stripe attention mechanism (CSA) that can capture long-range dependencies of target pixels in the vertical direction. This significantly improves the discriminative ability of algorithm towards vertical structures and stripes, with minimal computational cost. Extensive experiments show that the proposed method can achieve promising results and has better universality for different infrared scenarios.

Recent advances in deep learning-based methods have led to significant progress in the hyperspectral super-resolution (SR). However, the scarcity and the high dimension of data have hindered further development since deep models require sufficient data to learn stable patterns. Moreover, the huge domain differences between hyperspectral image (HSI) datasets pose a significant challenge in generalizability. To address these problems, we present a general hyperspectral SR framework via meta-transfer learning (MTL). We randomly sample various spectral ranges for SR tasks during MTL, allowing the model to accumulate diverse task experiences. Additionally, we implement a task schedule to gradually expand the number of bands, bridging the significant domain differences between datasets. By leveraging multiple datasets, we are able to achieve better performance and greater generalizability, making it applicable under various circumstances. Meanwhile, as a general framework, our scheme can be applied to existing methods to obtain performance improvements. In addition, we design an advanced network architecture based on the multifusion features to further improve the performance. Experiments demonstrate that our method not only achieves superior performance in both qualitative and quantitative terms but also can adapt robustly to a new and difficult sample, where few epochs can yield quite considerable results.

Enhancing low-light images improves both the visibility and quality of the images. Existing methods primarily focus on the enhancement process and heavily rely on the supervised learning strategy , where low/normal-light image pairs are used as the training dataset. In this paper, we propose a novel method called Learn Enhancement by Learning Degradation (LELD) to achieve efficient light adjustment and scene fidelity. We use a carefully designed degradation network (DNet) to guide the enhancement network (ENet). Specifically, the role of DNet is transforming normal-light images into low-light images. For better generalization ability , we employ an unsupervised learning strategy and a generative adversarial network framework. The training is totally dependent on unpaired datasets. Inspired by Retinex theory , we propose a fidelity loss to maintain color and detail during the degradation process . The ENet exhibits a straightforward architecture and achieves efficient enhancement. Experimental results demonstrate the advantages of our method over state-of-the-art methods in terms of visual quality and enhancement efficiency.

Deep learning (DL) has been widely utilized in image compressive sensing (CS) to enhance the quality and speed of reconstruction. The typical deep network for CS reconstruction comprises an initial reconstruction subnetwork, followed by a cascaded deep refinement reconstruction subnetworks. This paper introduces a new low-complexity image CS deep reconstruction framework, GSRCS, which leverages multi-image based deep super-resolution technology to better address the cost constraints of practical applications. The proposed initial reconstruction module generates multiple low-resolution images in parallel by grouping the input measurements, while a high-quality, high-resolution reconstructed image is produced through a multi-image deep super-resolution network. The theoretical derivation and experimental results demonstrate that this method significantly reduces system complexity in terms of parameters and floating-point arithmetic operations, while achieving competitive reconstruction performance compared to the state-of-the-arts. Specifically, the average number of parameters is reduced by over 63%, and the computational complexity is decreased by more than 88%.

With the lack of sufficient prior information, unsupervised hyperspectral unmixing (HU) has been a preprocessing step in the hyperspectral image (HSI) processing pipeline, which can provide the types of material and corresponding abundance information of HSI, to further provide assistance for downstream higher level semantic tasks to overcome the limitation caused by mixed pixels. However, the unmixing results obtained by current unsupervised HU methods are unstable and unprecise under the guidance of the least reconstruction error (RE), which have no consistency with the performance of high-level tasks. To solve this problem, this article takes the hyperspectral anomaly detection (HAD) as an entry point and proposes a novel algorithm based on deep clustering which can jointly perform HU and HAD in an end-to-end manner. A mutual feedback mechanism is formed between the upstream HU process and the downstream HAD process, and through joint optimization, both two tasks can achieve relatively good performances. However, the low dimensional abundance has a limited representation, which may lead to the increase of false alarm rate. To overcome this limitation, the principal components (PCs) of HSI are fused with the abundance to enhance the representation ability. Moreover, we use the reweighted reconstruction loss strategy to enhance the role of anomalies in the HU process. Experiments performed on several real datasets verify the rationality and superiority of the proposed UADNet algorithm.

The scarcity of hyperspectral images (HSIs) hinders the development of processing methods and downstream applications. HSI synthesis, which aims to generate realistic samples from the existing datasets, is undoubtedly a prospective and economical solution for the HSI data shortage problem. Inspired by the impressive performance of the diffusion model (DM) in image synthesis tasks, this article initiatively proposes an unmixing diffusion (UnmixDiff) model for high-quality HSI generation. The method starts with training an unmixing network to learn the distribution characteristic of objects (abundance). By incorporating the unmixing autoencoder into the DM, the UnmixDiff transforms the HSI generation into the abundance domain, which maintains the consistency of the generated spectral profile, reduces the computational complexity, and introduces a clear physical interpretation into the hyperspectral image synthesis tasks. After that, we construct a diffusion generation model in abundance space to generate realistic abundance maps. Instead of synthesizing original hyperspectral images, the proposed UnmixDiff synthesizes abundance maps to simulate objects’ distribution rather than superficial textures. In this way, realistic HSI samples are generated by mixing the synthesized abundance with the scene end-members. With the comparative experiments, the proposed method achieves state-of-the-art performance in HSI synthesis tasks, effectively alleviating the HSI data scarcity and supporting widespread HSI applications. The code is available athttps://github.com/yuyang95/UnmixingDM.

In this paper, we propose an effective method for mismatch removal, termed as graph neighborhood motion consensus, to address the feature matching problem which plays a pivotal role in various computer vision tasks. In our method, we convert each feature correspondence into a motion field sample and model it with the probabilistic graphical model (PGM). To differentiate mismatches from true matches, we firstly design a metric based on neighborhood topology consensus and neighborhood interaction to evaluate the correctness of each match. We also design a variance-based similarity search module to make the information used more reliable for better matching performance. To derive the solution of PGM, we build a model to transform the problem into an integer quadratic programming problem and obtain its closed-form solution with linear time complexity. Extensive experiments on general feature matching, fundamental matrix estimation and image registration tasks demonstrate that our proposed method can achieve superior performance over several state-of-the-art approaches.

Low-light image enhancement aims to recover normal-light images from the images captured under dim environments. Most existing methods could just improve the light appearance globally whereas failing to handle other degradation such as dense noise, color offset and extremely low-light. Moreover, unsupervised methods proposed in recent years lack reliable physical model as the basis, thus universality is greatly limited. To address these problems, we propose a novel low-light image enhancement method via Retinex-inline cycle-consistent generative adversarial network named Cycle-Retinex, whose training is totally dependent on unpaired datasets. Specifically, we organically combine Retinex theory with CycleGAN, by which we decouple low-light image enhancement task into two sub-tasks, i.e. illumination map enhancement and reflectance map restoration. Retinex theory helps CycleGAN simplify low-light image enhancement problem and CycleGAN provides synthetic paired images to guide the training of Retinex decomposition network. We further introduce a self-augmented method to address the color distortion and noise problem, thus making the network learn to enhance low-light images adaptively. Extensive experiments show that the proposed method can achieve promising results.

Hyperspectral unmixing (HU) based on linear mixing model (LMM) has received much attention in the remote sensing community over the past decades. However, many HU algorithms are based on the fixed endmember and do not fully exploit the inherent spatial characteristics of the hyperspectral image (HSI). In this letter, a new HU algorithm named GLRl1-GMM is proposed to solve these problems. We use Gaussian mixture model (GMM) to represent the endmember variability. Then, considering the local smoothness in the abundance map, a graph Laplacian regularization based onl1-norm (GLRl1) is embedded in the prior of abundance. Under the Bayesian framework, the objective density function leads to a maximum a posterior (MAP) problem, which can be solved by a generalized expectation-maximization (GEM) algotithm. Experiments on two real datasets demonstrates the effectiveness of proposed algorithm compared with other state-of-the-art methods. The source code is available at https://github.com/lwdinwhu/GMM-GLRl1.

With the development of modern weapons such as UAV swarms and multi-warhead missiles, infrared (IR) cluster small target detection technology has become increasingly important. However, the difficulty in characterizing cluster multi-targets leads to poor detection performance of existing methods. On the one hand, this paper proposes improved DBSCAN (IDBSCAN) to accurately extract the features of cluster multi-targets with unknown number and distribution. On the other hand, IDBSCAN-based difference measure (IDBSCAN-DM) is proposed, which fuses saliency and distribution features to further enhance cluster multi-targets. Specifically, we first design the multiscale sliding window to quickly extract candidate targets. Then, the IDBSCAN-based local window is constructed and IDBSCAN-DM is computed for better target enhancement and background suppression. Finally, adaptive threshold segmentation is performed on the IDBSCAN-DM map to detect real targets. Extensive comparative experiments demonstrate that the proposed method achieves better target enhancement and higher probability of detection.

Limited by the imaging paradigm, stripes is pervasive in remote sensing scenes, and its intensity, density, and periodicity differ dramatically among different imaging systems. Worse, it always co-exists with random noises caused by unstable imaging condition. However, current destriping methods are victim to undue ideal assumptions and fail to accurately eliminate stripes against diverse practical degradation, yielding excessive or inadequate destriping results. This study proposes a progressive hyperspectral destriping method with an adaptive frequency focus for accurate destriping and delicate restoration. Specifically, a hierarchical decomposition and reconstruction framework based on progressive wavelet learning encodes the degraded input to the frequency domain with smaller scales, easing the difficulty of restoration. Then, to avoid excessive or insufficient destriping, we devote specific efforts to finely separating noise and preserving details in the high-frequency domain. First, we devise a gradient-aware frequency attention block based on the prominent unidirectional pattern of stripes, empowering to adaptively assign weights according to their sensitivity to the spatial gradient. Second, we design a focal high-frequency loss item that is dynamically scaled according to feature distance in the high-frequency domain, profiting in identifying and preserving details. Extensive experiments conducted on data with synthetic stripes and realistic satellite scenes validate the superiority of the proposed method over the current state-of-the-art methods. The code is available at https://github.com/EtPan/PHID.

Multispectral image (MSI) and hyperspectral image (HSI) fusion can combine the best of both worlds to produce images with both high spatial and spectral resolution. In this paper, we have designed a network for fusing MSIs and HSIs, called DSPNet. On the one hand, in order to ensure the accuracy of the spectral dimension, i.e. spectral fidelity, we designed the spectral pyramid (SpePy) module and the multiscale spectral information fusion (MLSIF) module. The former extracts the multiscale local spectral information that captures the subtle spectral details and variations between different spectra. The latter establishes long-range dependency in the spectral dimension through the spectral-wise multi-head hybrid-attention (S-MHA) mechanism, thus enabling the network to focus on the local spectral information needed to recover the spectral details. On the other hand, to address the spatial information of MSIs, we designed the spatial pyramid (SpaPy) module. The SpaPy module can extract the non-local spatial information of MSIs at different scales, which enables the network to adapt to different remote-sensing scenes. Experiments performed on simulated and real data demonstrate the superiority of our method over the state-of-the-art methods both qualitatively and quantitatively.

For robust feature matching, a popular and particularly effective method is to recover smooth functions from the data to differentiate the true correspondences (inliers) from false correspondences (outliers). In the existing works, the well-established regularization theory has been extensively studied and exploited to estimate the functions while controlling its complexity to enforce the smoothness constraint, which has shown prominent advantages in this task. However, despite the theoretical optimality properties, the high complexities in both time and space are induced and become the main obstacle of their application. In this article, we propose a novel method for multivariate regression and point matching, which exploits the sparsity structure of smooth functions. Specifically, we use compact Fourier bases for constructing the function, which inherently allows a coarse-to-fine representation. The smoothness constraint can be explicitly imposed by adopting a few low-frequency bases for representation, resulting in reduced computational complexities of the induced multivariate regression algorithm. To cope with potential gross outliers, we formulate the learning problem into a Bayesian framework with latent variables indicating the inliers and outliers and a mixture model accounting for the distribution of data, where a fast expectation-maximization solution can be derived. Extensive experiments are conducted on synthetic data and real-world image matching, and point set registration datasets, which demonstrates the advantages of our method against the current state-of-the-art methods in terms of both scalability and robustness.

Spectral unmixing (SU), which refers to extracting basic features (i.e., endmembers) at the subpixel level and calculating the corresponding proportion (i.e., abundances), has become a major preprocessing technique for the hyperspectral image analysis. Since the unmixing procedure can be explained as finding a set of low-dimensional representations that reconstruct the data with their corresponding bases, autoencoders (AEs) have been effectively designed to address unsupervised SU problems. However, their ability to exploit the prior properties remains limited, and noise and initialization conditions will greatly affect the performance of unmixing. In this article, we propose a novel technique network for unsupervised unmixing which is based on the adversarial AE, termed as adversarial autoencoder network (AAENet), to address the above problems. First, the image to be unmixed is assumed to be partitioned into homogeneous regions. Then, considering the spatial correlation between local pixels, the pixels in the same region are assumed to share the same statistical properties (means and covariances) and abundance can be modeled to follow an appropriate prior distribution. Then the adversarial training procedure is adapted to transfer the spatial information into the network. By matching the aggregated posterior of the abundance with a certain prior distribution to correct the weight of unmixing, the proposed AAENet exhibits a more accurate and interpretable unmixing performance. Compared with the traditional AE method, our approach can greatly enhance the performance and robustness of the model by using the adversarial procedure and adding the abundance prior to the framework. The experiments on both the simulated and real hyperspectral data demonstrate that the proposed algorithm can outperform the other state-of-the-art methods.

Hyperspectral images (HSIs) with both spatial and spectral information have found broad applications. Since most cameras have narrow viewing angle, generating panoramic images is essential to show a large-range view of the environment. So far, there are few studies on HSI stitching, and the stitching result still suffers from some problems, such as blurring and ghosting, geometric misalignment, visible seam, and spectral distortion. Hence, to address the above disadvantages, we propose a novel HSI stitching strategy using optimal seamline detection approach in this letter. First, we use a fast and robust seam estimation method to determine the seamline in each single band of HSI. This method works in RGB images, and we have modified it to be used in a single-band gray-scale image of HSI. Then, to guarantee the integrity of spatial and spectral information of hundreds of bands of HSI, we propose to apply the structural similarity (SSIM) index to select the optimal one among all band candidate seamlines and use the selected optimal seamline to stitch all the remaining bands. The experimental results demonstrate that our proposed approach outperforms traditional HSI stitching approach in both spatial and spectral performances.

Effective detection of small infrared (IR) targets buried in complex backgrounds and heavy noise plays an important role in IR search and track (IRST) systems. In this letter, an IR small target detection method called adaptive scale patch-based contrast measure (ASPCM) is proposed. Compared with existing detection methods based on the human visual system (HVS), our method can estimate the size of the potential target at each location. According to the estimated size, the local contrast between the target and the background is greatly enhanced, and the background clutter can be further suppressed. Experimental results on four real sequences of different complex background demonstrate that the proposed method can detect targets effectively and efficiently, even in the case of complex backgrounds and heavy noises.

Local contrast measure (LCM) proves effective in infrared (IR) small target detection. Existing LCM-based methods focus on mining local features of small targets to improve detection performance. As a result, they struggle to reduce false alarms while maintaining detection rates, especially with high-contrast background interference. To address this issue, this letter proposes global sparsity-weighted local contrast measure (GSWLCM), which fuses both global and local features of small targets. First, robust local contrast measure (RLCM) is proposed to remove low-contrast backgrounds and extract candidate targets. Then, to suppress high-contrast backgrounds, we customize the random walker (RW) to extract candidate target pixels, construct the global histogram and calculate global sparsity. Finally, GSWLCM fusing global and local features is calculated and the target is detected by adaptive threshold segmentation. Extensive experimental results show that the proposed method is effective in suppressing high-contrast backgrounds and has better detection performance than several state-of-the-art methods.

Anomaly detection of hyperspectral data has been gaining particular attention for its ability in detecting targets in an unsupervised manner. Autoencoder (AE), together with its variants can not only extract intrinsic features automatically but also detect anomalies that differ dramatically from others. Many AE-driven algorithms are, thus, proposed for anomaly detection in hyperspectral imagery (HSI), but they suffer from two problems: 1) when there exist anomalies in the training set, AE can generalize so well that it can also learn the abnormal patterns well, thereby reducing the ability to distinguish anomalies from the background and 2) geometric structure among samples are lost in latent space of AE, which is vital in hyperspectral anomaly detection. To tackle these problems, we propose a robust anomaly detector based on the AE framework, named robust graph AE (RGAE) detector, in this article. To be specific, we propose a robust AE framework with$\ell _{2,1}$-norm that is robust to noise and anomalies during training. Meanwhile, we embed a superpixel segmentation-based graph regularization term (SuperGraph) into AE. This strategy can preserve the geometric structure and the local spatial consistency of HSI simultaneously and also effectively reduce the searching space and execution time for each pixel. Extensive experiments are conducted on five datasets, and the results demonstrate that our method has a better detection performance, after comparing with other state-of-the-art hyperspectral anomaly detectors.

Spectral unmixing is a major technique for the further development of hyperspectral analysis. It aims to determine the corresponding proportion (fractional abundance) of the basic spectral signatures (endmembers) blindly at the subpixel level. Recently, the learning-based method has received much attention in hyperspectral unmixing, and autoencoders have been effectively designed to solve the unsupervised scenarios of unmixing. However, their ability to extract physically meaningful endmembers remains limited, and the performance has not been satisfactory. In this article, we propose a novel two-stream network, termed TANet, to address the above problems. The network consists of a two-stream architecture. First, superpixel segmentation is adopted as preprocessing to extract the endmember bundles from the image. Then, the first stream learns a mapping from the pseudopure pixels to their corresponding abundances. The second stream is conducting the same untied-weighted autoencoder to minimize reconstruction errors from the original pixel data. By learning from the pure or nearly pure candidate pixels to correct the weights of unmixing, the proposed TANet exhibits a more accurate and interpretable unmixing performance. Extensive experiments on both synthetic and real hyperspectral data demonstrate that the proposed TANet can outperform the other state-of-the-art approaches.

This article presents a novel end-to-end model based on encoder–decoder architecture for hyperspectral image (HSI) denoising, named spatial-spectral quasi-attention recurrent network, denoted as SQAD. The central goal of this work is to incorporate the intrinsic properties of HSI noise to construct a practical feature extraction module while maintaining high-quality spatial and spectral information. Accordingly, we first design a spatial-spectral quasi-recurrent attention unit (QARU) to address that issue. QARU is the basic building block in our model, consisting of spatial component and spectral component, and each of them involves a two-step calculation. Remarkably, the quasi-recurrent pooling function in the spectral component could explore the relevance of spatial features in the spectral domain. The spectral attention calculation could strengthen the correlation between adjacent spectra and provide the intrinsic properties of HSI noise distribution in the spectral dimension. Apart from this, we also design a unique skip connection consisting of channelwise concatenation and transition block in our model to convey the detailed information and promote the fusion of the low-level features with the high-level ones. Such a design helps maintain better structural characteristics, and spatial and spectral fidelities when reconstructing the clean HSI. Qualitative and quantitative experiments are performed on publicly available datasets. The results demonstrate that SQAD outperforms the state-of-the-art methods of visual effect and objective evaluation metrics.

Capturing accurate multiscale semantic information from the images is of great importance for high-quality semantic segmentation. Over the past years, a large number of methods attempt to improve the multiscale information capturing ability of the networks via various means. However, these methods always suffer unsatisfactory efficiency (e.g., speed or accuracy) on the images that include a large number of small-scale objects, for example, aerial images. In this article, we propose a new network named cross fusion net (CF-Net) for fast and effective extraction of the multiscale semantic information, especially for small-scale semantic information. In particular, the proposed CF-Net can capture more accurate small-scale semantic information from two aspects. On the one hand, we develop a channel attention refinement block to select the informative features. On the other hand, we propose a cross fusion block to enlarge the receptive field of the low-level feature maps. As a result, the network can encode more accurate semantic information from the small-scale objects, and the segmentation accuracy of the small-scale objects is improved accordingly. We have compared the proposed CF-Net with several state-of-the-art semantic segmentation methods on two popular aerial image segmentation data sets. Experimental results reveal that the average$F_{1}$score gain brought by our CF-Net is about 0.43% and the$F_{1}$score gain of the small-scale objects (e.g., cars) is about 2.61%. In addition, our CF-Net has the fastest inference speed, which proves its superiority in the aerial scenes. Our code will be released at:https://github.com/pcl111/CF-Net.

In this article, an array thermal camera equipment is developed to capture multiple infrared images with spatial and parallax information. Based on the captured images, an end-to-end method called spatial–parallax prior network (SPPN) is proposed. Specifically, we design a spatial–parallax prior block with two symmetric branches to extract spatial and parallax features in an interactive guidance manner. Then, to effectively integrate spatial and parallax features, we introduce a channel attention mechanism to enable the network to focus on and fuse the most useful information adaptively. In this way, spatial and parallax information can be fully utilized without any explicit alignment operation. Finally, considering the scarcity and poor quality of infrared training data, we leverage transfer learning to better train the network. Extensive experimental results demonstrate that the proposed SPPN consistently outperforms the current state-of-the-art methods, providing a highly effective and scalable solution for the improvement of infrared image quality.

Autoencoder can not only extract features in an unsupervised manner, but also selects samples out that differs significantly from others. However, autoencoder is sensitive to noise and anomalies during training, and the relationships between pixels are discarded. In order to tackle these problems, we propose a robust graph autoencoder (RGAE) for hyperspectral anomaly detection. To be specific, we first redesign the objective function to encourage the network more robust to noise and anomalies. Meanwhile, a superpixel segmentation-based graph regularization term (SuperGraph) is incorporated into AE to preserve the geometric structure and spatial information simultaneously. Experiments with three real data sets are conducted to evaluate the performance, and the detection results demonstrate that our method outperforms other state-of-the-art hyperspectral anomaly detectors.

Recently, the learning-based method has received much attention in the unsupervised hyperspectral unmixing, yet their ability to extract physically meaningful endmembers remains limited and the performance has not been satisfactory. In this paper, we propose a novel two-stream Dirichlet-net, termed as uTDN, to address the above problems. The weight-sharing architecture makes it possible to transfer the intrinsic properties of the endmembers during the process of unmixing, which can help to correct the network converging towards a more accurate and interpretable unmixing solution. Besides, the stick-breaking process is adopted to encourage the latent representation to follow a Dirichlet distribution, where the physical property of the estimated abundance can be naturally incorporated. Extensive experiments on both synthetic and real hyperspectral data demonstrate that the proposed uTDN can outperform the other state-of-the-art approaches.

Since CapsNet [1] shattered all previous records of algorithms for image recognition, the capsule's conception has attracted bright attention. It interprets an object by the geometrical arrangement of parts. We think it can be transferred to hyperspectral images. In a hyperspectral data cube, each pixel spectrum can be regarded as a continuous curve representing its inherent properties. In the spatial domain, there are various spatial distributions in different positionsand there is usually a specific structural relationship between adjacently distributed categories. Based on HSI data's aforementioned structural characteristics, combined with the stacked capsule autoencoder, we propose our model to achieve an unsupervised HSI classification. In our model, the ConvLSTM is employed to discover part capsules of HSI, and we utilize Set Transformer to encode relations among all parts and indicate object capsules. The decoders of both phases use Gaussian mixture models to reconstruct specific information. Experimental results of the Pavia Center dataset show the exceptional of our model.

Anomaly detection (AD) has become a hotspot in hyperspectral imagery (HSI) processing due to its advantage in detecting potential targets without prior knowledge, and a variety of algorithms are proposed for a better performance. However, they usually either fail to extract intrinsic features underlying HSIs, or suffer from the contamination of noise and anomalies. To address these problems, we propose a new anomaly detector by integrating fractional Fourier transform (FrFT) with low rank and sparse matrix decomposition (LRaSMD). First, distinctive features of HSI data are extracted via FrFT. Then, row-constrained LRaSMD (RC-LRaSMD), which is more practical and stable than the traditional LRaSMD, is employed to separate background from noise and anomalies. Finally, we implement an atom-selection strategy to construct the background covariance matrix for detection. The experimental results with several HSI data sets demonstrate satisfying detection performance compared with other state-of-the-art detectors.

Effective and efficient infrared (IR) small target detection is essential for IR search and tracking (IRST) systems. The current methods have some limitations in background suppression or detection of targets close to each other. In this letter, a double-neighborhood gradient method (DNGM) is proposed. First, a new technology of the tri-layer sliding window is designed to measure the double-neighborhood gradient. Then, the DNGM is obtained by multiplying the double-neighborhood gradient. In this way, even the sizes of the targets may vary, ranging from 2 ×1 to 9 ×9 pixels, the target can be better highlighted under a fixed scale, and background interference can be suppressed. Finally, the target is segmented from the DNGM salience map by an adaptive threshold. Experiments illustrate that the proposed method can avoid the “expansion effect” of the traditional multiscale human vision system (HVS) method and can accurately detect multiple targets close to each other. Besides, the proposed method is more robust and real-time than the existing methods.

Abstract In this paper, a novel multi-exposure image fusion method based on generative adversarial networks (termed as GANFuse) is presented. Conventional multi-exposure image fusion methods improve their fusion performance by designing sophisticated activity-level measurement and fusion rules. However, these methods have a limited success in complex fusion tasks. Inspired by the recent FusionGAN which firstly utilizes generative adversarial networks (GAN) to fuse infrared and visible images and achieves promising performance, we improve its architecture and customize it in the task of extreme exposure image fusion. To be specific, in order to keep content of extreme exposure image pairs in the fused image, we increase the number of discriminators differentiating between fused image and extreme exposure image pairs. While, a generator network is trained to generate fused images. Through the adversarial relationship between generator and discriminators, the fused image will contain more information from extreme exposure image pairs. Thus, this relationship can realize better performance of fusion. In addition, the method we proposed is an end-to-end and unsupervised learning model, which can avoid designing hand-crafted features and does not require a number of ground truth images for training. We conduct qualitative and quantitative experiments on a public dataset, and the experimental result shows that the proposed model demonstrates better fusion ability than existing multi-exposure image fusion methods in both visual effect and evaluation metrics.

Gaussian mixture model (GMM) can estimate not only the abundances and distribution parameters but also distinct end-member set for each pixel. However, the traditional GMM unmixing model only has proper smoothness and sparsity prior constraints on the abundances and thus cannot excavate the local spatial information in hyperspectral image (HSI). Thus, we propose a new unmixing method with superpixel segmentation (SS) and low-rank representation (LRR) based on GMM called GMM-SS-LRR, which can consider the local spatial correlation of HSI. First, we adopt the principal component analysis (PCA) to obtain the first principal component of HSI, which contains the most information for the entire HSI. Then, we adopt the SS in the first principal component of HSI to obtain the homogeneous regions, and the abundances in each homogeneous region have the underlying low-rank property. Finally, we unmix the pixels in each homogeneous region of HSI depending on the low-rank property of abundances. Experiments on synthetic datasets and real H-SIs demonstrate that the proposed GMM-SS-LRR is efficient compared with other current popular methods.

Neural networks have been successfully used to extract deep features for many hyperspectral tasks. In this study, we propose a tiny effective model based on gate recurrent unit (GRU) with spectral-spatial information for hyperspectral image classification. In our method, the core GRU cell can learn interspectral correlations within an entirely continuous spectrum input, and spatial information is the initial state of this GRU cell as a prior. Experimental results demonstrate that our method can fully utilize spectral and spatial information to obtain competitive performance.

Deep neural networks have been successfully applied to extracting deep features for many hyperspectral tasks. Attention mechanism has been widely used in computer vision, inspired by this, we have designed a joint attention network for spectral-spatial classification of hyperspectral image. In our method, recurrent neural network (RNN) with attention can learn inner spectral correlations within a continuous spectrum, convolutional neural network (CNN) with attention is designed to focus on saliency features and spatial dependency in the neighbor regions. Experimental results demonstrate that our method can fully utilize spectral and spatial information to obtain competitive performance.

Recently, collaborative representation has received much attention in the hyperspectral image (HSI) classification due to its simplicity and effectiveness. However, the existing collaborative representation-based HSI classification methods ignore the correlation among different classes. To overcome this problem, we propose a discriminative kernel collaborative representation and Tikhonov regularization method (DKCRT) for HSI classification, which can make the kernel collaborative representation of different classes to be more discriminative. Specifically, the kernel trick is adopted to map the original HSI into a high space to improve the class separability. Besides, distance-weighted kernel Tikhonov regularization is adopted to enforce these training samples to have large representation coefficients, which are similar to the test sample in the high-dimensional feature space. Moreover, we add a discriminative regularization term to further enhance the separability of different classes, which can take the correlation among different classes into consideration. Furthermore, to take the spatial information of HSI into consideration, we extend the DKCRT to a joint version named JDKCRT. Experiments on real HSIs demonstrate the efficiency of the proposed DKCRT and JDKCRT.

In this paper, we propose a probabilistic method for feature matching of near-duplicate images undergoing non-rigid transformations. We start by creating a set of putative correspondences based on the feature similarity, and then focus on removing outliers from the putative set and estimating the transformation as well. This is formulated as a maximum likelihood estimation of a Bayesian model with latent variables indicating whether matches in the putative set are inliers or outliers. We impose the non-parametric global geometrical constraints on the correspondence using Tikhonov regularizers in a reproducing kernel Hilbert space. We also introduce a local geometrical constraint to preserve local structures among neighboring feature points. The problem is solved by using the Expectation Maximization algorithm, and the closed-form solution of the transformation is derived in the maximization step. Moreover, a fast implementation based on sparse approximation is given which reduces the method computation complexity to linearithmic without performance sacrifice. Extensive experiments on real near-duplicate images for both feature matching and image retrieval demonstrate accurate results of the proposed method which outperforms current state-of-the-art methods, especially in case of severe outliers.

Sparse unmixing (SU) of hyperspectral data have recently received particular attention for analyzing remote sensing images, which aims at finding the optimal subset of signatures to best model the mixed pixel in the scene. However, most SU methods are based on the commonly admitted linear mixing model, which ignores the possible nonlinear effects (i.e., nonlinearity), and the nonlinearity is merely treated as outlier. Besides, the traditional SU algorithms often adopt the$\ell _{2}$norm loss function, which makes them sensitive to noises and outliers. In this paper, we propose a robust SU (RSU) method with$\ell _{2,1}$norm loss function, which is robust for noises and outliers. Then, the RSU can be solved by the alternative direction method of multipliers. Finally, the experiments on both synthetic data sets and real hyperspectral images demonstrate that the proposed RSU is efficient for solving the hyperspectral SU problem compared with the state-of-the-art algorithms.

Recently, low-rank representation (LRR) based methods have been used for hyperspectral image (HSI) denoising, which can simultaneously remove different types of noise: Gaussian noise, impulse noise, dead lines, and so on. However, the LRR based method does not make full use of the spatial information in HSI. In this paper, we integrate the superpixel segmentation (SS) into the LRR, and propose a novel denoising method named SS-LRR. We first use the principle component analysis (PCA) to obtain the first principle component of HSI. Then the superpixel segmentation is adopted to the first principle component of HSI to get homogeneous regions. Finally, we employ the LRR to each homogeneous region of HSI, which enable us to simultaneously remove all the above mentioned mixed noise. Extensive experiments on both simulated and real hyperspectral images demonstrate that the proposed SS-LRR is efficient for HSI denoising.

In this paper, we propose a novel formulation for building accurate pixel-wise alignments between remote sensing images under non-rigid distortions. Our formulation involves two variables: the first is a discrete displacement flow field similar to optical flow which controls the pixel-wise correspondence and allows piecewise smoothness, while the second is a continuous spatial transformation which fits for a few confidential sparse feature correspondences. An additional term is introduced to ensure the coherence between the two variables, and the continuous spatial transformation plays a role of anchor for optimizing the discrete displacement flow field. Experiments on real remote sensing images demonstrate that our approach greatly outperforms state-of-the-art methods.

Sparse unmixing (SU) of hyperspectral data has recently received particular attention for analyzing remote sensing images, which aims at finding the optimal subset of signatures to best model the mixed pixel in the scene. However, most SU methods are based on the commonly admitted linear mixing model (LMM), which ignores the possible nonlinear effects (i.e. nonlinearity), and the nonlinearity is merely treated as outlier. Besides, the traditional SU algorithms often adopt the ℒ2norm loss function, which makes them sensitive to noises and outliers. In this paper, we propose a robust sparse unmixing (RSU) method with ℒ2,1norm loss function, which is robust for noises and outliers. Then, the RSU can be solved by the alternative direction method of multipliers (ADMM). Finally, experiments on synthetic datasets demonstrate that the proposed RSU is efficient for solving the hyperspectral SU problem compared with state-of-the-art algorithms.

Recently, sparse unmixing (SU) of hyperspectral data has received particular attention for analyzing remote sensing images. However, most of SU methods are based on the commonly admitted linear mixing model (LMM), which ignores the possible nonlinear effects (i.e. nonlinearity). In this paper, we proposed a new method named robust collaborative sparse regression (RCSR) for hyperspectral unmixing, which is based on the robust LMM (rLMM). The rLMM takes the nonlinearity into consideration, and the nonlinearity is merely treated as outlier, which has the underlying sparse property. The RCSR takes the collaborative sparse property of the abundance and sparsely distributed additive property of the outlier into consideration, which can be formed as a robust joint sparse regression problem. Experiments on synthetic datasets demonstrate that the proposed RCSR is efficient for solving the hyperspectral SU problem compared with other five state-of-the-art algorithms.

Infrared (IR) small target detection with high detection rate, low false alarm rate, and multiscale detection ability is a challenging task since raw IR images usually have low contrast and complex background. In recent years, robust human visual system (HVS) properties have been introduced into the IR small target detection field. However, existing algorithms based on HVS, such as difference of Gaussians (DoG) filters, are sensitive to not only real small targets but also background edges, which results in a high false alarm rate. In this letter, the difference of Gabor (DoGb) filters is proposed and improved (IDoGb), which is an extension of DoG but is sensitive to orientations and can better suppress the complex background edges, then achieves a lower false alarm rate. In addition, multiscale detection can be also achieved. Experimental results show that the IDoGb filter produces less false alarms at the same detection rate, while consuming only about 0.1 s for a single frame.

The generalized bilinear model (GBM) has been widely used for the nonlinear unmixing of hyperspectral images, and traditional GBM solvers include the Bayesian algorithm, the gradient descent algorithm, the semi-nonnegative-matrix-factorization algorithm, etc. However, they suffer from one of the following problems: high computational cost, sensitive to initialization, and the pixelwise algorithm hinders us from applying to large hyperspectral images. In this letter, we apply Nesterov's optimal gradient method to solve the least-square problem under the bound constraint, which is named as the bound projected optimal gradient method (BPOGM). The BPOGM can achieve the optimal convergence rate of$O(1/k^{2})$, with$k$denoting the number of iterations in BPOGM. We further apply the BPOGM to solve the GBM-based unmixing problem. Experiments on both synthetic data sets and real hyperspectral images demonstrate that the BPOGM is efficient for solving the GBM-based unmixing problem.

Recently, the sparse representation-based classification (SRC) methods have been successfully used for the classification of hyperspectral imagery, which relies on the underlying assumption that a hyperspectral pixel can be sparsely represented by a linear combination of a few training samples among the whole training dictionary. However, the SRC-based methods ignore the sparse representation residuals (i.e., outliers), which may make the SRC not robust for outliers in practice. To overcome this problem, we propose a robust SRC (RSRC) method which can handle outliers. Moreover, we extend the RSRC to the joint robust sparsity model named JRSRC, where pixels in a small neighborhood around the test pixel are simultaneously represented by linear combinations of a few training samples and outliers. The JRSRC can also deal with outliers in hyperspectral classification. Experiments on real hyperspectral images demonstrate that the proposed RSC and JRSRC have better performances than the orthogonal matching pursuit (OMP) and simultaneous OMP, respectively. Moreover, the JRSRC outperforms some other popular classifiers.

In this letter, we propose a simple yet effective approach, named Progressive Vector Field Consensus (PVFC), for addressing the problem of finding more true feature correspondences between images. The key idea is to progressively perform feature matching based on Vector Field Consensus, and hence greatly boost the number of true matches as well as avoid false matches. More specifically, it uses matching results on a small putative correspondence set with high inlier ratio to guide the matching on a large putative correspondence set which probably covers the whole true correspondences. We model the transformation between images in a reproducing kernel Hilbert space, and a sparse approximation is applied to the transformation to avoid high computational complexity. Our results quantitatively show that our PVFC outperforms state-of-the-art methods, both in accuracy and in efficiency. Moreover, the progressive framework is general and can be applied to other cases for robust estimation.

In this letter, we propose an improved locally linear embedding (LLE) method based on robust spatial information (named RSLLE) for hyperspectral data dimensionality reduction. It explores and takes full account of the complexity of the spatial information for LLE. In RSLLE, when searching for spectral neighbors, a kind of spectral-spatial distance is used instead of the distance between two individual target pixels. Then, two additional steps, i.e., spatial neighbor sorting and spatial neighbor filtering, are presented to ensure the robustness of the spectral-spatial distance. Two classification experimental results indicate that the proposed RSLLE method significantly improves the performance when compared with other LLE methods, and the classification accuracy is competitive compared with other latest spectral-spatial classification methods.

Robust human visual system (HVS) properties can effectively improve the infrared (IR) small target detection capabilities, such as detection rate, false alarm rate, speed, etc. However, current algorithms based on HVS usually improve one or two of the aforementioned detection capabilities while sacrificing the others. In this letter, a robust IR small target detection algorithm based on HVS is proposed to pursue good performance in detection rate, false alarm rate, and speed simultaneously. First, an HVS size-adaptation process is used, and the IR image after preprocessing is divided into subblocks to improve detection speed. Then, based on HVS contrast mechanism, the improved local contrast measure, which can improve detection rate and reduce false alarm rate, is proposed to calculate the saliency map, and a threshold operation along with a rapid traversal mechanism based on HVS attention shift mechanism is used to get the target subblocks quickly. Experimental results show the proposed algorithm has good robustness and efficiency for real IR small target detection applications.