A high-resolution (HR) hyperspectral imaging at video rates can be achieved by fusing a multi-band low-resolution (LR) mosaiced image and a single-band HR panchromatic (PAN) image in a single shot within a short time. However, the current fusion methods always suffer from a spatial or spectral distortion. To alleviate this, we propose an equivariant Bayesian variational inference framework. Specifically, we decompose an HR hyperspectral image (HSI) into the principal component and sparsity residual, which are modeled as latent variables with Gaussian priors. Each component is estimated via a shared deep neural network (DNN) under a variational inference framework, leveraging the shared spatial structures to enhance parameter efficiency and reconstruction accuracy. Additionally, to tackle the challenge of unavailable ground truth in real-world scenarios, we integrate the equivariant imaging (EI) prior with the Bayesian framework. By enforcing the consistency between the transformed fusion result and the re-inference output, this strategy enables the network to learn beyond the range space. Furthermore, we propose to utilize the learnable degradation functions derived from the physical imaging model to enable the proposed framework, which ensures an enhanced performance by posing plausible constraints on parameters of the degradation functions. Specifically, we explicitly model the point spread function (PSF) and spectral response function (SRF) with learnable parameters and impose non-negativity and sum-to-one constraints. Extensive experiments conducted on both simulated and real-world datasets demonstrate the effectiveness of the proposed framework, paving the way for HSI computational imaging.

Semisupervised deep facial expression recognition (FER) tries to learn better representations from both labeled and unlabeled data to avoid the huge manual labeling cost in the supervised methods. Due to the imbalanced distribution of facial expression data and the varying difficulty in recognizing different expressions, conventional semisupervised FER methods that directly discard low-confidence unlabeled samples may exacerbate the deficiency of the minority classes, and trap the model into the Matthew effect. Therefore, an adaptive contrastive learning-based semisupervised deep facial expression recognition framework is proposed to utilize both high and low confidence unlabeled samples by two different contrastive learning strategies according to an adaptive threshold. The threshold varies from different classes and iterations, considering both the difficulty of different classes and current learning state of the model during the training process. After the adaptive thresholding, the class-aware contrastive learning is applied to the high confidence samples, while the distance-aware contrastive learning is to the low confidence ones. Our proposed method achieves the state-of-the-art performance through extensive experiments on two widely-used datasets RAF-DB and AffectNet.

Natural disasters pose a threat to the safety of human life and buildings. Rapid and accurate building damage assessment (BDA) on remote sensing images is crucial for disaster response and recovery. However, most methods are constructed on the ideal co-registered bitemporal remote sensing images, neglecting the misalignment in practice. In this paper, we propose a novel building damage assessment method, termed FlowMamba, which can effectively handle the offset between the pre- and post-disaster images in BDA task. Specifically, a vision mamba backbone with four stages is utilized to extract multi scale features from the pre- and post-disaster images. Then, a differential optical flow alignment module is designed to estimate shift matrix to align pre- and post-disaster features. Furthermore, a category distance-aware loss function is tailored for the BDA task, which replaces fixed binary values of the penalty factors in to soft values of inter-class distance. Extensive experiments on the xBD dataset, the BRIGHT dataset and four out-of-distribution disaster scenarios validate the robustness and effectiveness of the proposed FlowMamba. Our code is available at https://github.com/flying318/FlowMamba.

Multispectral filter array (MSFA) cameras enable snapshot spectral data acquisition but require effective demosaicing algorithms to recover spatial-spectral information. Existing methods either rely on iterative optimization with limited high-frequency recovery, or require large-scale paired data and sophisticated network architectures that introduce considerable computational overhead. Moreover, insufficient frequency awareness in both spectral image initialization and the training process often leads to biased representations and unstable optimization under unsupervised settings. To address these issues, we propose a lightweight unsupervised spectral demosaicing framework, termed Frequency-Guided Lightweight Network (FGLN). First, we propose the Frequency-Parameterized Convolution (FPC) as the spectral initialization strategy, enhancing frequency coverage by explicitly embedding frequency-aware parameterization into convolutional weights. Second, we develop a frequency-aware unsupervised training framework based on equivariant imaging to regulate the optimization dynamics and suppress degenerate solutions. Finally, the Lightweight Spectral Demosaicing Unit (LSDU) improves reconstruction quality while maintaining low computational complexity. Extensive experiments on both synthetic and real-world datasets demonstrate that FGLN achieves competitive performance compared with state-of-the-art methods. The code will be uploaded at https://github.com/Matsuri247/FGLN.

Visible and infrared light images reflect object characteristics in different aspects, which has attracted much attention for object detection in recent years. Nevertheless, the existing multimodal detection networks may fail in the absence of modality. In order to address this problem, a new cross-modal knowledge distillation network (CMKD-net) is proposed for oriented object detection in visible and infrared images. In brief, a teacher-student (T-S) learning network is constructed, where the T-network aims to learn a discriminative feature representation from multimodal images and then guides the S-network training with incomplete modality. Here, multi-dimensional feature distillation (MDFD) and inter-instance relation distillation (IIRD) are designed for cross-modal knowledge propagation. Specifically, the MDFD considers pushing the T-S networks to learn a similar data distribution and feature representation through channel-spatial dimensional feature consistency constraints. The IIRD contributes to retaining the relation structure between individual targets in multimodal images via inter-instance relation modeling and similarity distance measurement. Moreover, to avoid the bias of feature extraction caused by discrete quantization in traditional pooling operations, a rotation-adaptive RoI Pooling (RA-RoI Pooling) is introduced by calculating the continuous double integral within each bin of oriented objects. Ablation experiments and comparison experiments on the VEDAI and DroneVehicle datasets can demonstrate the effectiveness of the proposed CMKD-net.

Multi-source remote sensing data classification refers to the process of categorizing ground objects by integrating complementary strengths of multiple remote sensing data, such as hyperspectral image (HSI), light detection and ranging (LiDAR) and synthetic aperture radar (SAR) data. However, current Mamba-based multisource remote sensing data classification approaches rely on fixed scanning patterns that are inadequate in characterizing spectral-spatial information. Additionally, current fusion techniques adopt concatenation or attention-based fusion rules without considering the complementary characteristics between different modalities. To address these limitations, we propose a spectral-spatial dynamic scan Mamba (SDSM) for multi-source remote sensing data classification. Specifically, a dynamic scan Mamba network is proposed to extract the spectral-spatial features of multi-source remote sensing data, in which a dynamic scan module is designed to adaptively capture the important spatial and spectral information. Furthermore, a bidirectional cross-modal fusion rule is proposed to merge the extracted features, in which a global-local frequency feature extraction module is designed to extract the salient structural features of multi-source remote sensing data as clues to guide heterogeneous feature fusion. Comprehensive experiments on four multi-source remote sensing datasets, i.e., MUUFL, Augsburg, Italy and Yellow River, demonstrate that the proposed method outperforms other state-of-the-art methods with respect to quantitative and qualitative results. The code of this article is available at https://github.com/PuhongDuan/SDSM.

Existing mosaic-based snapshot hyperspectral imaging systems struggle to capture high resolution (HR) hyperspectral image (HSI), limiting its application. Fusing a low resolution (LR) mosaiced image with an HR panchromatic (PAN) image serves as a feasible solution to obtain the HR HSI. Therefore, we propose a dual-sensor based HSI imaging system, combining a $4\times 4$ spectral filter array (SFA) mosaiced image sensor with a co-aligned PAN image sensor to provide complementary spatial-spectral information. To reconstruct HR HSI, we propose an unsupervised equivariant imaging (EI)-based training framework with a learnable degradation function, overcoming the inaccessibility of ground truth and spectral response function (SRF). Specifically, we formulate the degradation process as a combination of $8\times 8$ mosaicing and $2\times 2$ average downsampling for the LR mosaiced image, while modeling the PAN image as a linear projection of the HR HSI using SRF. Since parameters of SRF are inaccessible, we propose to make them learnable to have an accurate estimation. By enforcing transformation equivariance between the input-output pair of the fusion network, the proposed framework ensures the reconstructed HSI preserves spatial-spectral consistency without relying on paired supervision. Furthermore, we instantiate the proposed HSI imaging system and collect a real-world dataset of 60 paired mosaiced / PAN images. The mosaiced image exhibits 16 spectral bands ranging from 722 to 896 nm and $1020\times 1104$ spatial pixels while the PAN image exhibits $2040\times 2208$ spatial pixels. Comprehensive experiments demonstrate that the proposed method exhibits high spatial consistency and spectral fidelity while maintaining computational efficiency.

Snapshot hyperspectral imaging based on narrowband mosaic array encoding suffers from limitations such as low signal-to-noise ratio, limited spectral range, and low spatial resolution. To address these challenges, we propose a novel hyperspectral imaging system that integrates broadband mosaic image with high-resolution (HR) panchromatic (PAN) image of the same scene, establishing a new paradigm for HR hyperspectral image (HSI) acquisition. To fully leverage the complementary information from multi-source images, we introduce a Deep Error-aware Iterative Optimization Network (EIONet), which iteratively reduces reconstruction errors to successfully reconstruct images with both high spatial resolution and high spectral quality. Specifically, we design a Hierarchical Error-aware Cube Updating Mechanism (HECUM) that dynamically partitions image regions based on their reconstruction difficulty during iterations. By prioritizing the enhancement of feature representation in high-difficulty areas, it effectively suppresses the accumulation and propagation of errors. Meanwhile, we employ a Physics-Based Spectral Degradation Modeling approach, constructing a spectral response function with well-defined physical meaning to accurately model the degradation process from the target domain to the observation domain. Experimental results on two public datasets demonstrate that EIONet achieves state-of-the-art performance across multiple evaluation metrics. The related code is available at: https://github.com/Xiexieiii/EIONet.

Cross-view localization aims to estimate the precise position and orientation of a ground-view image by aligning it with satellite imagery. However, existing homography-based methods are typically evaluated under limited orientation noise (±45°) and exhibit limited refinement capability, as their local correlation-based refinement relies on a reasonably good initial orientation estimate. To address these limitations, we propose a fine-grained cross-view localization method based on orientation-guided homography (OGH-Net), whose core idea is to predict an initial orientation prior that explicitly guides subsequent homography refinement. Specifically, we first design a hybrid bird's-eye-view (BEV) transformation to generate BEV images with preserved central geometry and expanded coverage. Then, we introduce a lightweight orientation-prior module that provides a coarse yaw estimate across the full ±180° range. Finally, we develop a multiscale iterative homography module that progressively refines the projection matrix through hierarchical iterations across multiple feature resolutions. Under cross-area, unknown-orientation conditions, it reduces mean localization error by 11% and mean orientation error by 27% on VIGOR, and further reduces mean localization error by 27% on KITTI compared with previous state-of-the-art methods. Moreover, OGH-Net runs in real time at up to 107 FPS on a single RTX 3090 GPU, offering a favorable trade-off between accuracy and efficiency. The code and trained models will be released at https://github.com/YC-Zhang2025/OGH-Net.

Hyperspectral fusion imaging is challenged by high computational cost due to the abundant spectral information. We find that pixels in regions with smooth spatial-spectral structure can be reconstructed well using a shallow network, while only those in regions with complex spatial-spectral structure require a deeper network. However, existing methods process all pixels uniformly, which ignores this property. To leverage this property, we propose a Selective Re-Learning Fusion Network (SRLF) that initially extracts features from all pixels uniformly and then selectively refines distorted feature points. Specifically, SRLF first employs a Preliminary Fusion Module with robust global modeling capability to generate a preliminary fusion feature. Afterward, it applies a Selective Re-Learning Module to focus on improving distorted feature points in the preliminary fusion feature. To achieve targeted learning, we present a novel Spatial-Spectral Structure-Guided Selective Re-Learning Mechanism (SSG-SRL) that integrates the observation model to identify the feature points with spatial or spectral distortions. Only these distorted points are sent to the corresponding re-learning blocks, reducing both computational cost and the risk of overfitting. Finally, we develop an SRLF-Net, composed of multiple cascaded SRLFs, which surpasses multiple state-of-the-art methods on several datasets with minimal computational cost.

Most of the existing methods for video understanding primarily focus on videos only lasting tens of seconds, with limited exploration of techniques for handling long videos. The increased number of frames in long videos poses two main challenges: difficulty in locating key information and performing long-range reasoning. Thus, we propose DrVideo, a document-retrieval-based system designed for long video understanding. Our key idea is to convert the long-video understanding problem into a long-document understanding task so as to effectively leverage the power of large language models. Specifically, DrVideo first transforms a long video into a coarse text-based long document to initially retrieve key frames and then updates the documents with the augmented key frame information. It then employs an agent-based iterative loop to continuously search for missing information and augment the document until sufficient question-related information is gathered for making the final predictions in a chain-of-thought manner. Extensive experiments on long video benchmarks confirm the effectiveness of our method. DrVideo significantly outperforms existing LLM-based state-of-the-art methods on EgoSchema benchmark (3 minutes), MovieChat-1K benchmark (10 minutes), and the long split of Video-MME benchmark (average of 44 minutes). Code is available at https://github.com/Upper9527/DrVideo.

Prompt learning has been widely adopted to efficiently adapt vision-language models (VLMs) like CLIP for various downstream tasks. Despite their success, current VLM-based facial expression recognition (FER) methods struggle to capture fine-grained textual-visual relationships, which are essential for distinguishing subtle differences between facial expressions. To address this challenge, we propose a multimodal prompt alignment framework for FER, called MPA-FER, that provides fine-grained semantic guidance to the learning process of prompted visual features, resulting in more precise and interpretable representations. Specifically, we introduce a multi-granularity hard prompt generation strategy that utilizes a large language model (LLM) like ChatGPT to generate detailed descriptions for each facial expression. The LLM-based external knowledge is injected into the soft prompts by minimizing the feature discrepancy between the soft prompts and the hard prompts. To preserve the generalization abilities of the pretrained CLIP model, our approach incorporates prototype-guided visual feature alignment, ensuring that the prompted visual features from the frozen image encoder align closely with class-specific prototypes. Additionally, we propose a cross-modal global-local alignment module that focuses on expression-relevant facial features, further improving the alignment between textual and visual features. Extensive experiments demonstrate our framework outperforms state-of-the-art methods on three FER benchmark datasets, while retaining the benefits of the pretrained model and minimizing computational costs.

Segmenting ultra-high-resolution (UHR) images poses a significant challenge due to constraints on GPU memory, leading to a trade-off between detailed local information and a comprehensive contextual understanding. Current UHR methods often employ a multi-branch encoder to handle local and contextual information, which can be memory-intensive. To address the need for both high accuracy and low memory usage in processing UHR images, we introduce a memory-efficient semantic segmentation approach by squeezing context information into local patches (SCPSeg). Our method integrates the processing of local and contextual information within a single-branch encoder. Specifically, we introduce a context squeezing module (CSM) designed to compress global context details into local patches, enabling segmentation networks to perceive broader image contexts. Additionally, we propose a super-resolution guided local feature alignment (LFA) technique to improve segmentation precision by aligning local feature relationships. This approach calculates similarities within sliding windows, avoiding heavy computational costs during the training phase. We evaluate the effectiveness of our proposed method on four widely used UHR segmentation benchmarks. Experimental results demonstrate that our approach enhances UHR segmentation accuracy without incurring additional memory overhead during the inference stage. The code is available at https://github.com/StuLiu/SCPSeg.

AI-generated videos have achieved near-perfect visual realism (e.g., Sora), urgently necessitating reliable detection mechanisms. However, detecting such videos faces significant challenges in modeling high-dimensional spatiotemporal dynamics and identifying subtle anomalies that violate physical laws. In this paper, we propose a physics-driven AI-generated video detection paradigm based on probability flow conservation principles. Specifically, we propose a statistic called Normalized Spatiotemporal Gradient (NSG), which quantifies the ratio of spatial probability gradients to temporal density changes, explicitly capturing deviations from natural video dynamics. Leveraging pre-trained diffusion models, we develop an NSG estimator through spatial gradients approximation and motion-aware temporal modeling without complex motion decomposition while preserving physical constraints. Building on this, we propose an NSG-based video detection method (NSG-VD) that computes the Maximum Mean Discrepancy (MMD) between NSG features of the test and real videos as a detection metric. Last, we derive an upper bound of NSG feature distances between real and generated videos, proving that generated videos exhibit amplified discrepancies due to distributional shifts. Extensive experiments confirm that NSG-VD outperforms state-of-the-art baselines by 16.00\% in Recall and 10.75\% in F1-Score, validating the superior performance of NSG-VD. The source code is available at \url{https://github.com/ZSHsh98/NSG-VD}.

Large models have accelerated the development of intelligent interpretation in remote sensing. Many remote sensing foundation models (RSFM) have emerged in recent years, sparking a new wave of deep learning in this field. Fine-tuning techniques serve as a bridge between remote sensing downstream tasks and advanced foundation models. As RSFMs become more powerful, fine-tuning techniques are expected to lead the next research frontier in numerous critical remote sensing applications. Advanced fine-tuning techniques can reduce the data and computational resource requirements during the downstream adaptation process. Current fine-tuning techniques for remote sensing are still in their early stages, leaving a large space for optimization and application. To elucidate the current development and future trends of remote sensing fine-tuning techniques, this survey offers a comprehensive overview of recent research. Specifically, this survey summarizes the applications and innovations of each work and categorizes recent remote sensing fine-tuning techniques into six types: adapter-based, prompt-based, reparameterization-based, hybrid methods, partial tuning, and improved tuning. In the final section, this survey suggests nine areas worth exploring in this field. Remote sensing fine-tuning methods in this survey can be found at https://github.com/DongshuoYin/Remote-Sensing-Tuning-A-Survey.

Weakly-Supervised Semantic Segmentation (WSSS) aims to train segmentation models by weak labels, which is receiving significant attention due to its low annotation cost. Existing approaches focus on generating pseudo labels for supervision while largely ignoring to leverage the inherent semantic correlation among different pseudo labels. We observe that pseudo-labeled pixels that are close to each other in the feature space are more likely to share the same class, and those closer to the distribution centers tend to have higher confidence. Motivated by this, we propose to model the underlying label distributions and employ cross-label constraints to generate more accurate pseudo labels. In this paper, we develop a unified WSSS framework named Adaptive Gaussian Mixtures Model, which leverages a GMM to model the label distributions. Specifically, we calculate the feature distribution centers of pseudo-labeled pixels and build the GMM by measuring the distance between the centers and each pseudo-labeled pixel. Then, we introduce an Online Expectation-Maximization (OEM) algorithm and a novel maximization loss to optimize the GMM adaptively, aiming to learn more discriminative decision boundaries between different class-wise Gaussian mixtures. Based on the label distributions, we leverage the GMM to generate high-quality pseudo labels for more reliable supervision. Our framework is capable of solving different forms of weak labels: image-level labels, points, scribbles, blocks, and bounding-boxes. Extensive experiments on PASCAL, COCO, Cityscapes, and ADE20 K datasets demonstrate that our framework can effectively provide more reliable supervision and outperform the state-of-the-art methods under all settings.

The fusion of low-resolution hyperspectral images (LR HSI) and high-resolution multispectral images (HR MSI) is a crucial approach for generating hyperspectral images (HSI). However, existing hyperspectral image fusion methods often rely on a single feature mapping process, which makes it difficult to accommodate the significant differences in features between the source images and the real images. Consequently, the generated images frequently exhibit varying degrees of information loss across different spectral bands and limit the overall performance of the fusion. To address this issue, we propose a novel hyperspectral image fusion network. Specifically, an asymptotic spectral mapping module is designed to enhance the detail information fitting capabilities of the fusion network. This module transforms the fitting process of missing information into multiple sets of fitting processes with varying degrees, which can map features with different spectral fidelity to various scales and gradually fit spectral information, thereby reducing spectral distortion. Additionally, we introduce an adaptive defect optimization loss that guides the network to focus on reconstructing regions with substantial spectral differences between LR HSI and HSI, optimizing the network’s constraints regarding the similarity between predicted and real images. Experimental results demonstrate that the proposed fusion network outperforms existing state-of-the-art methods across diverse datasets.

Hyperspectral images (HSIs) are often affected by noise originating from both internal imaging mechanisms and external environmental factors. Therefore, denoising serves as a crucial preprocessing step for HSIs. In real-world scenarios, HSI denoising is particularly challenging due to the complex and band-dependent nature of noise. Current Mamba-based models, although capable of sequential modeling, are highly sensitive to input spectral order and still suffer from information loss over relatively long sequences, potentially leading to local over-sharpening and spectral distortion. To address these issues, we propose a novel linear attention Mamba (LaMamba) for HSI denoising. To capture the intrinsic nature of HSI, a 3D selective scan mechanism is designed to convert the input HSI into spectral-spatial continuous sequences using six bidirectional scan orders. Additionally, a linear attention state space model is proposed to capture long-range correlation. Experimental results on both synthetic and real-world hyperspectral datasets demonstrate that our model significantly outperforms other advanced methods in reconstructing spectral fidelity and spatial visual effect. The code is released at https://github.com/PuhongDuan/LaMamba.

Visual grounding for remote sensing images (RSVG) is a fundamental vision-language task, which aims to locate the objects referred to by the natural language expression from the RS images. Natural language expressions often rely heavily on detailed directional information to describe target objects. Therefore, the thorough and effective utilization of spatial directional information is crucial for accurately locating the referred objects within complex RS images. However, most existing RSVG methods fail to fully leverage directional information, leading to suboptimal outcomes. This paper introduces a novel directional semantic enhanced method for RSVG, dubbed DSEVG. Specifically, we propose a scale-adaptive language-guided interaction (SLI) module that derives scale-specific language features through a hierarchical language adaptation mechanism. These scale-specific language features guide the visual backbone to extract visual features highly relevant to referring expressions. Furthermore, we present a directional semantic enhancement (DSE) module that implicitly enhances directional semantics in referring expressions by leveraging visual spatial information. It also employs an explicit spatial semantic alignment loss to supervise this process, generating localization prompts with enhanced directional representations. These prompts are injected into the queries of each decoder layer to guide the model in effectively utilizing directional semantics. Experimental results on the DIOR-RSVG and OPT-RSVG benchmark datasets validate the effectiveness of the proposed method and demonstrate state-of-the-art performance. Code is available at: https://github.com/WH231203/DSEVG.

Remote sensing scene classification (RSSC) plays a vital role in a variety of applications and has attracted much more attention. In recent years, much progress has been made to release diverse datasets or develop all kinds of techniques for scene classification of multispectral remote sensing images. Nevertheless, very few studies have focused on hyperspectral image scene classification. Moreover, the existing scene classification approaches fail to fully employ the rich spectral information of the input images, which cannot achieve satisfactory performance for hyperspectral images. To alleviate these issues, this work proposes a spectral-spatial fusion network (SSFNet) for hyperspectral RSSC (HRSSC). First, a multiscale regional growth search (MSRGS) method is designed to extract salient object regions from the hyperspectral remote sensing scene. Then, a three-stream network architecture is proposed to extract the global spatial, local spatial, and spectral features, respectively. Finally, the fully connected layer is performed on the extracted features to obtain a class score followed by a decision fusion scheme to generate the final classification result. To evaluate the effectiveness of the proposed SSFNet, we created a publicly available benchmark for the HRSSC dataset, which contains 1445 hyperspectral images, covering 11 scene classes. Experiments on the HRSSC database claim that the proposed SSFNet can attain superior classification performance with respect to other state-of-the-art scene classification techniques. The code of the proposed SSFNet will be available athttps://github.com/PuhongDuan/SSFNet.

Object detection algorithms are pivotal components of UAV imaging systems, extensively employed in complex fields. However, images captured by high-mobility UAVs often suffer from motion blur cases, which significantly impedes the performance of advanced object detection algorithms. To address these challenges, we propose an innovative object detection algorithm specifically designed for blurry images, named dual-stream restoration embedding blur-feature fusion network (DREB-Net). First, DREB-Net addresses the particularities of blurry image object detection problem by incorporating a blurry image restoration auxiliary branch (BRAB) during the training phase. Second, it fuses the extracted shallow features via multilevel attention-guided feature fusion (MAGFF) module, to extract richer features. Here, the MAGFF module comprises local attention modules and global attention modules, which assign different weights to the branches. Then, during the inference phase, the deep feature extraction of the BRAB can be removed to reduce computational complexity and improve detection speed. In loss function, a combined loss of mean squared error (MSE) and SSIM is added to the BRAB to restore blurry images. Finally, DREB-Net introduces fast Fourier transform in the early stages of feature extraction, via a learnable frequency domain amplitude modulation module (LFAMM), to adjust feature amplitude and enhance feature processing capability. Compared to the baseline, DREB-Net achieved an approximate 7% increase in both mAP50 and mAR50 across two experimental datasets. Experimental results indicate that DREB-Net can still effectively perform object detection tasks under motion blur in captured images, showcasing excellent performance and broad application prospects. Our source code will be available athttps://github.com/EEIC-Lab/DREB-Net.git.

Satellite video multi-object tracking (MOT) is fundamentally challenged by extremely weak inter-frame displacement. Most existing methods capture motion at the image level, limiting the effective utilization of fine-grained motion information. To address this limitation, we propose a Flow-driven Motion-Aware Network(FMA-Net) that performs pixel-level motion modeling via the designed Flow-driven Motion Estimator (FME). The captured fine-grained motion is then fully exploited by the Motion-Aware Feature Fusion (MAF) module to guide inter-frame feature fusion, allowing the network to focus on truly dynamic targets during detection. The Motion-Aware Refinement (MAR) module leverages the motion priors to maintain identity consistency and reduce mismatches during association, improving tracking robustness. By jointly modeling and exploiting fine-grained motion cues throughout both detection and association, FMA-Net significantly enhances tracking accuracy in satellite video MOT. Extensive experiments on three satellite video MOT benchmarks demonstrate that FMA-Net achieves state-of-the-art performance across diverse object categories, validating its effectiveness and generalization capability. The code will be publicly available at: https://github.com/luweiqing/FMA-Net.

Semi-supervised semantic segmentation has gained significant attention as a method to reduce the substantial expense associated with pixel-level labeling. The existing methods primarily rely on consistency regularization or self-training. Recent consistency regularization methods augment the input images with weak or strong augmentation (SA) to improve the performance. However, such simple augmentations are not sufficient to simulate the variations in remote sensing images. The self-training methods exclude the noisy pseudo labels by some selection from the unsupervised training process to obtain better performance. The selection may lead to semantic information loss and bias of latent distribution. To solve the above two problems, we propose hierarchical augmentation (HA) and region-aware contrastive (RC) learning, namely HARC, for remote sensing images. The HA strategy simulates three levels of remote sensing image variations, i.e., spatial variations, uniform spectral variations, and uneven spectral variations. It can significantly enhance the model’s capability to handle more intricate variations. The RC learning learns a class-wise feature distribution of all unlabeled samples instead of some screened unlabeled samples. It can eliminate semantic information loss and enhance the model’s resistance to noise from pseudo labels. Our method is evaluated on three public remote sensing datasets, and the experimental results demonstrate its superiority over state-of-the-art (SOTA) semi-supervised methods.

Hyperspectral image (HSI) encompasses abundant spatial and spectral details, while Light Detection and Ranging (LiDAR) delivers precise elevation data. The amalgamation of HSI and LiDAR data significantly improves the precision of image classification. However, most methods focus solely on spatial features while neglecting frequency domain information, limiting the ability of deep models to characterize land cover. Furthermore, how to establish a sufficient interaction between different modalities is also an important issue. In this paper, we propose a novel multiscale cross-domain fusion network (MCFNet) for joint classification of HSI and LiDAR data. The main idea is that the wavelet transform can provide details at different resolutions simultaneously, supplementing spatial domain information and enriching feature representation. In addition, the multimodal fusion module (MFM) guided by HSI and the cross-domain fusion module (CDFM) strategy are developed to integrate features from diverse modalities and domains, respectively. Specifically, frequency domain features are extracted by discrete wavelet transform, and spatial domain features of the image are captured through a set of convolution operations. Then interactive fusion is performed by MFM and CDFM, and finally the integrated features are categorized using a classification module. Extensive experiments on three widely-used HSI and LiDAR datasets indicate that MCFNet outperforms the SOTA methods. The code will be available at https://github.com/MSFLabX/MCFNet.

Existing hyperspectral image classification (HSIC) models based on the Mamba architecture predominantly center on characterizing the original spectral and spatial domains, with limited exploration of time–frequency analysis. In this study, we propose a novel wavelet decomposition-based spectral–spatial Mamba network for HSIC, dubbed “WD-SSMamba.” This model incorporates both 1-D and 2-D wavelet decompositions to extract spectral and spatial features in the frequency domain, respectively. Specifically, we design an innovative frequency feature extraction (FE) block, which comprises a spectral wavelet convolution (SWC) module for spectral FE and a wavelet separable convolution (WSC) module for spatial FE. Furthermore, to address the challenges of integrating spectral–spatial features in traditional Mamba models for HSIC, we devise a dual-branch Mamba block featuring a cross-fusion structure, termed the “HyperMamba” block, to efficiently extract and fuse spectral and spatial features. Comprehensive experiments were conducted on four publicly available hyperspectral image (HSI) datasets, namely, Pavia University, WHU-Hi-LongKou, WHU-Hi-HongHu, and Houston 2013. The results show that the WD-SSMamba model achieves overall accuracies (OAs) of 92.48%, 97.91%, 89.98%, and 87.00% on these datasets, respectively, with fewer than 20 training samples per class, surpassing those of other competing models across all tested datasets. Moreover, it significantly reduces the number of parameters to less than 50k and floating-point operations (FLOPs) to less than 3.02M, thereby fully showcasing the immense potential of frequency analysis and the Mamba structure in HSIC.

The modulation transfer function tailored image filter (MTF-TIF) has long been regarded as the optimal filter for multispectral image pansharpening. It excels at simulating the camera's frequency response, thereby capturing finer image details and significantly improving pansharpening performance. However, we are skeptical about whether the pre-measured MTF is sufficient to describe the characteristics of actually acquired panchromatic image (PAN) and multispectral image (MSI). For example, any image resampling operations in geometric correction or image registration inevitably change the sharpness of acquired PAN and MSI, and the processed images no longer conform to the camera's MTF. Further, following the Wald protocol, in deep learning (DL) methods using MTF-TIF for downsampling images to construct training data does not satisfy the generalization consistency of training and testing. To prove our point, we propose a pair of symmetric frameworks based on DL in this paper, to find better image filters suitable for both traditional and DL pansharpening methods. We embed two learnable filters into the frameworks to simulate the optimal image filter, namely anisotropic Gaussian image filter and arbitrary image filter. Further, the proposed frameworks can capture subtle offsets between images and maintain the smoothness of the global deformation field. Extensive experiments on various satellite datasets demonstrate that the proposed frameworks can find better image filters than MTF-TIFs, which can achieve better pansharpening performance with stronger generalization ability.

Image matching is a critical task in computer vision research, focusing on aligning two or more images with similar features. Feature detection and description constitute the core of image matching. Handcrafted detectors are capable of obtaining distinctive points but these points may not be repeatable on the image pairs especially those with dramatic appearance changes. On the contrary, the learned detectors can extract a large number of repeatable points but many of them tend to be ambiguous points with low distinctiveness. Moreover, in the scenarios of dramatic appearance change, commonly used contrast or triplet loss in the training of descriptors employ the hard negative mining strategy, which may obtain overly challenging negative samples by global sampling, resulting in sluggish convergence or even overfitting. Those learned descriptors may not guarantee that the corresponding points enjoy larger similarities than unmatched ones, leading to inaccurate matches. To address those issues, we propose a hierarchically learned detector and descriptor (HLDD) for robust image matching, which contains three modules: a handcrafted-learned detector, a hierarchically learned descriptor, and a coarse-to-fine matching strategy. The handcrafted-learned detector integrates the advantages of handcrafted and learned detectors. It extracts distinctive feature points from a learned repeatability map robust to image changes and eliminates the ambiguous ones according to a learned distinctiveness map. The descriptor is trained by a proposed hierarchical triplet loss, which employs a dual window strategy. It can obtain the hardest negative samples in local windows, which are comparatively easier over global sampling, ensuring the effective training of descriptors. The coarse-to-fine matching strategy performs global and local mutual nearest neighbor matching on the coarse and fine descriptor maps respectively to improve the matching accuracy progressively. By comparing with other matching methods, experimental results demonstrate the superiority of the proposed method in the task of image matching, homography estimation, visual localization, and relative pose estimation. Moreover, ablation studies illustrate the effectiveness of the three proposed modules.

Cross-domain image segmentation plays a crucial role in the field of remote sensing. Current approaches often rely on a mean-teacher model that is integrated from student models to guide the training of the student model itself. However, the feature space of the mean-teacher model exhibits significant domain discrepancy and considerable class overlap, which results in suboptimal performance. Motivated by the idea of learning from stronger teachers, we introduce a robust domain adaptation method called LFMDA. This novel approach is the first to explicitly enhance cross-domain semantic segmentation performance by leveraging vision foundation models (VFMs) within remote sensing applications. Specifically, we propose a prototypical contrastive knowledge distillation loss (PCD) that enables the student model to produce domain-invariant yet category-discriminative features by distilling knowledge from a domain-generalized VFM teacher. Additionally, we introduce a local region homogenization strategy (LRH) to generate high-quality and high-quantity pseudo-labels by incorporating a Segment Anything Model (SAM). Extensive empirical evaluations demonstrate that our method outperforms existing approaches, setting a new state-of-the-art (SOTA) method in domain-adaptive remote sensing image segmentation. The code is available at https://github.com/StuLiu/LFMDA.

Existing hyperspectral fusion computational imaging methods primarily rely on using high-resolution multispectral images (HRMSI) to provide spatial details for low-resolution hyperspectral images (LRHSI), thereby enabling the reconstruction of hyperspectral images. However, these methods are often limited by the low spectral resolution of the HRMSI, making the sampled tensors unable to provide effective information for the LRHSI in a finer spectral range. To achieve more accurate computational imaging results, we propose a Heterospectral Structure Compensation Sampling (HSC-sampling) mechanism. Unlike traditional spatial sampling methods, which directly calculate the interpolation between adjacent pixels, this mechanism analyzes the structural complementarity among different bands in LRHSI. It utilizes the information from other bands to compensate for the missing details in the current band. Additionally, a novel Multi-phase Mixed Modeling (M2M) approach is designed, expanding the model's analytical capabilities into multiple phases to accommodate the high-dimensional nature of HSI data. Specifically, it extracts fusion features from three phases and organizes the generated features along with the input features into a multi-variate mixed cube based on phase relationships, thereby capturing feature correlations across different phases. Based on the HSC-sampling mechanism and the M2M approach, we construct a Merging Residual Concatenation (MRC) hyperspectral fusion computational imaging network. Compared to other state-of-the-art methods, this network achieves significant improvements in fusion performance across multiple datasets. Moreover, the effectiveness of the HSC-sampling mechanism has been demonstrated in various hyperspectral imaging tasks. Code is available at: https://github.com/1318133/HSC-Sampling.

Camouflaged object detection (COD) aims to discover objects that are seamlessly embedded in the environment. Existing COD methods have made significant progress by typically representing features in a discrete way with arrays of pixels. However, limited by discrete representation, these methods need to align features of different scales during decoding, which causes some subtle discriminative clues to become blurred. This is a huge blow to the task of identifying camouflaged objects from clear subtle clues. To address this issue, we propose a novel continuous feature representation network (CFRN), which aims to represent features of different scales as a continuous function for COD. Specifically, a Swin transformer encoder is first exploited to explore the global context between camouflaged objects and the background. Then, an object-focusing module (OFM) deployed layer by layer is designed to deeply mine subtle discriminative clues, thereby highlighting the body of camouflaged objects and suppressing other distracting objects at different scales. Finally, a novel frequency-based implicit feature decoder (FIFD) is proposed, which directly decodes the predictions at arbitrary coordinates in the continuous function with implicit neural representations, thus propagating clearer discriminative clues. Extensive experiments on four challenging COD benchmarks demonstrate that our method significantly outperforms state-of-the-art methods. The source code will be available at https://github.com/SongZeHNU/CFRN.

Single spectral image demosaicing for multispectral filter array (MSFA) is an essential task in spectral imaging, aiming to recover a mosaic-free spectral image from its mosaic raw counterpart. Existing deep learning-based methods typically improve the reconstruction performance by indiscriminately stacking CNN-based blocks, failing to effectively handle the intertwined spatio-spectral correlations caused by spatial sub-sampling and spectral aliasing. In this paper, we propose Mosaic Pattern Excavation Transformer (MPEFormer) to achieve better reconstruction by effectively modelling the intertwined spatio-spectral correlations. Specifically, the proposed three-branch model integrates low-frequency information, edge information, and fine high-frequency details essential for spectral image reconstruction, with the third branch serving as the core component. In this branch, we design the Dual Fusion Self-attention Block (DFSAB) and the Mosaic Pattern-guided Spectral Modulation Module (MPSM). DFSAB incorporates the Mosaic Pattern Excavation Self-attention (MPESA) mechanism, which effectively captures non-local spatio-spectral correlations induced by the MSFA pattern distributed across the whole image, thereby enhancing the expressive capability of the model. By dynamically integrating various MSFA pattern-related dependencies, MPSM enables adaptive recalibration of spectral information. Extensive experimental results demonstrate the effectiveness of our MPEFormer, highlighting its greater potential over the state-of-the-art MSFA demosaicing methods. The code will be uploaded at https://github.com/Matsuri247/MPEFormer.

Open set recognition (OSR) aims to identify whether a test sample belongs to a semantic class in the classifier training set. Existing OSR methods exhibit prominent performance on various image datasets. However, they are primarily designed for general object recognition rather than more complex camouflaged object recognition. When an object is camouflaged, i.e., it exhibits a similar pattern to the background, it is difficult to finely identify it and differentiate between known and unknown categories. To address this problem, we propose a novel multi-granularity context perception network (MCPNet) for OSR of camouflaged objects, which can accurately identify camouflaged objects by fusing coarse-grained and fine-grained context features. In MCPNet, the vision transformer is first utilized to extract coarse-grained context features to locate the approximate location of camouflaged objects. Then, an adaptive local focus module (ALFM) is proposed to pick out the most discriminative regions and learn the fine-grained context of these regions. Finally, multi-granular context features are fused to obtain recognition results. During the training, a contrastive clustering module (CCM) is introduced to guide the network to effectively utilize multi-granularity context to generate high-confidence decision boundaries. We also built two camouflaged object classification datasets named ACOC and NCOC which mainly consist of artificial camouflage and natural camouflage respectively to facilitate research in OSR of camouflaged objects. Experimental results on two datasets show that MCPNet outperforms state-of-the art methods.

Spectral super-resolution has attracted the attention of more researchers for obtaining hyperspectral images (HSIs) in a simpler and cheaper way. Although many convolutional neural network (CNN)-based approaches have yielded impressive results, most of them ignore the low-rank prior of HSIs resulting in huge computational and storage costs. In addition, the ability of CNN-based methods to capture the correlation of global information is limited by the receptive field. To surmount the problem, we design a novel low-rank tensor reconstruction network (LTRN) for spectral super-resolution. Specifically, we treat the features of HSIs as 3-D tensors with low-rank properties due to their spectral similarity and spatial sparsity. Then, we combine canonical-polyadic (CP) decomposition with neural networks to design an adaptive low-rank prior learning (ALPL) module that enables feature learning in a 1-D space. In this module, there are two core modules: the adaptive vector learning (AVL) module and the multidimensionwise multihead self-attention (MMSA) module. The AVL module is designed to compress an HSI into a 1-D space by using a vector to represent its information. The MMSA module is introduced to improve the ability to capture the long-range dependencies in the row, column, and spectral dimensions, respectively. Finally, our LTRN, mainly cascaded by several ALPL modules and feedforward networks (FFNs), achieves high-quality spectral super-resolution with fewer parameters. To test the effect of our method, we conduct experiments on two datasets: the CAVE dataset and the Harvard dataset. Experimental results show that our LTRN not only is as effective as state-of-the-art methods but also has fewer parameters. The code is available at https://github.com/renweidian/LTRN.

Recently, low-rank tensor regularization has received more and more attention in hyperspectral and multispectral fusion (HMF). However, these methods often suffer from inflexible low-rank tensor definition and are highly sensitive to the permutation of tensor modes, which hinder their performance. To tackle this problem, we propose a novel generalized tensor nuclear norm (GTNN)-based approach for the HMF. First, we define a novel GTNN by extending the existing third-mode-based tensor nuclear norm (TNN) to arbitrary mode, which conducts the Fourier transform on an arbitrary single mode and then computes the TNN for each mode. In this way, we can not only capture more extensive correlations for the three modes of a tensor, and also omit the adverse effect of permutation of tensor modes. To utilize the correlations among spectral bands, the high-resolution hyperspectral image (HSI) is approximated as low-rank spectral basis multiplication by coefficients, and we estimate the spectral basis by conducting singular-value decomposition (SVD) on HSI. Then, the coefficients are estimated by addressing the proposed GTNN regularized optimization. In specific, to exploit the non-local similarities of the HSI, we first cluster the patches of the coefficient into a 3-D, which contains spatial, spectral, and non-local modes. Since the collected tensor contains the strong non-local spatial-spectral similarities of the HSI, the proposed low-rank tensor regularization is imposed on these collected tensors, which fully model the non-local self-similarities. Fusion experiments on both simulated and real datasets prove the advantages of this approach. The code is available at https://github.com/renweidian/GTNN.

Spectral super-resolution aims to reconstruct a hyperspectral image (HSI) from its corresponding RGB image, which has drawn much more attention in remote sensing field. Recent advances in the application of deep learning models for spectral super-resolution have demonstrated great potential. However, these methods only work in spectral-spatial domain while rarely explore the potential property in the frequency domain. In this work, we first attempt to address spectral super-resolution in the frequency domain. To well merge the frequency information into the super-resolution network, a spectral-spatial-frequency domain fusion network (SSFDF) is designed, which consists of three key parts: frequency-domain feature learning, spectral-spatial domain feature learning, and feature fusion module. In more detail, a frequency-domain feature learning network is first exploited to dig the frequency-domain information of the input data. Then, a symmetric convolutional neural network (CNN) is developed to acquire the spectral-spatial features of the input data, where a parameter-sharing strategy is utilized to reduce network parameters. Finally, a feature fusion module is proposed to reconstruct HSI. Comprehensive experiments on several datasets reveal that our method can attain state-of-the-art reconstruction result with respect to other spectral super-resolution techniques.

Interactive image segmentation (IIS) has emerged as a promising technique for decreasing annotation time. Substantial progress has been made in pre- and post-processing for IIS, but the critical issue of interaction ambiguity, notably hindering segmentation quality, has been under-researched. To address this, we introduce ADAPTIVE CLICK - a click-aware transformer incorporating an adaptive focal loss (AFL) that tackles annotation inconsistencies with tools for mask- and pixel-level ambiguity resolution. To the best of our knowledge, AdaptiveClick is the first transformer-based, mask-adaptive segmentation framework for IIS. The key ingredient of our method is the click-aware mask-adaptive transformer decoder (CAMD), which enhances the interaction between click and image features. Additionally, AdaptiveClick enables pixel-adaptive differentiation of hard and easy samples in the decision space, independent of their varying distributions. This is primarily achieved by optimizing a generalized AFL with a theoretical guarantee, where two adaptive coefficients control the ratio of gradient values for hard and easy pixels. Our analysis reveals that the commonly used Focal and BCE losses can be considered special cases of the proposed AFL. With a plain ViT backbone, extensive experimental results on nine datasets demonstrate the superiority of AdaptiveClick compared to state-of-the-art methods. The source code is publicly available at https://github.com/lab206/AdaptiveClick.

Real-time semantic segmentation plays an important role in auto vehicles. However, most real-time small object segmentation methods fail to obtain satisfactory performance on small objects, such as cars and sign symbols, since the large objects usually tend to devote more to the segmentation result. To solve this issue, we propose an efficient and effective architecture, termed small objects segmentation network (SOSNet), to improve the segmentation performance of small objects. The SOSNet works from two perspectives: methodology and data. Specifically, with the former, we propose a dual-branch hierarchical decoder (DBHD) which is viewed as a small-object sensitive segmentation head. The DBHD consists of a top segmentation head that predicts whether the pixels belong to a small object class and a bottom one that estimates the pixel class. In this situation, the latent correlation among small objects can be fully explored. With the latter, we propose a small object example mining (SOEM) algorithm for balancing examples between small objects and large objects automatically. The core idea of the proposed SOEM is that most of the hard examples on small-object classes are reserved for training while most of the easy examples on large-object classes are banned. Experiments on three commonly used datasets show that the proposed SOSNet architecture greatly improves the accuracy compared to the existing real-time semantic segmentation methods while keeping efficiency. The code will be available at https://github.com/StuLiu/SOSNet.

Infrared image (IR) and visible image (VI) fusion creates fusion images that contain richer information and gain improved visual effects. Existing methods generally use the operators of manual design, such as intensity and gradient operators, to mine the image information. However, it is hard for them to achieve a complete and accurate description of information, which limits the image fusion performance. To this end, a novel information measurement method is proposed to achieve IR and VI fusion. Its core idea is to guide a generator in achieving image fusion by learning the denoisers. Specifically, by using denoisers to restore fusion images with different noise interference to source images, a mutual competition relationship is formed between denoisers, which helps the generator thoroughly explore the data specificity of the source images and guide it to achieve more accurate feature representation. In addition, a semantic adaptive measurement loss function is proposed to constrain the generator, which fuses semantic information adaptively by considering the semantic information density of different source images. The results of quantitative and qualitative experiments have shown that the proposed method can achieve a higher quality information fusion and has a faster fusion speed on three public datasets when compared with advanced methods.

Multimodal recommendation systems integrate diverse multimodal information into the feature representations of both items and users, thereby enabling a more comprehensive modeling of user preferences. However, existing methods are hindered by data sparsity and the inherent noise within multimodal data, which impedes the accurate capture of users' interest preferences. Additionally, discrepancies in the semantic representations of items across different modalities can adversely impact the prediction accuracy of recommendation models. To address these challenges, we introduce a novel diffusion-based contrastive learning (DiffCL) framework for multimodal recommendation. DiffCL employs a diffusion model (DM) to generate contrastive views that effectively mitigate the impact of noise during the contrastive learning phase. Furthermore, it improves semantic consistency across modalities by aligning distinct visual and textual semantic information through stable ID embeddings. Finally, the introduction of the item-item graph (I-I graph) enhances multimodal feature representations, thereby alleviating the adverse effects of data sparsity on the overall system performance. We conduct extensive experiments on three public datasets, and the results demonstrate the superiority and effectiveness of the DiffCL.

Knowledge distillation (KD) improves the performance of a compact student network by transferring learned knowledge from a cumbersome teacher network. In the existing approaches, the multiscale feature knowledge is transferred via densely connected paths, which increases the optimization difficulty. Moreover, correlations among the labels are neglected despite their capability to enhance the intraclass similarity of samples. To solve these issues, we propose cascade fusion and correlation enhancement for KD (CC-KD). The multiscale feature knowledge is transferred via much simpler paths, which are constructed by fusing features of different scales with cross-scale attention (CSA) in a cascade manner, thereby reducing the optimization difficulty. On the other hand, the relational knowledge of teacher logits is further enhanced by correlations of the corresponding labels, so that the student can produce more similar logits for the samples in the same category. Extensive experimental results on five public datasets (i.e., CIFAR100/10, ImageNet, RAF-DB, and FERPlus) indicate superior performance of the proposed method over several state-of-the-arts (SOTAs). More specifically, our method obtains an accuracy of 71.70% on ImageNet and achieves a new record of 90.20% on RAF-DB with fewer calculations and parameters.

Bipartite graph (BiG) has been proven to be efficient in handling massive multiview data for clustering. However, how to regulate the structural information of view-specific anchors and view-shared BiG is still open and needs to be further studied. Hence, a novel dual-structural BiG learning (DsBiGL) method is proposed in the article. It transforms BiG learning into a joint optimization problem of IntrA-view and InteR-view subspace learning (IASL and IRSL) with the structural constraints, such as k-nearest neighbor (KNN) and low-rank. On one hand, IASL uses the KNN and view-specific low-rank constraints to enhance the discriminativeness of view-specific anchors. On the other hand, IRSL uses an adaptive weighting strategy to obtain view-shared BiG directly from multiview samples, where the KNN and view-shared low-rank constraints are adopted to encode local connectivity and cluster information between samples. Note that IASL and IRSL are integrated into a unified optimization model, which ensures the interactive enhancement of view-specific anchor representation and view-shared BiG learning. Finally, an algorithm based on iterative optimization is designed to solve the proposed DsBiGL model. Experimental results on various multiview datasets have demonstrated the superiority of DsBiGL in terms of clustering results when compared with other comparative methods.

Spectral super-resolution (SSR) aims to restore a hyperspectral image (HSI) from a single RGB image, in which deep learning has shown impressive performance. However, the majority of the existing deep-learning-based SSR methods inadequately address the modeling of spatial-spectral features in HSI. That is to say, they only sufficiently capture either the spatial correlations or the spectral self-similarity, which results in a loss of discriminative spatial-spectral features and hence limits the fidelity of the reconstructed HSI. To solve this issue, we propose a novel SSR network dubbed multistage spatial-spectral fusion network (MSFN). From the perspective of network design, we build a multistage Unet-like architecture that differentially captures the multiscale features of HSI both spatialwisely and spectralwisely. It consists of two types of the self-attention mechanism, which enables the proposed network to achieve global modeling of HSI comprehensively. From the perspective of feature alignment, we innovatively design the spatial fusion module (SpatialFM) and spectral fusion module (SpectralFM), aiming to preserve the comprehensively captured spatial correlations and spectral self-similarity. In this manner, the multiscale features can be better fused and the accuracy of reconstructed HSI can be significantly enhanced. Quantitative and qualitative experiments on the two largest SSR datasets (i.e., NTIRE2022 and NTIRE2020) demonstrate that our MSFN outperforms the state-of-the-art SSR methods. The code implementation will be uploaded at https://github.com/Matsuri247/MSFN-for-Spectral-Super-Resolution.

Facing the pervasive problem of missing supervised information, cross-scene hyperspectral image (HSI) classification tasks based on unsupervised domain adaptation (UDA) techniques have emerged. However, lacking an integral view of feature-level alignment and decision-level analysis, most UDA methods treat target data with differential domain shift equally. Focused on this problem, a novel consistency-aware customized learning (CACL) approach is proposed, in this paper. Specifically, we develop a convolution-based domain-invariant feature learning network. First, the feature extractor is employed to extract spectral-spatial category prototypes. At the same time, domain-level distribution alignment is performed with the domain discriminator. Then, a customized learning strategy, i.e., inter/intra-domain contrast learning, is designed based on whether the pseudo-labels are consistent with the spectral-spatial prototype matchability labels. In addition, focal loss is introduced for information mining of hard samples. The experimental results demonstrate the state-of-the-art of the method.

Remote sensing image change captioning is a crucial image interpretation technique that auto-generates language captions of differences between multi-temporal remote sensing images. Previous attention-based methods were difficult to generate accurate captions due to their inability to precisely locate crucial visual change areas. To address this challenge, this paper introduces a novel method that aims to leverage explicit visual change information to enhance its change description capabilities. Specifically, the proposed model comprises three key components: 1) the change-visual enhancement module leverages the change image containing object-level visual information to enhance the multi-temporal images at the image level; 2) the multi-temporal feature fusion module captures accurate visual change features through a meticulously designed feature fusion at feature level; 3) the caption generation module inputs the visual change features into transformer-based generator to produce desired captions of multi-temporal remote sensing images. Experimental results on LEVIR-CC dataset demonstrate that our method has achieved state-of-the-art performance.

Unsupervised domain adaption (UDA) has gained popularity in narrowing performance gaps across domains in remote sensing image semantic segmentation (RSISS). However, current UDA methods suffer from serious noisy pseudo-labels, adversely affecting domain adaptation performance. In this work, a local region homogenizing domain adaptation method (RegDA) is proposed to tackle this issue. Specifically, a generalized segment anything model (SAM) is utilized to obtain the semantic-consistent regions for the images in the target domain. Furthermore, a pixel-level voting scheme is proposed to get the semantic label for each local region and assign it to each pixel within this region. In this way, more reliable pseudo-labels are obtained and domain adaptation performance is improved. Experiment results on ISPRS datasets demonstrate that the proposed RegDA outperforms previous UDA approaches for RSISS. The code will be available at https://github.com/StuLiu/RegDA.

Semi-supervised semantic segmentation has attracted a lot of attention due to the high cost of obtaining pixel-level labels. To get a good grade for semi-supervised semantic segmentation, many methods based on self-training have been proposed. The existing self-training method achieves semi-supervised learning by identifying the samples with pseudo labels with a threshold of its confidence or a classifier. The samples with pseudo labels enrich the training dataset so that the semantic segmentation performance is improved. However, it inevitably introduces samples with wrong pseudo labels, which may harm the final performance. To tackle this problem, we proposed a new supervision to make the model trained well in the pseudo labels that never be filtered. In particular, the region-aware contrastive learning constructs a class-level feature space. Then the class-level feature space is subsequently used for supervising the training on unlabeled images. The experimental results on Vaihingen dataset show that our method outperforms the state-of-the-art semi-supervised methods.

Unsupervised cross-scene hyperspectral image classification transfers the learnable knowledge from a labeled source scene to an unlabeled target scene. Currently, many statistical distribution alignment methods are introduced to mitigate domain discrepancy. However, these methods ignore the finer class specific structure which may cause negative transfer. To solve this issue, a prototype-based inter-intra domain alignment network is proposed for unsupervised cross-scene hyperspectral image classification. Specifically, a prototype-based inter-intra alignment method is proposed to narrow the feature distribution gap. Furthermore, an uncertainty estimation is developed to obtain highly reliable pseudo-labels in the target scene. Experiment results on several datasets imply that the proposed method outperform several cutting-edge unsupervised classification methods.

Detecting eye contact is essential for embodied robots to engage in natural interactions with humans, enhancing the intuitiveness and comfort of these exchanges. However, eye contact detection often presents a significant challenge due to a variety of factors, such as low contrast and various forms of occlusions. Existing methods incorporate convolutional neural networks (CNNs) or Transformers to learn discriminative representations, but usually ignore the influence of noisy or less relevant regions in facial images. To address this gap, we propose the deep feature selection and fusion network (FSFNet) for eye contact detection in multi-party conversations. Our proposed method adaptively selects fine-grained visual features and reduces the impacts of irrelevant features. Specifically, we present a local feature selection scheme that leverages the attention scores to progressively concentrate on the most informative features. By integrating the carefully selected features into the multi-head self-attention module, we can maintain the superior properties of Transformers while simultaneously reducing the overall computational demands. We evaluate the proposed method on the official eye contact detection datasets, which achieves promising results of 0.8174 and 0.79 on the validation and test sets, respectively. We have made the source code publicly accessible in https://github.com/ma-hnu/FSFNet.

Temporal answer grounding in instructional video (TAGV) is a new task naturally derived from temporal sentence grounding in general video (TSGV). Given an untrimmed instructional video and a text question, this task aims at locating the frame span from the video that can semantically answer the question, i.e., visual answer. Existing methods tend to solve the TAGV problem with a visual span-based predictor, taking visual information to predict the start and end frames in the video. However, due to the weak correlations between the semantic features of the textual question and visual answer, current methods using the visual span-based predictor do not work well in the TAGV task. In this paper, we propose a visual-prompt text span localization (VPTSL) method, which introduces the timestamped subtitles for a text span-based predictor. Specifically, the visual prompt is a learnable feature embedding, which brings visual knowledge to the pre-trained language model. Meanwhile, the text span-based predictor learns joint semantic representations from the input text question, video subtitles, and visual prompt feature with the pre-trained language model. Thus, the TAGV is reformulated as the task of the visual-prompt subtitle span localization for the visual answer. Extensive experiments on five instructional video datasets, namely MedVidQA, TutorialVQA, VehicleVQA, CrossTalk and Coin, show that the proposed method outperforms several state-of-the-art (SOTA) methods by a large margin in terms of mIoU score, which demonstrates the effectiveness of the proposed visual prompt and text span-based predictor.

Facial expression recognition (FER) in the wild is extremely challenging due to occlusions, variant head poses under unconstrained conditions and incorrect annotations (e.g., label noise). In this paper, we aim to improve the performance of in-the-wild FER with Transformers and online label correction. Different from pure CNNs based methods, we propose a Transformer-augmented network (TAN) to dynamically capture the relationships within each facial patch and across the facial patches. Specifically, the TAN translates a number of facial patch images into a set of visual feature sequences by a backbone convolutional neural network. The intra-patch Transformer is subsequently utilized to capture the most discriminative features within each visual feature sequence. The position-disentangled attention mechanism of the intra-patch Transformer is proposed to better incorporate the positional information for feature sequences. Furthermore, we propose the inter-patch Transformer to model the dependencies across these feature sequences. More importantly, we present the online label correction (OLC) framework to correct suspicious hard labels and accumulate soft labels based on the predictions of the model, which strengthens the robustness of our model against label noise. We validate our method on several widely-used datasets (RAF-DB, FERPlus, AffectNet), realistic occlusion and pose variation datasets, and synthetic noisy datasets. Extensive experiments on these benchmarks demonstrate that the proposed method performs favorably against state-of-the-art methods. The source code will be made publicly available.

Recently, deep learning presents a promising performance in the joint classification of multimodal remote sensing (RS) data. However, most of the approaches adopt a supervised learning manner, where the discrimination capability is limited by the paucity of labeled samples. Though some attempts have been made to develop semi-supervised methods, they prefer to select the highly confident predictions as pseudo ground truth and discard those unreliable ones. Actually, unreliable samples can also provide useful information, e.g., indicating the categories to which samples may belong and definitely not belong. Focused on this, a novel uncertainty-aware contrastive learning (UACL) method is proposed. Here, label uncertainty analysis based on multi-level probability estimation is first conducted to separate reliable and unreliable samples, which are then processed with a designed hybrid (“hard” or “soft”) contrastive learning (CL) strategy. For reliable samples, the “hard” CL pushes the network to learn features that will minimize the intra-class distance while maximizing the inter-class distance, according to the pseudo-labels. For unreliable samples, the “soft” CL aims to learn the similarity and difference among samples, where the predicted class probabilities are queried to estimate a soft mask for an adaptive feature similarity measurement. Moreover, a multimodal spectral-spatial joint feature representation pipeline of triple branches, i.e., one spectral branch for hyperspectral images (HSIs) and two spatial branches for multimodal data, is also introduced. By jointly learning from both labeled and unlabeled samples, more discriminative spectral-spatial feature representation will lead to a further boost in classification performance. Extensive experiments on four well-known multimodal datasets prove the effectiveness of the proposed semi-supervised classification method. Codes are available at https://github.com/Ding-Kexin/UACL.

In this article, a two-dimensional (2-D) spectral representation is proposed for the visualization and classification of hyperspectral images (HSIs). First, several sequence data processing methods, i.e., Gramian angular field (GAF) algorithm, Markov transition field (MTF), and recurrence plot (REP), are applied to obtain multiple 2-D features of a one-dimensional (1-D) spectrum. Second, the 2-D spectral features are stacked together to form the final 2-D spectral representation. Finally, many excellent classifiers in computer vision field are applied on the 2-D spectral representation to obtain the final classification result. Furthermore, 114 target spectral visualization maps are established based on their 1-D spectra. Experimental results reveal that the 2-D spectral representation has multiple advantages in terms of better visual quality and classification accuracies. The code of this work is available athttps://github.com/zhuyongxiang1/two-dimensional-spectral-representation.

Oriented object detection with remote sensing scenes has made excellent progress in recent years, especially using anchor-free detectors. Without the limitation of inherent prior spatial information, anchor-free detectors regress the detection boxes from the object center or edge in an elegant way. However, anchor-free detectors suffer severe feature misalignment and inconsistency between classification and regression. Especially in remote sensing scenes, there are densely arranged instances and multi-scale representations, which will affect the detection accuracy. Therefore, a feature augmentation module (FAM) and an oriented feature alignment (OFA) module are proposed for oriented object detection called FAA-Det. More specifically, we first introduce a FAM to enhance the object representation. After that, the augmented feature maps will be fed into OFA for feature alignment and accurate detection. OFA has two independent branches for classification and regression, and their separate structures can alleviate the inconsistency in detection. FAM and OFA comprise the FAA-Head in our detector. Extensive evaluation demonstrates the effectiveness of our proposed FAA-Det that performs the state-of-the-art (SOTA) mean average precision (mAP) on the DOTA and HRSC2016 datasets without bells and whistles. Our code will be available athttps://github.com/jimuIee/FAA-Det.

Domain adaptive segmentation has recently gained more and more attention in the remote sensing field. However, current methods often generate a significant number of uncertain examples, i.e., noisy pseudo-labels, in the target domain, which adversely affects model convergence. To solve this issue, an uncertain example mining network is proposed for domain adaptive segmentation of remote sensing images. Specifically, a novel strategy called multilevel pseudo-label correcting (MPC) is proposed to correct the pseudo-labels in class, pixel, and superpixel levels. In this way, more reliable pseudo-labels can be selected for the subsequent training stage. Furthermore, a noise-robust example mining strategy, termed uncertainty-based valuable example mining (UVEM), is proposed to prioritize confident examples with significant gradients for training effectively. Extensive empirical evaluations on IsprsDA and LoveDA datasets demonstrate that the proposed method outperforms previous approaches, establishing state-of-the-art results in domain adaptive remote sensing image segmentation (RSIS). The code will be available athttps://github.com/StuLiu/UemDA.

Remote sensing image change detection can analyze alterations on the Earth’s surface within a specific region. However, the accuracy of change detection has consistently been hindered by the style differences in captured images caused by seasonal or lighting variations, as well as the challenge of distinguishing similar features between the background and foreground in the scene. To this end, a multidomain differential excavating network (MDENet) for change detection is introduced. Using the novel multidomain differential collaboration module (MDCM) to precisely capture object features on the frequency and spatial domains across diverse temporal domains, it enables simultaneous querying of global and local change information. Moreover, the multineighborhood frequency gate attention (MFGatt) is devised to eliminate the impact of image style relevance information and consolidate attention toward object localization, thereby enhancing the adaptability of the network to variations in image style. Extensive experiments have illustrated that our proposed network achieves better detection accuracy compared with current state-of-the-art (SOTA) methods on various datasets.

Nowadays, high-resolution remote sensing images provide rich data sources and deep learning models show powerful feature representation capability for remote sensing object detection. However, due to the complex object structure as well as the changeable rotation angle, how to efficiently estimate the oriented bounding box regarding the accurate location of objects is still an open issue. Focused on this, a new one-stage structure-adaptive oriented object detection (SOOD) network is proposed, in this article. First, we designed a new rotation angle encoder (RAE), where an angle coordinate system is adopted and periodic angle correction is conducted. Different from the traditional longe-edge definition for angle estimation, the RAE can mitigate boundary discontinuity and square-like problems. Then, structure-adaptive label assignment (SALA) and confidence estimation (SACE) are introduced, to locate the position of objects more accurately. On the one hand, the anchor box determines the label assignment according to the affiliation relationship between the center point and the object’s inner ellipse boundary. By constraining the ellipse boundary and employing non-parametric label assignment, high-quality anchor boxes are initially selected, and low-quality anchor boxes are suppressed. On the other hand, the integration of intersection over union (IoU) prediction and uncertainty prediction constructs a quality evaluation function to guide. In this manner, this function dynamically evaluates the localization and classification ability of each prediction box. Extensive experiments on publicly available datasets such as DOTA1.0, DOTA1.5, DIOR, and MAR20 demonstrate the effectiveness of the proposed model. The source code will be available athttps://github.com/fan609/SOOD.

In the past few years, there has been significant progress in hyperspectral image classification (HSIC). However, when the trained classifier on the source scene is directly applied to a new scene, the classification performance tends to dramatically decrease because of the spectral shift phenomenon. Most existing techniques use feature alignment to learn knowledge from labeled scenes to unlabeled scenes, often overlooking the impact of noisy samples and outliers. To tackle this issue, the classwise prototype-guided alignment network (CPGAN) is proposed for cross-scene HSIC. The core idea is that classwise prototypes across scenes are employed as alignment intermediaries to guide cross-scene feature alignment. Specifically, first, spectral-spatial features from different scenes are extracted with a common feature extractor. Then, an uncertainty-aware pseudolabel selection (UPS) is designed to obtain high-confidence pseudolabels for unlabeled target scenes. Finally, a novel classwise prototype-guided alignment method is proposed to simultaneously achieve interdomain and intradomain alignment (IntraDA). The experimental results conducted on three datasets show that our method achieves superior performance compared to other cutting-edge classification algorithms.

In recent years, enormous studies have been conducted to improve the land cover (LC) classification performance of multimodal remote sensing (RS) data, which outperforms single-modal-based methods by a large margin due to information diversity. To go a step further, we develop a two-branch patch-based convolutional neural network (CNN) with an encoder–decoder (ED) module to fuse multimodal RS data information. A knowledge distillation in model (DIM) module is proposed to guild per-modality encoder learning with the final fused information to enable multimodal data fusion more effectively. Moreover, utilizing multimodal information to guide single-modal learning still remains to be explored. To this end, a knowledge distillation cross-model (DCM) module is designed to improve single-modal LC classification with multimodal knowledge distillation, which bridges the gap between single-modal-based and multimodal-based methods. In particular, the multimodal-based method is taken as a teacher to transfer knowledge to single-modal-based methods. Extensive experiments are carried out on two multimodal RS datasets, including hyperspectral (HS) and light detection and ranging (LiDAR) data, i.e., the Houston2013 dataset, and HS and synthetic aperture radar (SAR) data, i.e., the Berlin dataset. The results demonstrate the effectiveness and superiority of the proposed multimodal fusion strategy in comparison with several state-of-the-art multimodal RS data classification methods. Also, the proposed DCM module improves the LC classification performance of single-modal methods by a large margin.

Hyperspectral images (HSIs) have been widely used for target detection due to their abundant spatial and spectral information. In this article, a shadow-insensitive hyperspectral target detection (HTD) framework based on exposure fusion is proposed, which consists of the following major steps. First, the input HSI is divided into two parts, namely the shadow region and the nonshadow region. Second, total variation-based feature extraction and overexposure operation are performed on the input image to produce two feature images, i.e., the original feature image and the overexposure image. Third, a self-guided constrained energy minimization (SGCEM) detector is performed on the two feature images to detect the targets in shadow and nonshadow regions, respectively. Finally, the detection results obtained on the original feature image and the overexposure image are fused to acquire the final detection result. Extensive experiments conducted on real-world data illustrate that the proposed method can achieve satisfactory results when shadow exists.

Recently, Vision Transformer (ViT)-based deep learning models have achieved remarkable performance gains in hyperspectral image classification (HSIC) due to their abilities to model long-range dependencies and extract global spatial features. However, ViT is built with a stack of Transformer blocks and faces the challenge of learning a large number of parameters when processing hyperspectral data. Besides, the inherent modeling of global correlation in Transformer ignores the effective representation of local spatial and spectral features. To address these issues, we propose a lightweight ViT network known as Groupwise Separable Convolutional Vision Transformer (GSC-ViT). Firstly, a Groupwise Separable Convolution (GSC) module, which is a combination of grouped pointwise convolution and group convolution, is designed to significantly decrease the number of convolutional kernel parameters, and effectively capture local spectral-spatial information in hyperspectral image. Secondly, a Groupwise Separable Multi-Head Self-Attention (GSSA) module is employed to substitute the conventional Multi-Head Self-Attention (MSA) in ViT, in which the Groupwise Self-Attention(GSA) provides local spatial feature extraction, and the Pointwise Self-Attention(PWSA) provides global spatial feature extraction. Thirdly, a simple pointwise layer with enhanced skip connection mechanism is employed to substitute the Multi-Layer Perceptron (MLP) layer in all Transformer blocks of ViT, so as to eliminate unnecessary nonlinear transformations and facilitate the fusion of features derived from GSC and GSSA modules. Extensive experiments on four benchmark hyperspectral datasets reveal that our GSC-ViT can achieve surprising classification performance with relatively few training samples as compared with some existing HSIC approaches. The source code is available at https://github.com/flyzzie/TGRS-GSC-VIT.

Remote sensing super-resolution (SR) technique, which aims to generate high-resolution image with rich spatial details from its low-resolution counterpart, play a vital role in many applications. Recently, more and more studies attempt to explore the application of Transformer in remote sensing field. However, they suffer from the high computational burden and memory consumption for remote sensing super-resolution. In this paper, we propose an efficient Swin Transformer (ESTNet) via channel attention for SR of remote sensing images, which is composed of three components. First, a three-layer convolutional operation is utilized to extract shallow features of the input low-resolution image. Then, a residual group-wise attention module is proposed to extract the deep features, which contains an efficient channel attention block (ECAB) and a group-wise attention block (GAB). Finally, the extracted deep features are reconstructed to generate high-resolution remote sensing images. Extensive experimental results proclaim that the proposed ESTNet can obtain better super-resolution results with low computational burden. Compared to the recently proposed Transformer-based remote sensing super-resolution method, the number of parameters is reduced by 82.68% while the computational cost is reduced by 87.84%. The code of the proposed ESTNet will be available at https://github.com/PuhongDuan/ESTNet for reproducibility.

Interactive image segmentation (IIS) has been widely used in various fields, such as medicine, industry, etc. However, some core issues, such as pixel imbalance, remain unresolved so far. Different from existing methods based on pre-processing or post-processing, we analyze the cause of pixel imbalance in depth from the two perspectives of pixel number and pixel difficulty. Based on this, a novel and unified Click-pixel Cognition Fusion network with Balanced Cut (CCF-BC) is proposed in this paper. On the one hand, the Click-pixel Cognition Fusion (CCF) module, inspired by the human cognition mechanism, is designed to increase the number of click-related pixels (namely, positive pixels) being correctly segmented, where the click and visual information are fully fused by using a progressive three-tier interaction strategy. On the other hand, a general loss, Balanced Normalized Focal Loss (BNFL), is proposed. Its core is to use a group of control coefficients related to sample gradients and forces the network to pay more attention to positive and hard-to-segment pixels during training. As a result, BNFL always tends to obtain a balanced cut of positive and negative samples in the decision space. Theoretical analysis shows that the commonly used Focal and BCE losses can be regarded as special cases of BNFL. Experiment results of five well-recognized datasets have shown the superiority of the proposed CCF-BC method compared to other state-of-the-art methods. The source code is publicly available at https://github.com/lab206/CCF-BC.

Image geo-localization aims to locate a query image from source platform (e.g., drones, street vehicle) by matching it with Geo-tagged reference images from the target platforms (e.g., different satellites). Achieving cross-modal or cross-view real-time (>30fps) image localization with the guaranteed accuracy in a unified framework remains a challenge due to the huge differences in modalities and views between the two platforms. In order to solve this problem, a novel fine-grained overlap estimation based image geo-localization method is proposed in this paper, the core of which is to estimate the salient and subtle overlapping regions in image pairs to ensure correct matching. Specifically, the high-level semantic features of input images are extracted by a deep convolutional neural network. Then, a novel overlap scanning module (OSM) is presented to mine the long-range spatial and channel dependencies of semantic features in various subspaces, thereby identifying fine-grained overlapping regions. Finally, we adopt the triplet ranking loss to guide the proposed network optimization so that the matching regions are as close as possible and the most mismatched regions are as far away as possible. To demonstrate the effectiveness of our FOENet, comprehensive experiments are conducted on three cross-view benchmarks and one cross-modal benchmark. Our FOENet yields better performance in various metrics and the recall accuracy at top 1 (R@1) is significantly improved, with a maximum improvement of 70.6%. In addition, the proposed model runs fast on a single RTX 6000, reaching real-time inference speed on all datasets, with the fastest being 82.3 FPS.

Integration of diverse visual prompts like clicks, scribbles, and boxes in interactive image segmentation significantly facilitates users' interaction as well as improves interaction efficiency. However, existing studies primarily encode the position or pixel regions of prompts without considering the contextual areas around them, resulting in insufficient prompt feedback, which is not conducive to performance acceleration. To tackle this problem, this paper proposes a simple yet effective Probabilistic Visual Prompt Unified Transformer (PVPUFormer) for interactive image segmentation, which allows users to flexibly input diverse visual prompts with the probabilistic prompt encoding and feature post-processing to excavate sufficient and robust prompt features for performance boosting. Specifically, we first propose a Probabilistic Prompt-unified Encoder (PPuE) to generate a unified one-dimensional vector by exploring both prompt and non-prompt contextual information, offering richer feedback cues to accelerate performance improvement. On this basis, we further present a Prompt-to-Pixel Contrastive (P2C) loss to accurately align both prompt and pixel features, bridging the representation gap between them to offer consistent feature representations for mask prediction. Moreover, our approach designs a Dual-cross Merging Attention (DMA) module to implement bidirectional feature interaction between image and prompt features, generating notable features for performance improvement. A comprehensive variety of experiments on several challenging datasets demonstrates that the proposed components achieve consistent improvements, yielding state-of-the-art interactive segmentation performance. Our code is available at https://github.com/XuZhang1211/PVPUFormer.

Multi-focus image fusion can extract the focus regions from different source images and combine them into a fully clear image. Existing unsupervised methods typically use gradient information to measure the focus regions in images and generate a fusion weight map, but ordinary gradient operators are difficult to measure information accurately in regions with weaker textures. In addition, using only gradient information as a constraint cannot make the model fully distinguish all the focus regions in the image, which seriously restricts the clarity of the fusion image. To address these issues, a novel unsupervised multi-focus image fusion method is proposed in this paper. Specifically, a neighborhood information fusion network is designed to generate an initial fusion weight map. It can capture features within different neighborhood ranges at once, which enhances the information association between different regions. In addition, to further improve the feature extraction ability of the model in the regions with low texture information, a local difference evaluation loss function is proposed. It is combined with the gradient measure loss function to constrain the network. Finally, a fusion weight optimization module is proposed to improve the clarity of the fusion image in the repeated defocusing regions and overexposed regions of different source images, which redistributes the weights of different source images. The proposed fusion method is compared with advanced methods on three public multi-focus datasets. Experimental results indicate that the proposed method has achieved better performance in qualitative and quantitative aspects.

Visible-infrared object detection aims to improve the detector performance by fusing the complementarity of visible and infrared images. However, most existing methods only use local intramodality information to enhance the feature representation while ignoring the efficient latent interaction of long-range dependence between different modalities, which leads to unsatisfactory detection performance under complex scenes. To solve these problems, we propose a feature-enhanced long-range attention fusion network (LRAF-Net), which improves detection performance by fusing the long-range dependence of the enhanced visible and infrared features. First, a two-stream CSPDarknet53 network is used to extract the deep features from visible and infrared images, in which a novel data augmentation (DA) method is designed to reduce the bias toward a single modality through asymmetric complementary masks. Then, we propose a cross-feature enhancement (CFE) module to improve the intramodality feature representation by exploiting the discrepancy between visible and infrared images. Next, we propose a long-range dependence fusion (LDF) module to fuse the enhanced features by associating the positional encoding of multimodality features. Finally, the fused features are fed into a detection head to obtain the final detection results. Experiments on several public datasets, i.e., VEDAI, FLIR, and LLVIP, show that the proposed method obtains state-of-the-art performance compared with other methods.

To obtain a high-resolution hyperspectral image (HR-HSI), fusing a low-resolution hyperspectral image (LR-HSI) and a high-resolution multispectral image (HR-MSI) is a prominent approach. Numerous approaches based on convolutional neural networks (CNNs) have been presented for hyperspectral image (HSI) and multispectral image (MSI) fusion. Nevertheless, these CNN-based methods may ignore the global relevant features from the input image due to the geometric limitations of convolutional kernels. To obtain more accurate fusion results, we provide a spatial-spectral transformer-based U-net (SSTF-Unet). Our SSTF-Unet can capture the association between distant features and explore the intrinsic information of images. More specifically, we use the spatial transformer block (SATB) and spectral transformer block (SETB) to calculate the spatial and spectral self-attention, respectively. Then, SATB and SETB are connected in parallel to form the spatial-spectral fusion block (SSFB). Inspired by the U-net architecture, we build up our SSTF-Unet through stacking several SSFBs for multiscale spatial-spectral feature fusion. Experimental results on public HSI datasets demonstrate that the designed SSTF-Unet achieves better performance than other existing HSI and MSI fusion approaches.

In recent years, deep-learning-based pixel-level unified image fusion methods have received more and more attention due to their practicality and robustness. However, they usually require a complex network to achieve more effective fusion, leading to high computational cost. To achieve more efficient and accurate image fusion, a lightweight pixel-level unified image fusion (L-PUIF) network is proposed. Specifically, the information refinement and measurement process are used to extract the gradient and intensity information and enhance the feature extraction capability of the network. In addition, these information are converted into weights to guide the loss function adaptively. Thus, more effective image fusion can be achieved while ensuring the lightweight of the network. Extensive experiments have been conducted on four public image fusion datasets across multimodal fusion, multifocus fusion, and multiexposure fusion. Experimental results show that L-PUIF can achieve better fusion efficiency and has a greater visual effect compared with state-of-the-art methods. In addition, the practicability of L-PUIF in high-level computer vision tasks, i.e., object detection and image segmentation, has been verified.

Spoken language understanding (SLU) is the core of the speech-centric human-robot interaction system, which mainly involves intent detection and slot filling. The recent SLU research focuses on the joint modeling of the two tasks due to their correlation. Furthermore, the slot information consists of slot position and slot type. Although the slot types are semantically related to the intent, the slot positions of the same intent may vary a lot in different utterances due to the diversity of spoken language. Thus, the conventional one-stage slot filling task may introduce unrelated information for slot position prediction in the slot-intent interaction of the joint modeling. Therefore, we propose a novel two-stage selective fusion framework for joint intent detection and slot filling. Unlike the previous one-stage framework, the proposed framework decomposes the slot filling into two stages, i.e., the slot proposal and slot classification. The slot proposal network consisting of BERT and bidirectional long short-term memory (Bi-LSTM)-conditional random field (CRF) predicts the slot positions. Instead of the tokenwise fusion in the existing methods, the slot-intent feature fusion is only performed in the slot classification. A selective fusion mechanism is designed to facilitate the slot-intent interaction within each slot candidate for more accurate slot-type classification. Experiments on five standard benchmarks (i.e., ATIS, SNIPS, MixATIS, MixSNIPS, and DSTC4) show that the proposed framework achieves the best performance in comparison with several state-of-the-art methods.

Camouflaged object detection (COD) aims to identify object pixels visually embedded in the background environment. Existing deep learning methods fail to utilize the context information around different pixels adequately and efficiently. In order to solve this problem, a novel pixel-centric context perception network (PCPNet) is proposed, the core of which is to customize the personalized context of each pixel based on the automatic estimation of its surroundings. Specifically, PCPNet first employs an elegant encoder equipped with the designed vital component generation (VCG) module to obtain a set of compact features rich in low-level spatial and high-level semantic information across multiple subspaces. Then, we present a parameter-free pixel importance estimation (PIE) function based on multiwindow information fusion. Object pixels with complex backgrounds will be assigned with higher PIE values. Subsequently, PIE is utilized to regularize the optimization loss. In this way, the network can pay more attention to those pixels with higher PIE values in the decoding stage. Finally, a local continuity refinement module (LCRM) is used to refine the detection results. Extensive experiments on four COD benchmarks, five salient object detection (SOD) benchmarks, and five polyp segmentation benchmarks demonstrate the superiority of PCPNet with respect to other state-of-the-art methods.

Hyperspectral image (HSI) classification methods have made great progress in recent years. However, most of these methods are rooted in the closed-set assumption that the class distribution in the training and testing stages is consistent, which cannot handle the unknown class in open-world scenes. In this work, we propose a feature consistency-based prototype network (FCPN) for open-set HSI classification, which is composed of three steps. First, a three-layer convolutional network is designed to extract the discriminative features, where a contrastive clustering module is introduced to enhance the discrimination. Then, the extracted features are used to construct a scalable prototype set. Finally, a prototype-guided open-set module (POSM) is proposed to identify the known samples and unknown samples. Extensive experiments reveal that our method achieves remarkable classification performance over other state-of-the-art classification techniques.

We introduce a new task, named video corpus visual answer localization (VCVAL), which aims to locate the visual answer in a large collection of untrimmed instructional videos using a natural language question. This task requires a range of skills - the interaction between vision and language, video retrieval, passage comprehension, and visual answer localization. In this paper, we propose a cross-modal contrastive global-span (CCGS) method for the VCVAL, jointly training the video corpus retrieval and visual answer localization subtasks with the global-span matrix. We have reconstructed a dataset named MedVidCQA, on which the VCVAL task is benchmarked. Experimental results show that the proposed method outperforms other competitive methods both in the video corpus retrieval and visual answer localization sub-tasks. Most importantly, we perform detailed analyses on extensive experiments, paving a new path for understanding the instructional videos, which ushers in further research1.

Previous methods for dynamic facial expression recognition (DFER) in the wild are mainly based on Convolutional Neural Networks (CNNs), whose local operations ignore the long-range dependencies in videos. Transformer-based methods for DFER can achieve better performances but result in higher FLOPs and computational costs. To solve these problems, the local-global spatio-temporal Transformer (LOGO-Former) is proposed to capture discriminative features within each frame and model contextual relationships among frames while balancing the complexity. Based on the priors that facial muscles move locally and facial expressions gradually change, we first restrict both the space attention and the time attention to a local window to capture local interactions among feature tokens. Furthermore, we perform the global attention by querying a token with features from each local window iteratively to obtain long-range information of the whole video sequence. In addition, we propose the compact loss regularization term to further encourage the learned features have the minimum intra-class distance and the maximum inter-class distance. Experiments on two in-the-wild dynamic facial expression datasets (i.e., DFEW and FERV39K) indicate that our method provides an effective way to make use of the spatial and temporal dependencies for DFER.

Understanding and elucidating human behavior across diverse scenarios represents a pivotal research challenge in pursuing seamless human-computer interaction. However, previous research on multi-participant dialogues has mostly relied on proprietary datasets, which are not standardized and openly accessible. To propel advancements in this domain, the MultiMediate'23 Challenge presents two sub-challenges: Eye contact detection and Next speaker prediction, aiming to foster a comprehensive understanding of multi-participant behavior. To tackle these challenges, we propose a multi-scale conformer fusion network (MSCFN) for enhancing the perception of multi-participant group behaviors. The conformer block combines the strengths of transformers and convolution networks to facilitate the establishment of global and local contextual relationships between sequences. Then the output features from all Conformer blocks are concatenated to fusion multi-scale representations. Our proposed method was evaluated using the officially provided dataset, and it achieves the best and second best performance in next speaker prediction and gaze detection tasks of MultiMediate'23, respectively.

Fusing hyperspectral images (HSIs) with multispectral images (MSIs) of higher spatial resolution has become an effective way to sharpen HSIs. Recently, deep convolutional neural networks (CNNs) have achieved promising fusion performance. However, these methods often suffer from the lack of training data and limited generalization ability. To address the above problems, we present a zero-shot learning (ZSL) method for HSI sharpening. Specifically, we first propose a novel method to quantitatively estimate the spectral and spatial responses of imaging sensors with high accuracy. In the training procedure, we spatially subsample the MSI and HSI based on the estimated spatial response and use the downsampled HSI and MSI to infer the original HSI. In this way, we can not only exploit the inherent information in the HSI and MSI, but the trained CNN can also be well generalized to the test data. In addition, we take the dimension reduction on the HSI, which reduces the model size and storage usage without sacrificing fusion accuracy. Furthermore, we design an imaging model-based loss function for CNN, which further boosts the fusion performance. The experimental results show the significantly high efficiency and accuracy of our approach.

Recently, fusing a low-resolution hyperspectral image (LR-HSI) with a high-resolution multispectral image (HR-MSI) of different satellites has become an effective way to improve the resolution of an HSI. However, due to different imaging satellites, different illumination, and adjacent imaging time, the LR-HSI and HR-MSI may not satisfy the observation models established by existing works, and the LR-HSI and HR-MSI are hard to be registered. To solve the above problems, we establish new observation models for LR-HSIs and HR-MSIs from different satellites, then a deep-learning-based framework is proposed to solve the key steps in multi-satellite HSI fusion, including image registration, blur kernel learning, and image fusion. Specifically, we first construct a convolutional neural network (CNN), called RegNet, to produce pixel-wise offsets between LR-HSI and HR-MSI, which are utilized to register the LR-HSI. Next, according to the new observation models, a tiny network, called BKLNet, is built to learn the spectral and spatial blur kernels, where the BKLNet and RegNet can be trained jointly. In the fusion part, we further train a FusNet by downsampling the registered data with the learned spatial blur kernel. Extensive experiments demonstrate the superiority of the proposed framework in HSI registration and fusion accuracy.

Facial Expression Recognition (FER) in the wild is extremely challenging due to occlusions, variant head poses, face deformation and motion blur under unconstrained conditions. Although substantial progresses have been made in automatic FER in the past few decades, previous studies were mainly designed for lab-controlled FER. Real-world occlusions, variant head poses and other issues definitely increase the difficulty of FER on account of these information-deficient regions and complex backgrounds. Different from previous pure CNNs based methods, we argue that it is feasible and practical to translate facial images into sequences of visual words and perform expression recognition from a global perspective. Therefore, we propose the Visual Transformers with Feature Fusion (VTFF) to tackle FER in the wild by two main steps. First, we propose the attentional selective fusion (ASF) for leveraging two kinds of feature maps generated by two-branch CNNs. The ASF captures discriminative information by fusing multiple features with the global-local attention. The fused feature maps are then flattened and projected into sequences of visual words. Second, inspired by the success of Transformers in natural language processing, we propose to model relationships between these visual words with the global self-attention. The proposed method is evaluated on three public in-the-wild facial expression datasets (RAF-DB, FERPlus and AffectNet). Under the same settings, extensive experiments demonstrate that our method shows superior performance over other methods, setting new state of the art on RAF-DB with 88.14%, FERPlus with 88.81% and AffectNet with 61.85%. The cross-dataset evaluation on CK+ shows the promising generalization capability of the proposed method.

Oil spill detection has attracted increasing attention in recent years since marine oil spill accidents severely affect environments, natural resources, and the lives of coastal inhabitants. Hyperspectral remote sensing images provide rich spectral information which is beneficial for the monitoring of oil spills in complex ocean scenarios. However, most of the existing approaches are based on supervised and semi-supervised frameworks to detect oil spills from hyperspectral images (HSIs), which require a massive amount of effort to annotate a certain number of high-quality training sets. In this study, we make the first attempt to develop an unsupervised oil spill detection method based on isolation forest for HSIs. First, a Gaussian statistical model is designed to remove the bands corrupted by severe noise. Then, kernel principal component analysis (KPCA) is employed to reduce the high dimensionality of the HSIs. Next, the probability of each pixel belonging to one of the classes of seawater and oil spills is estimated with the isolation forest, and a set of pseudo-labeled training samples is automatically produced using the clustering algorithm on the detected probability. Finally, an initial detection map can be obtained by performing the support vector machine (SVM) on the dimension-reduced data, and the initial detection result is further optimized with the extended random walker (ERW) model so as to improve the detection accuracy of oil spills. Experiments on hyperspectral oil spill data (HOSD) created by ourselves demonstrate that the proposed method obtains superior detection performance with respect to other state-of-the-art detection approaches. We will make HOSD and our developed library for oil spill detection publicly available at https://github.com/PuhongDuan/HOSD to further promote this research topic.

Multispectral image pan-sharpening aims to increase the spatial details of multispectral images by fusing multispectral and panchromatic images. Existing component substitution-based deep learning pan-sharpening is generally regarded as a black box and fails to mine the image interaction relation with physical significance in each step of pan-sharpening, which not only limits the improvement of image resolution, but also ignores the physical interpretability of the models. To improve this situation, according to the traditional component substitution-based detail injection pan-sharpening model, we consider the matrix calculation in each step as the transformation between image pixel values and carry out linear transformations, and therefore the pan-sharpened multispectral image is represented as the sum of two multispectral images. Then given the spatial and spectral heterogeneity, the two summed images are decomposed based on the fact that any real number can be expressed as the product of two real numbers. Ultimately, the multispectral image pan-sharpening model can be constructed as the sum of two Hadamard products. We design a dual-branch network with attention mechanisms that merges the sum and the Hadamard products into a concise formulation. This method not only enhances physical interpretability but also improves spatial resolution. Experiments on five real-world datasets validate that the proposed multispectral image pan-sharpening model can improve performance.

Hyperspectral oil spill mapping aims to distinguish the type of oil spill. Recently, most hyperspectral oil spill detection methods are based on supervised methods that work well with rich training samples. However, in the marine oil spill detection scenario, pixel annotations are difficult and costly. Moreover, the labels obtained by domain experts within a hyperspectral image (HSI) are often scarce. To address these issues, a self-supervised spectral-spatial transformer network is proposed for hyperspectral oil spill mapping. First, we propose a transformer-based contrastive learning network to extract the deep discriminative features. Then, the learned features are transferred to the downstream classification network that is fine-tuned with very few labeled samples. Experiments on hyperspectral oil spill database (HOSD) constructed by ourselves indicate that the proposed method can obtain more promising performance than several state-of-the-art oil spill classification techniques in discriminating different types of oil spills, i.e., thick oil, thin oil, sheen, and seawater.

Cloud-free image reconstruction is of great significance for improving the quality of optical satellite images that are vulnerable to bad weather. When cloud cover makes it impossible to obtain information under the cloud, auxiliary data is indispensable to guide the reconstruction of the cloud-contaminated area. Additionally, the areas that require continuous observation are mostly regions with complex features, which puts higher demands on the restoration of texture, color, and other details in data reconstruction. In this paper, we propose a Transformer-based generative adversarial network for cloud-free multispectral image reconstruction via multi-sensor data fusion in satellite images (TransGAN-CFR). Synthetic Aperture Radar (SAR) images that are not affected by clouds are used as auxiliary data and paired with cloudy optical images into the GAN generator. To take advantage of the deep-shallow features and global-local geographical proximity in remote sensing images, the proposed generator employs a hierarchical Encoder-Decoder structure, in which the Transformer blocks adopt a non-overlapping window multi-head self-attention (WMSA) mechanism and a modified feed-forward network though depth-wise convolutions and the gating mechanism. Besides, we introduce a Triplet loss function specifically designed for cloud removal tasks to provide the generated cloud-less image with greater proximity to the ground truth. Compared with seven state-of-the-art deep learning-based cloud removal models, our network can yield more natural cloud-free images with better visual performance and more accurate results in quantitative evaluation on the SEN12MS-CR dataset.

Due to the impact of climate change, the frequency of flood events has increased in recent years, which puts forward an urgent need for timely and accurate flood mapping for emergency response. As the synthetic aperture radar (SAR) enables all-time monitoring regardless of bad weather conditions, it fits far better than passive optical sensors to delineate submerged areas during flood events. However, the universal, rapid, and accurate detection of flood extent remains a challenge. Drawing inspiration from the analysis of time-series variation in representative ground objects caused by flood events, as observed in a dual-polarized SAR time series over a hydrological year, we construct a novel window-based variation model. This model can be used to capture both long-term trends and short-term fluctuations of flood features across different polarization modes. Subsequently, we introduce an unsupervised flood mapping framework that integrates spatiotemporal flood features extracted by fuzzy-based methods. Given the distinct backscatter value of short vegetation, a flooded short vegetation activation model is designed and performed to enhance flood mapping accuracy in complex regions. The proposed method, tested on the 2020 East Dongting Lake flood in China, surpasses three unsupervised flood mapping methods and two deep learning methods in terms of quantitative evaluation and visual performance. The uncertainty of our proposed framework is tested through parameter sensitivity analyses, comparisons with flood mapping results from other sensor images, and extensive experiments on floods at different locations and times, thereby demonstrating its effectiveness, stability, and universality.

The spaceborne hyperspectral (HS) imagery with fine spectral information has broad application aspects; however, the low spatial resolution has limited the potential application values. Over the past few decades, a general strategy to improve the spatial resolution of the HS is to fuse the low spatial resolution (LR) HS with an auxiliary moderate spatial resolution (MR) multispectral (MS) or a high spatial resolution (HR) panchromatic (PAN) image. However, most of the existing methods mainly focus on two-sensor fusion with the LR HS and MR MS images (i.e., MS-HS fusion) or the LR HS and HR PAN images (i.e., the PAN-HS fusion). How to comprehensively combine the complementary spatial and spectral advantages of the LR HS, MR MS, and HR PAN observations, to obtain the optimal high-fidelity HR HS image is interesting and challenging. In this paper, we propose a multi-scale grouping feedback embedded integrated fusion network (MGFEI-Net) for the LR HS, MR MS, and HR PAN images. Specifically, an attention-based hybrid-scale integrated module is designed by considering the spatial scale diversity of the HR PAN, MR MS, and LR HS images. Moreover, a multi-scale grouping feedback embedded module with a top-to-bottom manner is proposed to capture more usual spatial-spectral features. Experiments were performed on the simulated and real datasets. Moreover, the robustness of the proposed PAN-MS-HS fusion under different large spatial resolution ratios (such as 8, 16, 32, 64) was analyzed. The experimental results demonstrated the competitive performance of the proposed method.

Single-modal image registration methods are generally not feasible for visible and infrared images. Besides, multi-modal image registration methods still suffer from uneven distribution of extracted features, low repeatability, and ambiguous features. To address these issues, a coarse-to-fine infrared and visible image registration approach for dual sensor UAV imaging system is proposed, which is resilient to the difference of focal lengths and field of view. First, in the coarse registration step, the infrared image is transformed to the same scale as the visible image by using the similarity transformation. This operation makes the proposed method robust to the variation of field of view. Then, the feature point pairs are initialized using feature detectors in the infrared image’s blocked phase congruency feature map. Next, the feature point pairs are optimized by estimating the offset based on the relationship between the constructed feature descriptors. Finally, using elastic deformation, the pixel-level registered infrared image is obtained. Extensive experiments demonstrate the superior performance of the proposed coarse-to-fine image registration methodology in the real infrared-visible image pairs. The code and dataset are available at https://drive.google.com/drive/folders/1mpUWwHUbKTrBdOrNMNRRnuJclDUAC7nU?usp=sharing.

Change detection (CD) aims to find areas of specific changes in multi-temporal remote sensing images. The existing methods fail to adequately explore the cross-temporal global context, making the establishment of spatial-temporal deep global associations insufficient and inefficient. As a result, their performance is vulnerable to complex and various objects in changing scenes. Hence, we propose a cross-temporal context learning network, termed as CCLNet, where the intra- and inter-temporal long-range dependency are mined and interactively fused, to fully exploit the cross-temporal context information. Specifically, a lightweight convolutional neural network is first used to extract deep semantic features. Then, a well-designed cross-temporal fusion transformer (CFT) is proposed to locate the changing objects in the scene by establishing the long-range dependency across bitemporal images. Thanks to this, the temporal-specific information extraction and cross-temporal information integration are seamlessly integrated into the same network, thereby significantly improving the discriminative features of changing objects. Furthermore, this allows us using naive backbones with low computational cost to achieve reliable CD performance. Experiments on mainstream benchmarks show that our proposed method can handle CD task faster than state-of-the-art methods while maintaining better or comparable matching accuracy on a single RTX3090.

With the improvement of imaging equipment, hyperspectral underwater target detection (HUTD) has raised much interest in recent years. The existing HUTD methods do not fully utilize spectral characteristics and need prior information about targets. Besides, the detection performance lacks verification in natural scenarios. In this paper, the authors propose a Feature Bands based Unsupervised underwater target Detection method (FBUD), which aims at finding the optimal feature bands to identify the underwater target near the coastline. Specifically, the normalized difference water index (NDWI) and unmixing technique are adopted to find the target and background pixels. Then, the spectral difference between the target and background is used to find the feature bands. With a simple and fast math operation of the feature bands, the probability map of the underwater target can be easily obtained. Besides, a new unmanned aerial vehicle (UAV)-borne hyperspectral image dataset named HNU-UTD is built for underwater target detection in real-world scenes. Experimental results obtained with the HNU-UTD dataset confirm the accuracy and effectiveness of the proposed detection method, which even outperforms supervised detection methods.

Integrating a low-resolution hyperspectral image and a high-resolution multispectral image is widely acknowledged as an effective approach for generating a high-resolution hyperspectral image. Recent studies have highlighted the nuclear norm as an efficient method for this problem through the utilization of low-rankness. However, the standard nuclear norm has a limitation due to treating singular values equally. To address this issue, we have incorporated the concept of the weighted nuclear norm from the image denoising problem into hyperspectral image fusion, ensuring the retention of crucial data components. Furthermore, we propose a unified framework which integrates the weighted nuclear norm, a sparse prior, and total variation regularization. This framework utilizes the ℓ1norm of coefficients to promote spatial-spectral sparsity in the fused images, while total variation is employed to preserve the spatial piecewise smooth structure. To efficiently solve the proposed model, we have designed an alternating direction method of multipliers. The experimental results show that our proposed approach surpasses the state-of-the-art methods.

Nowadays, convolutional neural network (CNN)-based deep learning models have been popularized in hyperspectral image classification (HSIC) and achieved significant accuracy gains, which is due to their hierarchical and nonlinear feature learning patterns. However, too deeper network structures may induce a huge amount of parameters and excessive computing overhead, leading to the need for plenty of labeled samples for training. Besides, highly abstract semantic features may not be the most suitable for hyperspectral land-cover classification tasks. To address these issues, we propose a fairly lightweight network model for HSIC, which is built on a type of exquisitely designed convolution module, namelygrouped separable convolution. Compared with the standard convolution, the designed grouped separable convolution module combines grouped convolution with point-wise convolution, which not only greatly reduces the number of parameters of convolution kernels, but also caters to the inherent 3D cube style of hyperspectral image data. Moreover, Gabor filters are introduced to modulate the grouped separable convolution kernels, so as to further use relatively few convolution kernels with additional prior orientation and scale information for feature extraction. The experiments are carried out on four real hyperspectral datasets, and the experimental results reveal that the proposed model has low training cost and memory overhead. Compared with some existing deep network models that have been applied to HSIC, our proposed model can achieve competitive classification accuracy with fewer training samples.

Camouflaged object detection (COD) aims to discover objects that blend in with the background due to similar colors or textures, etc. Existing deep learning methods do not systematically illustrate the key tasks in COD, which seriously hinders the improvement of its performance. In this paper, we introduce the concept of focus areas that represent some regions containing discernable colors or textures, and develop a two-stage focus scanning network for camouflaged object detection. Specifically, a novel encoder-decoder module is first designed to determine a region where the focus areas may appear. In this process, a multi-layer Swin transformer is deployed to encode global context information between the object and the background, and a novel cross-connection decoder is proposed to fuse cross-layer textures or semantics. Then, we utilize the multi-scale dilated convolution to obtain discriminative features with different scales in focus areas. Meanwhile, the dynamic difficulty aware loss is designed to guide the network paying more attention to structural details. Extensive experimental results on the benchmarks, including CAMO, CHAMELEON, COD10K, and NC4K, illustrate that the proposed method performs favorably against other state-of-the-art methods.

Automatic speech recognition (ASR) is the major human-machine interface in many intelligent systems, such as intelligent homes, autonomous driving, and servant robots. However, its performance usually significantly deteriorates in the presence of external noise, leading to limitations of its application scenes. The audio-visual speech recognition (AVSR) takes visual information as a complementary modality to enhance the performance of audio speech recognition effectively, particularly in noisy conditions. Recently, the transformer-based architectures have been used to model the audio and video sequences for the AVSR, which achieves a superior performance. However, its performance may be degraded in these architectures due to extracting irrelevant information while modeling long-term dependences. In addition, the motion feature is essential for capturing the spatio-temporal information within the lip region to best utilize visual sequences but has not been considered in the AVSR tasks. Therefore, we propose a multimodal sparse transformer network (MMST) in this article. The sparse self-attention mechanism can improve the concentration of attention on global information by selecting the most relevant parts wisely. Moreover, the motion features are seamlessly introduced into the MMST model. We subtly allow motion-modality information to flow into visual modality through the cross-modal attention module to enhance visual features, thereby further improving recognition performance. Extensive experiments conducted on different datasets validate that our proposed method outperforms several state-of-the-art methods in terms of the word error rate (WER).

In the last few years, enormous strides have been made for object detection and data association, which are vital subtasks for one-stage online multi-object tracking (MOT). However, the two separated submodules involved in the whole MOT pipeline are processed or optimized separately, resulting in a complex method design and requiring manual settings. In addition, few works integrate the two subtasks into a single end-to-end network to optimize the overall task. In this study, we propose an end-to-end MOT network called joint detection and association network (JDAN) that is trained and inferred in a single network. All layers in JDAN are differentiable, and can be optimized jointly to detect targets and output an association matrix for robust multi-object tracking. What’s more, we generate suitable pseudo-labels to address the data inconsistency between object detection and association. The detection and association submodules could be optimized by the composite loss function that is derived from the detection results and the generated pseudo association labels, respectively. The proposed approach is evaluated on two MOT challenge datasets, and achieves promising performance compared with classic and latest methods.

Human behavior analysis in multi-person conversations has been one of the most important research issues for natural human-robot interaction. However, previous datasets and studies mainly focus on single-person behavior analysis, therefore, can hardly be generalized in real-world application scenarios. Fortunately, the MultiMediate'22 Challenge provides various video clips of multi-party conversations. In this paper, we present a unified network named TA-CNN for both sub-challenges. Our TA-CNN can not only model the spatio-temporal dependencies for eye contact detection, but also capture the group-level discriminative features for multi-label next speaker prediction. We empirically evaluate the performance of our method on the officially provided datasets. Our method achieves the state-of-the-art result of 0.7261 for eye contact detection in terms of accuracy and the UAR of 0.5965 for next speaker prediction on the corresponding test sets.

Cropland segmentation is one of the critical techniques in the agriculture Remote Sensing (RS). Although the Deep Learning (DL) methods have achieved remarkable performance in the natural vision, the cropland segmentation of RS images still suffers from cropland adhesion due to the interference from the surrounding environment and the cropland cover. To tackle this problem, this letter proposes a two stage DL method with spectral-spatial priors. In the first stage, the Multi-feature Extraction Module (MEM) is designed to predict the boundary, an important spatial prior of the cropland. In the second stage, the spatial prior is further fused with the spectral prior by MEMs to get accurate cropland prediction. To evaluate the effectiveness and robustness of the proposed method, we construct a data set called Jiaxiang Cropland Set (JCS) and propose a region level evaluation indicator namely the Plot Mean Intersection over Union (PMIoU). The experiment results on the JCS demonstrate that the proposed method is both qualitatively and quantitatively competitive compared with the state-of-the-art methods.

Pansharpening aims to sharpen a low spatial resolution (LR) multispectral (MS) image using a high spatial resolution (HR) panchromatic (PAN) image to obtain the HR MS image. Though large numbers of pansharpening methods have been proposed, and many advanced methods have shown high quantitative results, few of them are widely used in real applications. This may be attributed to their instability for different images with different ground surface features, or the complexity to be implemented and the time-consuming process for some state-of-the-art methods. In this letter, we proposed a simple, adjustable, and robust fusion (SARF) method. In the proposed method, a spatial-spectral coenhanced strategy was proposed, and several details of the proposed fusion model were specifically designed for the “simple, adjustable, robust” features. It was tested and verified by four-band and eight-band MS images based on reduced resolution (RR) and full resolution (FR) experiments. The experimental results demonstrated the promising spatial visuality of the proposed method, and the spectral fidelity was more robust than most of component substitution (CS)-based and multiresolution analysis (MRA)-based methods.

Recently, learning-based representation techniques have been well exploited for grayscale face image hallucination. For color images, the previous methods only handle the luminance component or each color channel individually, without considering the abundant correlations among different channels as well as the inherent geometrical structure of data manifold. In this article, we propose a learning-based model in quaternion space with graph representation for color face hallucination. Instead of the spatial domain, the color image is represented in the quaternion domain to preserve correlations among different color channels. Moreover, a quaternion graph is learned to smooth the quaternion feature space, which helps to not only stabilize the linear system but also enclose the inherent topology structure of quaternion patch manifold. Besides, considering that single low-resolution (LR) image patch can just provide limited informative information in representation, we propose to simultaneously encode the query smaller LR patch as well as a larger patch containing the surrounding pixels seated at the same position in the objective. The larger patch with rich patterns is used to compensate the lost information in the query LR patch, which further enhances the manifold consistency assumption between the LR and HR patch spaces. The experimental results demonstrated the efficiency of the proposed method in hallucinating color face images.

Hyperspectral images (HSIs) are inevitably contaminated by the mixed noise (such as Gaussian noise, impulse noise, deadlines, and stripes), which could influence the subsequent processing accuracy. Generally, HSI restoration can be transformed into the low-rank matrix recovery (LRMR). In the LRMR, the nuclear norm is widely used to substitute the matrix rank, but its effectiveness is still worth improving. Besides, the$l_{0}$-norm cannot capture the sparse noise’s structured sparsity property. To handle these issues, the adaptive rank and structured sparsity corrections (ARSSC) are presented for HSI restoration. The ARSSC introduces two convex regularizers, that is: 1) the rank correction (RC) and 2) the structured sparsity correction (SSC), to, respectively, approximate the matrix rank and the$l_{2,0}$-norm. The RC and the SSC can adaptively offset the penalization of large entries from the nuclear norm and the$l_{2,1}$-norm, respectively, where the larger the entry, the greater its offset. Therefore, the proposed ARSSC achieves a tighter approximation of the noise-free HSI low-rank structure and promotes the structured sparsity of sparse noise. An efficient alternative direction method of multipliers (ADMM) algorithm is applied to solve the resulting convex optimization problem. The superiority of the ARSSC in terms of the mixed noise removal and spatial-spectral structure information preserving, is demonstrated by several experimental results both on simulated and real datasets, compared with other state-of-the-art HSI restoration approaches.

Building change detection is a very important application in the field of remote sensing. Recently, deep learning (DL) has been introduced to solve the change detection task and achieved good performance, mainly due to the capability of automatically learning deep features. However, the lack of using prior knowledge (e.g., edge structure information) leads to inaccurate detection results, especially in the areas of building boundaries. To solve this problem, an end-to-end DL method for building change detection, named by edge-guided recurrent convolutional neural network (EGRCNN), is proposed in this article. The main idea is to incorporate both discriminative information and edge structure prior in one framework to improve change detection results, especially to generate more accurate building boundaries. First, a siamese convolutional neural network is trained to simultaneously extract primary multilevel features from multitemporal images. Then, a difference analysis module (DAM) is introduced to further produce discriminative features, which is constructed based on the basic long short-term memory module. Finally, both the discriminative features and the estimated edge structure information are jointly exploited to predict building change map. On one hand, the proposed DAM helps to enhance the discrimination between the changed and unchanged regions. On the other hand, the prior edge information is used to push the predicted changed buildings to preserve the original structure, which can further improve the accuracy of building change detection. Experimental results demonstrate that the performance of the proposed method outperforms several state-of-the-art approaches, in terms of objective metrics and visual comparison results.

Hyperspectral images (HSI) contain rich spatial and spectral detail information, while light detection and ranging (LiDAR) data can provide the elevation information. Thus, the fusion of HSI and LiDAR data can help for more accurate image classification, which becomes a hot research topic. However, it is difficult to capture complex local and global spatial-spectral associations, meanwhile, how to build an effective interaction between multi-modal data is another important issue. To this end, a novel global-local transformer network (GLT-Net) is proposed for the joint classification of HSI and LiDAR data, in this paper. The main idea is to fully exploit the advantage of the convolution operator in characterizing locally correlated features and the promising capability of transformer architecture in learning long-range dependencies. Moreover, multi-scale feature fusion and probabilistic decision fusion strategies are also designed in one framework, in order to further improve classification performance. Here, the proposed GLT-Net mainly consists of multi-scale local spatial feature learning, global spectral feature learning, and global-local feature fusion classification. In specific, multi-modal image cubes of different sizes are firstly extracted and sent into convolutional neural networks (CNNs) to learn local spatial features, which is followed by multi-modal information propagation and spatial-attention guided multi-scale feature fusion. Afterwards, by considering spectral feature channels from a sequential perspective, vision transformers are introduced to model the global spectral dependencies. Finally, multiple class estimations based on local and global features are integrated via a probabilistic decision fusion strategy. In this way, complementary information of multi-modal data as well as local/global spectral-spatial information can be fully mined and jointly utilized. Extensive experiments on three popular HSI and LiDAR datasets demonstrate that the proposed method performs superiority over state-of-the-art methods. The source code of the proposed method will be made publicly available at https://github.com/Ding-Kexin/GLT-Net.

Shadow removal is a challenging problem in hyperspectral remote sensing images due to its spatial-variant properties and diverse patterns. In this work, a shadow removal framework with multiexposure fusion is proposed for hyperspectral remote sensing images, which consists of three major steps. First, a color space conversion method is exploited to detect the shadow regions. Second, the principle of the intrinsic decomposition model is utilized to generate a set of differently exposed hyperspectral images (HSIs), i.e., multiexposure images. Third, the generated multiexposure images and the original HSIs are fused together with a two-stage image fusion method so as to remove the shadows in hyperspectral remote sensing images effectively. Experiments performed on three real hyperspectral datasets confirm that the performance of the proposed method outperforms other state-of-the-art shadow removal approaches.

Fusion of hyperspectral images with low-spatial and high-spectral resolution and multispectral images with high-spatial and low-spectral resolution is an important method to improve spatial resolution. Existing deep learning-based image fusion technologies usually neglect the ability of neural networks to understand differential features. In addition, the loss constraints do not stem from the physical characteristics of the hyperspectral imaging sensors. We propose the self-supervised loss and the spatially and spectrally separable loss, respectively. 1) The self-supervised loss: Different from the previous way of directly stacking the upsampled hyperspectral images and multispectral images as input, we expect the potentially processed hyperspectral images to ensure not only the integrity of hyperspectral image information, but also the most reasonable balance between overall spatial and spectral features. Firstly, the pre-interpolated hyperspectral images are decomposed into subspaces as self-supervised labels. Then, a network is designed to learn subspace information and obtain the most discriminative features. 2) The separable loss: According to the physical characteristics of hyperspectral images, the pixel-based mean square error loss is first divided into the domain loss and spectral domain loss, and then the similarity score of the images is calculated and used to construct the weighting coefficients of the two domain losses. At last, the separable loss is jointly expressed by the weights. Experiments on public benchmark datasets indicate that the self-supervised loss and separable loss can improve fusion performance.

Exploring fast and effective spectral-spatial feature extraction algorithms for hyperspectral image (HSI) classification is one of the most focus problems in current hyperspectral remote-sensing research. Generally, the size of homogeneous regions in HSIs is not consistent in real scenario and real scenario usually consist of ground objects of different scales. Multiscale strategy starts to be used to construct discriminative features at different scales for HSI classification in recent years. To efficiently characterize the multiscale spectral-spatial features of HSIs, a curvature filters-based multiscale feature extraction method with multiscale superpixel segmentation constraint is proposed. The proposed algorithm is composed of the following major stages. First, global multiscale spectral-spatial features are efficiently extracted via progressively curvature filtering and downsampling operations, which can be regarded as an image pyramid decomposition method. Next, a multiscale superpixel segmentation strategy is applied on the first layer of the image pyramid, and a weighted mean operation is applied within and among superpixels to extract the local multiscale spatial features (LMSFs). Finally, the global multiscale curvature features (GMCFs) and the superpixel segmentation-based LMSFs are fused to form the final multiscale spectral-spatial features for classification purposes. To verify the capabilities of the proposed method, comprehensive experiments are performed on five real hyperspectral datasets. Experimental results demonstrate that the proposed method can significantly improve the classification accuracies compared to several standard HSI feature extraction and classification methods, especially when the number of samples for training is limited.

Fog in hyperspectral images severely limits the visibility of imaging scene and reduces the image contrast, which has a negative effect on the following image interpretation. Defogging methods aim at restoring a high-quality image from the degraded image. Currently, most dehazing methods mainly depend on the atmospheric scattering model in computer vision and multispectral image communities. However, when these approaches are directly used to remove the fog from HSIs, they cannot produce satisfactory defogging performance. To alleviate this issue, we develop a novel fog model to achieve fog removal from hyperspectral images. First, a fog density map is calculated by differentiating the averaged bands falling into visible and infrared spectral ranges. Then, haze abundance in different spectral bands is estimated based on the pixel reflectance between two selected pixels with different haze levels. Finally, the high-quality hyperspectral image is restored by solving the defogging model. Experiments performed on a new benchmark created by ourselves demonstrate that the proposed method obtains favorable dehazing performance in contrast to other approaches in computer vision and remote sensing fields.

Remote sensing image super-resolution aims to boost the image resolution while recovering rich high-frequency details. Currently, most of super-resolution methods are based on an assumption that the degradation kernel is a specific downsampler. However, the degradation kernel is unknown and sophisticated for real remote sensing scenes, leading to a severe performance drop. To alleviate this problem, we propose a multi-layer degradation representation-guided blind super-resolution method for remote sensing images, which mainly consists of three key steps. First, an unsupervised representation learning is exploited to learn the degradation representation from low-resolution images. Then, a degradation-guided deep residual module is designed to model high-order features across different scales from original images. Finally, a multi-layer degradation-aware feature fusion mechanism is proposed to restore the finer details. Experiments on synthetic and real datasets demonstrate that the proposed method can achieve promising performance with respect to other state-of-the-art super-resolution approaches.

Semisupervised semantic segmentation is an effective way to reduce the expensive manual annotation cost and take advantage of the unlabeled data for remote sensing (RS) image interpretation. Recent related research has mainly adopted two strategies: self-training and consistency regularization. Self-training tries to acquire accurate pseudo-labels to explicitly expand the train set. However, the existing methods cannot accurately identify false pseudo-labels, suffering from their negative impact on model optimization. The consistency regularization constrains the model by producing consistent predictions robust to the perturbations introduced in the sample or feature domain but requires a sufficient number of training data. Therefore, we propose a strategy for the semisupervised semantic segmentation of the RS images. The proposed model in the generative adversarial network (GAN) framework is optimized by consistency self-training, learning the distributions of both labeled and unlabeled data. The discriminator is optimized by accurate pixel-level training labels instead of the image-level ones, thereby assessing the confidence for the prediction of each pixel, which is then used to reweight the loss of the unlabeled data in self-training. The generator is optimized with the consistency constraint with respect to all random perturbations on the unlabeled data, which increases the sample diversity and prompts the model to learn the underlying distribution of the unlabeled data. Experimental results on the the large-scale and densely annotated Instance Segmentation in Aerial Images Dataset (iSAID) datasets and the International Society for Photogrammetry and Remote Sensing (ISPRS) datasets show that our framework outperforms several state-of-the-art semisupervised semantic segmentation methods.

Spatio-temporal fusion aims to integrate multisource remote sensing images with complementary high spatial and temporal resolutions, so as to obtain time-series high spatial resolution fused images. Currently, deep learning (DL)-based spatio-temporal fusion methods have received broad attention. However, on one hand, most of the existing DL-based methods train the model in a band-by-band manner, ignoring the correlations among bands. On the other hand, the general coarse spatio-temporal changes in low spatial resolution images (e.g., MODIS) calculated at the pixel domain cannot completely cover the fine spatio-temporal changes in high spatial resolution images (e.g., Landsat), due to complex surface features and the general large spatial resolution ratio between fine and coarse images. Besides, the existing DL-based spatio-temporal fusion methods are insufficient in exploring multiscale information by only stacking convolutional kernels with different sizes. To alleviate the above challenges, we propose a progressive spatio-temporal attention fusion model in a multiband training manner based on generative adversarial network (PSTAF-GAN). Specifically, we design a flexible multiscale feature extraction architecture to extract multiscale feature hierarchies. Then, spatio-temporal changes are calculated on the feature domain in different feature hierarchies. Besides, a spatio-temporal attention fusion architecture is proposed to fuse the spatio-temporal changes and ground details in a coarse-to-fine manner, which can explore multiscale information more sufficient and gradually recover the target image. The results of quantitative and qualitative experiments on two publicly available benchmark datasets show that the proposed PSTAF-GAN can achieve the best performance compared with the state-of-the-art methods.

The normalized difference vegetation index (NDVI) can reflect the plant life cycle of growth and senescence and has become a widely used tool for many applications related to phenology, ecology, and environment. However, unwanted disturbance from cloud, snow, and other atmospheric effects greatly lowers the NDVI quality and hinders its further application. In this article, differing from the previous research attempting to approach the upper NDVI envelope by local adjustment or threshold-related iteration, a novel one-step global variational reconstruction (OGVR) method for NDVI time series is proposed via joint modeling of the gradual vegetation change and negatively biased atmospheric contamination. Two versions of the proposed method are designed for processing NDVI data with or without auxiliary flag information. Long-term and global-scale Advanced Very High Resolution Radiometer (AVHRR) global inventory monitoring and modeling system (GIMMS) data were applied in simulated and real-data experiments to verify the proposed method. The results show that the proposed method can successfully estimate the natural vegetation change from seriously contaminated NDVI time series and can conquer the problem of continuous low-value gaps. The qualitative and quantitative comparisons with five other widely used methods indicate that the proposed method has significant advantages in terms of both effectiveness and stability.

Pansharpening methods have been developed for nearly 40 years; however, how to quantitatively evaluate the quality of pansharpened images at full resolution (FR) is probably the most debated topic in this field due to the inherent unavailable of the real HR MS reference image. In this article, a novel blind FR quality evaluation method for pansharpening is proposed. In the proposed method, spatial and spectral features that are sensitive to spatial and spectral distortions of fused images are comprehensively considered and jointly learned based on online multivariate Gaussian (MVG) to construct the evaluation model. It directly outputs the quality of fused images, rather than the stepwise evaluation of spectral score, spatial score, and final overall quality score by the weighted combination of them, which may introduce contradictory results. First, a pristine benchmark evaluation model is established on the spatial features from the original high-spatial-resolution (HR) panchromatic (PAN) image and the spectral invariant assumption between ideal fused and original multispectral (MS) images. Second, a testing evaluation model for the fused image is founded. Finally, the quality of the fused image is measured based on the distance between the testing and benchmark models. The experimental results demonstrated the superior performance of the proposed method. Furthermore, the proposed method can be generalized to other interesting tasks, such as the nonreference evaluation for pansharpening with missing information and the nonreference evaluation for hyperspectral image fusion. The source code is available onhttps://github.com/yyxhpkq/MQNR.

Spatio-temporal fusion by combining the complementary spatial and temporal advantages of multi-source remote sensing images to obtain time-series high spatial resolution images is highly desirable in monitoring surface dynamics. Currently, deep learning (DL)-based fusion methods have received extensive attention. However, existing DL-based spatio-temporal fusion methods are generally limited in fusing the images with land cover changes. In this paper, we propose a spatio-temporal-spectral collaborative learning framework for spatio-temporal fusion to alleviate this problem. Specifically, the proposed method integrates the convolutional neural network and recurrent neural network into a unified framework, consisting of three sub-networks: multi-scale siamese convolutional neural network, multi-layer convolutional recurrent neural network, and adaptive weighting fusion network. The multi-scale siamese convolutional neural network has a flexible weight-sharing network to extract multi-scale spatial-spectral features from multi-source remote sensing images. The multi-layer convolutional recurrent neural network is constructed on the convolutional long-short term memory units to comprehensively learn the land cover changes by spatial, spectral, and temporal joint features. The adaptive weighting fusion network with a spatio-temporal-spectral change loss is proposed to further improve the interpretability and robustness. The experiments were performed on the publicly available benchmark datasets featured by phenology and land cover type changes, respectively. The experimental results demonstrated the competitive performance of the proposed method than other state-of-the-art fusion methods.

Pansharpening is a fundamental and hot-spot research topic in remote sensing image fusion. In recent years, self-attention-based transformer has attracted considerable attention in natural language processing (NLP) and introduced to attend to computer vision (CV) tasks. Inspired by great success of the vision transformer (ViT) in image classification, we propose an improved and advanced purely transformer-based model for pansharpening. In the proposed method, stacked multispectral (MS) and panchromatic (PAN) images are cropped into patches (i.e., tokens), and after a three-layer self-attention-based encoder, these tokens contain rich information. After upsampled and stitched, a high spatial resolution (HR) MS image is finally obtained. Instead of convolutional neural networks (CNNs) pursuing a short-distance dependency, our proposed method aims to build up a long-distance dependency, to make full use of more useful features. The experiments were conducted on an opening benchmark dataset, including IKONOS with four-band MS/PAN images and WorldView-2 MS images featured by eight bands. In addition, the experiments were performed on reduced and full-resolution datasets from both qualitative and quantitative evaluation aspects. The experimental results indicate the competitive performance of the proposed model than other pansharpening methods, including the state-of-the-art pansharpening algorithms based on CNN.

Unmanned aerial vehicle (UAV) hyperspectral imaging has been extensively applied in various fields. However, due to the limited imaging width, hyperspectral images (HSIs) captured by UAV need to be stitched, so as to effectively cover the study area. In this article, an effective seamless stitching method with deep feature matching and elastic warp is proposed for HSIs, which consists of the following major steps. First, for each input HSI, a single-band gray-scale image is obtained by fusing the bands corresponding to the red, green, and blue wavelengths. Second, the feature points of each HSI are obtained with a robust VGG-style network and matched with a graph neural network. After point pairs are obtained, the next step is to estimate the transformation matrix of adjacent images, and a spectral correction method based on intrinsic decomposition is proposed to ensure the spectral consistency of adjacent images. In the final stage, a seam-cutting and multiscale blending strategy is adopted to ensure the spatial consistency of the stitching results. Experimental results on real HSIs show that the proposed method is superior to six representative image stitching approaches.

The low resolution of remote sensing images often limits the land cover classification (LCC) performance. Super resolution (SR) can improve the image resolution, while greatly increasing the computational burden for the LCC due to the larger size of the input image. In this article, the SR-guided deep network (SRGDN) framework is proposed, which can generate meaningful structures from higher resolution images to improve the LCC performance without consuming more computational costs. In general, the SRGDN consists of two branches (i.e., SR branch and LCC branch) and a guidance module. The SR branch aims to increase the resolution of remote sensing images. Since high- and low-resolution image pairs cannot be directly provided by imaging sensors to train the SR branch, we introduce a self-supervised generative adversarial network (GAN) to estimate the downsampling kernel that can produce these image pairs. The LCC branch adopts the high-resolution network (HRNet) to retain as much resolution information with a few downsampling operations as possible. The guidance module teaches the LCC branch to learn the high-resolution information from the SR branch without the utilization of the higher-resolution images as the inputs. Furthermore, the guidance module introduces spatial pyramid pooling (SPP) to match the feature maps of different sizes in the two branches. In the testing stage, the guidance module and SR branch can be removed, and therefore do not create additional computational costs. Experimental results on three real datasets demonstrate the superiority of the proposed method over several well-known LCC approaches.

Coastal wetland monitoring plays an important role in the protection and restoration of ecosystems in this world. UAV-hyperspectral imaging, as an emerging technique for Earth observation and space exploration, provides the huge potential ability to identify different wetland species. In this work, a multilayer global spectral–spatial attention network (MGSSAN) is proposed for mapping coastal wetlands, which mainly consists of two major steps. First, a two-branch convolutional neural network (CNN) framework with residual connection is developed to obtain an initial classification probability map, in which one branch is used to capture the spectral information, the other branch is used to extract spatial information, and a global spectral–spatial attention module is designed to guide networks focusing on those features that are more discriminative. Second, an extended random walker method is utilized to optimize the initial classification probabilities, so as to yield the final map. Experiments performed on three wetland HSI datasets created by ourselves verify that the proposed method can obtain superior performance with respect to several state-of-the-art hyperspectral image classification methods.

Exploring effective spectral–spatial feature extraction methods is one of the most focused problems in current hyperspectral image (HSI) classification research. However, complex spectral–spatial structure characteristics in HSIs, e.g., shape-variable spatial structure and nonlinear spectral structure, are difficult to be effectively extracted and jointly represented. To overcome this issue, a novel superpixel-based Brownian descriptor (SBD) method for HSI classification is proposed in this article. In specific, superpixel segmentation is first used to extract shape-adaptive spatial structure information from dimension-reduced HSI, leading to generate nonoverlapping homogeneous 3-D image blocks. Then, similar pixels within the 3-D image block are jointly represented by a new local spectral–spatial feature based on the Brownian descriptor (BD). This is the first time that the BD is introduced to measure both linear and nonlinear correlations among different spectral bands in HSI. On one hand, the integration of superpixel and BD helps to provide much richer and more valuable information for better discrimination between different categories. On the other hand, the SBD can effectively represent the internal structure characters within each 3-D image block of different spatial shapes by a symmetric positive definite matrix of a united form. Finally, considering that the SBD lies on the Riemannian manifold space, a log-Euclidean kernel sparse representation (LKSR) classifier is introduced to obtain the classification results. Experimental results on three widely used real hyperspectral datasets indicate the performance superiority of the proposed SBD method over several state-of-the-art techniques.

Combining spectral and spatial information can significantly improve the classification performance of hyperspectral image (HSI). Currently, a lot of spectral–spatial HSI classification methods have been proposed. However, the task of HSI classification has remained challenging since the number of training samples is limited in real scenarios. In this article, we propose a novel HSI classification framework with single sample, in which the spectral self-similarity and spatial polygon structure information are fully combined to improve the classification performance. On the one hand, spectral self-similarity is used to expand training samples, which makes it possible to obtain sufficient samples with minimal cost. On the other hand, polygonal partition is introduced to acquire the geometrical structure of land covers in man-made environments. Specifically, the edge information of geometric objects is captured by polygonal partition, which can be utilized to constrain the spatial range of sample expansion and optimize the classification results. Experimental results on three real HSIs illustrate that the proposed method performs very well under small training sample size even when the number of samples is single per class.

For hyperspectral image (HSI) classification, it is an challenging problem to learn highly discriminative features, since the complex local/non-local spatial-spectral association is difficult to be accurately characterized. Focused on this issue, a novel superpixel-level hybrid discriminant analysis (SHDA) method is proposed, in this paper. Here, the SHDA method takes advantage of superpixel’s merit in characterizing spatial-spectral shape-adaptive structure and the powerful capability of discriminant analysis in enhancing class-separability to learn the feature representation. Moreover, the local/non-local spatial-spectral correlation information among/between superpixels is effectively excavated and fused in one framework to further improve the classification performance of features. This is achieved by first designing two specific discriminant analysis modules, i.e., superpixel-level local discriminant analysis (SLDA) and superpixel-level non-local discriminant analysis (SNDA). In the SLDA, adaptively weighted scatter matrices are defined to characterize the local spectral similarity within each superpixel and the discrepancy among adjacent superpixels. In the SNDA, superpixel-level graphs are built for capturing the non-local contextual information, where the weights of graphs are estimated based on the most similar and dissimilar superpixels. Then, the SLDA and the SNDA are effectively fused to construct the total intra/inter-superpixel scatter matrices. Finally, a joint projection transformation is obtained via solving a simple generalized eigenvalue problem. By this way, the HSI data can be projected from a high-dimensional data space into a low-dimensional feature space, where different classes of land-covers can be more accurately distinguished. Experimental results on three real hyperspectral data sets indicate that the proposed SHDA method outperforms several state-of-the-art techniques.

Human–Machine Interaction (HMI) in Intelligent and Connected Vehicles (ICVs) has drawn great attention in recent years due to its potentially significant positive impacts on the automotive revolution and travel experience. In this paper, we conduct an in-depth review of HMI in ICVs. Firstly, research and application development status are pointed out through the discussion on the cutting-edge technology classification, achievements, and challenges of the HMI technologies in ICVs, including recognition technology, multi-dimensional human vehicle interface, and emerging in-vehicle intelligent units. Then, the human factors issues of ICVs are discussed from three aspects: ICV acceptance, interaction quality of ICVs, and user experience of ICVs. Besides, based on the interaction technology and the mapping of the above issues, we conducted a visual analysis of the literature to realize the reflective thinking of the current HMI in ICVs. Finally, the challenges of HMI technology in ICVs are summarized. Moreover, the promising future opportunities are proposed from three aspects: utility optimization, experience reconfiguration, and value acquisition, to gaining insight into advanced and pleasant HMI in ICVs.

Image registration and the estimation of spatial and spectral blur kernels are essential steps before fusing panchromatic image (PAN) and multispectral image (MSI). Usually, these basic steps are performed separately, which will lead to error accumulation and ultimately affect the fusion performance. In this paper, we propose a novel deep learning (DL) based framework which can jointly register images and learn the blur kernels of original PAN and MSI. Specifically, we first construct a convolutional neural network (CNN) to learn the offsets between PAN and MSI, and the offsets are utilized to align the two images. Then, we analyze the relationship between the registered PAN and MSI, and design a tiny network for blur kernel learning. After solving the gradient derivation problems, we can combine the two networks and train them end-to-end. Experimental results on GF-2 satellite images demonstrate that the proposed method can significantly improve the fusion performance of some popular pansharpening methods.

Hyperspectral image classification has been a constant hot topic in remote sensing field, and achieved significant progress recently. Until now, most of the existing works are based on high-quality noise-free datasets, whereas in real applications, the images are often degraded by different types of noise, which makes the noise sensitivity become one of the key issues for classification assessment. In this paper, we study the noise effects on hyperspectral image classification including Guassian, salt-and-pepper, and stripe noise. The experimental results shows that noise has varying degrees of negative effects on different hyperspectral classification methods, which provides instructional information for method design and selection under noise environment in actual classification applications.

Fusion of hyperspectral images (HSIs) and multispectral images (MSIs) with different resolutions is an active research topic in the field of remote sensing. However, HSI-MSI fusion assessments in existing researches are basically conducted on simulated data gone through spectral or spatial downsampling, which cannot reflect the actual performances of fusion methods in application scenarios. To conquer this problem, a new remote sensing dataset- Hunan UAV-borne HSIs and MSIs fusion(HNU-HMiF) dataset is provided in this paper. The proposed dataset contains fine registered hyperspectral and multispectral image pairs captured by unmanned aerial vehicle (UAV) covering different ground objects, and can be used to evaluate, select, and even develop fusion methods for users or researchers. Successful applications including method evaluation and comparison confirm the validity and reliability of the proposed dataset.

In this paper, a novel unmanned aerial vehicle (UAV) hyperspectral image stitching framework based on radiation correction and seam-cutting blending is proposed. Firstly, spectral correlation constraints are introduced to eliminate mismatched pairs in the transform matrix estimation step. Then, a spectral correction method based on intrinsic images is proposed to ensure spectral consistency of stitching results. In order to obtain more natural stitching results without edge effect, a seam-cutting and multi-scale blending strategy is adopted in the final blending stage. Experimental results on real unmanned aerial vehicle hyperspectral strip images show that the proposed method is superior to a representative image stitching approach.

In this paper, a novel local Brownian descriptor (LBD) based feature extraction method is proposed for hyperspectral image (HSI) classification. Compared with the classical correlation feature that only characterizes the linear relationship among spectral bands, the LBD can measure both linear and nonlinear relationships to provide much richer spatial-spectral information for HSI classification. Specifically, the HSI is firstly mapped into a low-dimensional spectral subspace by utilizing the maximum noise fraction (MNF) method. Based on the subspace, pixels in the square neighborhood of each sample construct a sample block. Then, the LBD as the local correlation feature is calculated for each sample block. Finally, a kernel sparse representation (KSR) classifier is utilized on these feature descriptors, leading to the final classification results. Experiments conducted on a real hyperspectral image demonstrate the outstanding performance of the proposed method over several state-of-the-art methods.

It is well known that classification accuracy highly relies on the number of labeled samples. However, it is difficult to obtain sufficient labeled samples in real-world applications. To solve this issue, a novel hyperspectral image (HSI) classification method based on polygonal partition is proposed for crop mapping. This method only needs single sample per class as an initial training set. Specifically, multiscale polygonal partition is applied on the first three components of the HSI. Then, a spectral similarity-based sample expansion method is proposed to obtain more labeled samples. Next, a pixel-wise classifier, the support vector machine (SVM), is used to acquire an initial classification result. Finally, classification result is further optimized according to the partition maps. Experimental results show that classification performance of the proposed method is satisfactory even when the number of labeled sample is single for each class.

The inclusion of spatial information into spectral classifiers for fine-resolution hyperspectral imagery has led to significant improvements in terms of classification performance. The task of spectral-spatial hyperspectral image (HSI) classification has remained challenging because of high intraclass spectrum variability and low interclass spectral variability. This fact has made the extraction of spatial information highly active. In this work, a novel HSI classification framework using the fusion of dual spatial information is proposed, in which the dual spatial information is built by both exploiting pre-processing feature extraction and post-processing spatial optimization. In the feature extraction stage, an adaptive texture smoothing method is proposed to construct the structural profile (SP), which makes it possible to precisely extract discriminative features from HSIs. The SP extraction method is used here for the first time in the remote sensing community. Then, the extracted SP is fed into a spectral classifier. In the spatial optimization stage, a pixel-level classifier is used to obtain the class probability followed by an extended random walker-based spatial optimization technique. Finally, a decision fusion rule is utilized to fuse the class probabilities obtained by the two different stages. Experiments performed on three data sets from different scenes illustrate that the proposed method can outperform other state-of-the-art classification techniques. In addition, the proposed feature extraction method, i.e., SP, can effectively improve the discrimination between different land covers.

Driven by the urgent demand for managing remote sensing big data, large-scale remote sensing image retrieval (RSIR) attracts increasing attention in the remote sensing field. In general, existing retrieval methods can be regarded as visual-based retrieval approaches that search and return a set of similar images to a given query image from a database. Although these retrieval methods have delivered good results, there is still a question that needs to be addressed: can we obtain the accurate semantic labels of the returned similar images to further help analyzing and processing imagery? To this end, in this article, we redefine the image retrieval problem as visual and semantic retrieval of images. Especially, we propose a novel deep hashing convolutional neural network (DHCNN) to retrieve similar images and classify their semantic labels simultaneously in a unified framework. In more detail, a convolutional neural network (CNN) is used to extract high-dimensional deep features. Then, a hash layer is perfectly inserted into the network to transfer the deep features into compact hash codes. In addition, a fully connected layer with a softmax function is performed on the hash layer to generate the probability distribution of each class. Finally, a loss function is elaborately designed to consider the label loss of each image and similarity loss of pairs of images simultaneously. Experimental results on three remote sensing data sets demonstrate that the proposed method can achieve state-of-art retrieval and classification performance.

Hyperspectral image (HSI) and multispectral image (MSI) fusion, which fuses a low-spatial-resolution HSI (LR-HSI) with a higher resolution multispectral image (MSI), has become a common scheme to obtain high-resolution HSI (HR-HSI). This article presents a novel HSI and MSI fusion method (called as CNN-Fus), which is based on the subspace representation and convolutional neural network (CNN) denoiser, i.e., a well-trained CNN for gray image denoising. Our method only needs to train the CNN on the more accessible gray images and can be directly used for any HSI and MSI data sets without retraining. First, to exploit the high correlations among the spectral bands, we approximate the desired HR-HSI with the low-dimensional subspace multiplied by the coefficients, which can not only speed up the algorithm but also lead to more accurate recovery. Since the spectral information mainly exists in the LR-HSI, we learn the subspace from it via singular value decomposition. Due to the powerful learning performance and high speed of CNN, we use the well-trained CNN for gray image denoising to regularize the estimation of coefficients. Specifically, we plug the CNN denoiser into the alternating direction method of multipliers (ADMM) algorithm to estimate the coefficients. Experiments demonstrate that our method has superior performance over the state-of-the-art fusion methods.

Recently, many convolutional neural network (CNN) methods have been designed for hyperspectral image (HSI) classification since CNNs are able to produce good representations of data, which greatly benefits from a huge number of parameters. However, solving such a high-dimensional optimization problem often requires a large number of training samples in order to avoid overfitting. In addition, it is a typical nonconvex problem affected by many local minima and flat regions. To address these problems, in this article, we introduce the naive Gabor networks or Gabor-Nets that, for the first time in the literature, design and learn CNN kernels strictly in the form of Gabor filters, aiming to reduce the number of involved parameters and constrain the solution space and, hence, improve the performances of CNNs. Specifically, we develop an innovative phase-induced Gabor kernel, which is trickily designed to perform the Gabor feature learning via a linear combination of local low-frequency and high-frequency components of data controlled by the kernel phase. With the phase-induced Gabor kernel, the proposed Gabor-Nets gains the ability to automatically adapt to the local harmonic characteristics of the HSI data and, thus, yields more representative harmonic features. Also, this kernel can fulfill the traditional complex-valued Gabor filtering in a real-valued manner, hence making Gabor-Nets easily perform in a usual CNN thread. We evaluated our newly developed Gabor-Nets on three well-known HSIs, suggesting that our proposed Gabor-Nets can significantly improve the performance of CNNs, particularly with a small training set.

Sun glint, as the spectral reflection of solar radiation on non-flat water surfaces, is a serious confounding factor for coastal shallow-water environments. When the coastal areas are observed with a hyperspectral sensor, the existing sun glint in the produced images can seriously influence the quality of the image interpretation. To solve this issue, in this paper, we propose a novel sun glint removal method based on a variation model for hyperspectral images (HSIs). The proposed method aims to decompose the original HSI into a desired clean image and a sun glint image. To achieve this, we exploit a texture-aware total variation to remove the sun glint in HSIs, where the texture information is imposed on the total variation regularization to highlight sun glint. Experiments on simulated and real datasets demonstrate that our method can obtain outstanding performance with respect to other state-of-the-art approaches.

Hyperspectral imaging plays an important role for mineral mapping in a nondestructive and noninvasive way. In this paper, a novel resolution enhancement method is proposed based on the principle of intrinsic image decomposition for mineral mapping. This method is based on an assumption that hyperspectral image (HSI) can be decomposed into a reflectance component and an illumination component. Based on this idea, the RGB image is first transformed into Intensity-Hue-Saturation (IHS) space, and the intensity channel is considered as the illumination component of the HSI with an ideal high spatial resolution. Then, the reflectance component of the ideal HSI is estimated with the downsampled HSI image and the downsampled intensity channel. Finally, the HSI with high resolution can be reconstructed by utilizing the estimated illumination and the reflectance components. Experimental results validate the effectiveness of the proposed method qualitatively and quantitatively by outperforming several state-of-the-art approaches.

Deep learning (DL) for pansharpening has recently attracted considerable attentions. To construct training data, DL based pansharpening approaches often downsample the original multispectral image (MSI) and panchromatic image (PAN) with fixed blur kernel, which can be different from the real point spread functions (PSF) of the satellites. And a mismatched blur kernel will cause the pansharpening performance to drop dramatically. In this paper, we propose a novel blur kernel learning method for pansharpening, which can learn the spatial and spectral blur kernels between PAN and MSI in an unsupervised way. Specifically, we analyze the relationship between PAN and MSI, and then construct a mini net for blur kernel learning. Once the spatial blur kernel is found, a convolutional neural network (CNN) for pansharpening is trained on the downsampled dataset using the learned spatial blur kernel. Experimental results on GF-2 images demonstrate the superiority of the proposed method.

In this paper, in order to extract efficient spectral-spatial features for hyperspectral image classification, a Gaussian curvature (GC) filter based feature extraction method with multiscale segmentation constraint is proposed. The method consists of the following major steps: First, the maximum noise fraction (MNF) method is applied on the hyperspectral images (HSIs) to reduce the noise and computational complexity. The GC features are extracted from the dimension reduced HSIs via the GC filter. Next, a multiscale segmentation strategy is applied on the HSIs, and the multiscale spatial features are extracted by applying the weighted mean operations within and among superpixels. Finally, the GC features and dimension reduced multiscale spatial features are fused to form the final multiscale Gaussian curvature features (MGCFs) for classification purposes. To verify the effectiveness of the proposed method, we conduct experiments on the Indian Pines data set. Experimental results demonstrate that the proposed method can significantly improve the classification accuracies compared to several standard classification methods.

Vehicle detection in remote sensing(RS) images has been an active topic with the development of computer vision in recent years. However, directly applying conventional horizontal anchor-based detection methods in oriented vehicle detection often acquires poor performance. Although rotated anchors have been used to tackle this problem, this design leads to heavy computational cost because of thousands of rotated anchors generated in each level feature map. In this paper, we propose to detect vehicles with partial anchors, which greatly accelerates detection process. The novel Partial Anchors based Detection Network(PADeN) filter out redundant anchors with semantic information. To boost the performance of PADeN, the centerness mask branch is added into the network. The results demonstrate that PADeN significantly outperforms previous approaches in vehicle detection and achieves the mAP of 76.9%.

Noise usually appears in hyperspectral images (HSIs), and strongly affects the performance of the follow processing and analysis. In recent years, a large number of denoising algorithms have been proposed and it is known that the denoising effect is highly dependent on the accurate estimates of the type and level of noise present in an HSI. This paper focuses on analyzing the real noise in HSIs by separating and estimating the level of noise in HSIs. In consideration of the spectral correlation and the unique spatial structure of stripe noise, the developed method employs Fourier domain analysis and the high correlation among neighboring spectral bands to separate different types of noise. Experimental results show that the level of noise may be quite different for different bands of an HSI, and HSIs captured by different senors or in different scenes.

Recently, deep learning has been demonstrated to be an effective tool to detect changes in bitemporal remote sensing images. However, most existing methods based on deep learning obtain the ultimate change map by analyzing the difference image (DI) or the stacked feature vectors of input images, which cannot sufficiently capture the relationship between the two input images to obtain the change information. In this letter, a new method named bilinear convolutional neural networks (BCNNs) is proposed to detect changes in bitemporal multispectral images. The model can be trained end to end with two symmetric convolutional neural networks (CNNs), which are capable of learning the feature representation from bitemporal images and utilizing the relations between the two input images by a linear outer product operation in an effective way. Specifically, two sets of patches obtained from two multispectral images of different times are first input into two CNNs to extract deep features, respectively. Then, the matrix outer product is applied on the output feature maps to obtain the combined bilinear features. Finally, the ultimate change detected result can be produced by applying the softmax classifier on the combined features. Experimental results on real multispectral data sets demonstrate the superiority of the proposed method over several well-known change-detection approaches.

Combining a high-spatial-resolution multispectral image (HR-MSI) with a low-spatial-resolution hyperspectral image (LR-HSI) has become a common way to enhance the spatial resolution of the HSI. The existing state-of-the-art LR-HSI and HR-MSI fusion methods are mostly based on the matrix factorization, where the matrix data representation may be hard to fully make use of the inherent structures of 3-D HSI. We propose a nonlocal sparse tensor factorization approach, called the NLSTF_SMBF, for the semiblind fusion of HSI and MSI. The proposed method decomposes the HSI into smaller full-band patches (FBPs), which, in turn, are factored as dictionaries of the three HSI modes and a sparse core tensor. This decomposition allows to solve the fusion problem as estimating a sparse core tensor and three dictionaries for each FBP. Similar FBPs are clustered together, and they are assumed to share the same dictionaries to make use of the nonlocal self-similarities of the HSI. For each group, we learn the dictionaries from the observed HR-MSI and LR-HSI. The corresponding sparse core tensor of each FBP is computed via tensor sparse coding. Two distinctive features of NLSTF_SMBF are that: 1) it is blind with respect to the point spread function (PSF) of the hyperspectral sensor and 2) it copes with spatially variant PSFs. The experimental results provide the evidence of the advantages of the NLSTF_SMBF method over the existing state-of-the-art methods, namely, in semiblind scenarios.

Hyperspectral Image (HSI) visualization, which aims at displaying as much material information of original images as possible on a trichromatic monitor with natural color, plays an important role in image interpretation and analysis. However, most of the HSI visualization methods only focus on presenting the detail information of a scene without providing natural colors and distinguishing land covers with similar colors. In order to address this problem, this article proposes a multichannel pulse-coupled neural network (MPCNN)-based HSI visualization method, which consists of the following steps. First, the MPCNN is proposed and explored to fuse the original HSI so as to obtain a fused band with rich spatial details. Then, a color mapping scheme is proposed to determine the weights of red, green, and blue (RGB) channels. Finally, the weighted RGB channels are stacked together for visualization. Experiments performed on four hyperspectral data sets demonstrate that the proposed method not only displays the HSI with nature colors but also improves the details in the image. The effectiveness of the proposed method is demonstrated in terms of both visual effect and objective indexes.

Traditional hyperspectral imaging technique obtains numerous hyperspectral images (HSIs) with hundreds of spectral bands, leading to high cost in data acquisition, transmission, and storage. Compressed sensing (CS) theory provides a new imaging mechanism, which relies on the assumption that signals can be sparsely represented over a dictionary. By the CS imaging technique, original HSIs can be approximately reconstructed from only a few sampled measurements. In this article, a novel context-aware CS (CACS) method for HSIs is proposed by incorporating contextual prior to the dictionary learning and the sparse reconstruction. First, a patch-based online dictionary learning (ODL) algorithm is developed by introducing a joint sparse constraint. On the one hand, the online dictionary learning mechanism enables a more adaptive representation of HSIs with different scenes than using fixed-basis-based dictionaries, e.g., the discrete cosine transform (DCT) and the discrete wavelet transform dictionaries. On the other hand, the introduced joint sparse constraint promotes the learned dictionary to more sparsely and structurally represent spectral pixels. Then, with the well-learned dictionary, a weighted smoothing regularization is introduced to develop a new sparse reconstruction model. Considering the high spectral-spatial similarity of pixels in a neighborhood, the new sparse reconstruction model will encourage a locally smoothing reconstruction result. In this way, the spectral-spatial structures of the HSI can be well preserved, while possible artifacts can be effectively reduced. Experimental results demonstrate the superiority of the proposed method over some state-of-the-art hyperspectral compressive imaging methods.

The quantity and quality of training samples have a great influence on the performance of most hyperspectral image classification approaches. However, in a real scenario, manually annotating a large number of accurate training samples is extremely labor-intensive and time-consuming. In this article, a multilabel training sample augmentation method is proposed. Instead of giving an exact label to each pixel, we just precisely label a small number of pixels by giving them a single label (called single-label samples) and annotate a large number of pixels in certain regions together by giving them multiple labels (called multilabel samples). Furthermore, in order to make full use of the multilabel training samples, a superpixel segmentation and recursive filtering-based method is proposed. The proposed method consists of the following major steps: recursive filtering-based feature extraction, superpixel-based segmentation, and spectral-spatial similarity-based mislabeled sample removal. Experimental results demonstrate that the proposed method can significantly improve the classification accuracy of multiple classifiers by using the multilabel training samples.

Spectral-spatial Gabor filtering, which is based on 3-D local harmonic analysis, has been a powerful spectral-spatial feature extraction tool for hyperspectral image (HSI) classification. However, existing spectral-spatial Gabor approaches are prone to oversmoothing, neglecting the existences of edges and negatively affecting the classification. In this article, we propose a new HSI Gabor filtering concept, called spectral-spatial-range Gabor filtering, which intends to restrain edge interference from disturbing local spectral-spatial harmonic components. Contributions and novelties of our work can be identified as follows: 1) an HSI filtering framework is created, which can accommodate various Gabor filtering procedures and hence offer the potential to guide the design of new Gabor filters; 2) following such a unified filtering framework and taking into consideration both local spectral-spatial harmonic characteristics and range domain variations, we develop a new concept of spectral-spatial-range Gabor filtering; and 3) utilizing this proposed Gabor prototype and elaborating mathematical derivations, we achieve a novel discriminative spectral-spatial-range Gabor filtering method, which can deal with discriminative local harmonics and edge interference simultaneously along the spectral-spatial-range domain, obtaining highly discriminative Gabor features while yielding linear computational complexity. Our novel method is evaluated on four real HSI data sets and achieves excellent performances.

Most of the existing hyperspectral anomaly detectors are designed based on a single pixel-level feature. These detectors may not adequately utilize spectral-spatial information in hyperspectral images (HSIs) for detecting anomalies. To overcome this problem, this article introduces a novel subpixel-pixel-superpixel guided fusion (SPSGF) method for hyperspectral anomaly detection. This approach comprises three main steps. First, subpixel-, pixel-, and superpixel-level features are extracted from an HSI by employing the spectral unmixing, morphological operation, and superpixel segmentation techniques, respectively. Then, based on the spatial consistency of three features, a guided filtering-based weight optimization technique is developed to construct weight maps for fusion. Finally, a simple yet effective decision fusion method is adopted to utilize the complemental information of three features, and then generates a fused detection result. The performance of the proposed approach is evaluated on three real-scene HSIs and one synthetic HSI. Experimental results validate the advantages of the SPSGF method.

In this article, a novel game theory-based approach is proposed for anomaly detection in hyperspectral images (HSIs) via effectively exploring multiple spectral and spatial characteristics of anomalies. This approach comprises three main steps. First, spectral, extend morphological profiles (EMPs), and Gabor texture features are captured from an input HSI. Then, we define the anomaly detection problem as an anomaly game model, in which image regions (superpixels) of different features are modeled as players who select to be “anomaly” or “background” as their strategies. Three initial detection results are produced based on each player's strategy in the Nash equilibrium of the anomaly game. Last, a saliency-based decision fusion technique is used to combine the complementary information in different features, so as to obtain a fused detection map. The performance of the proposed anomaly detection technique is evaluated on four real-scene HSIs. Experimental results validate that our approach can outperform some state-of-the-art anomaly detection methods.

Active learning (AL) attempts to actively select the most representative or useful training samples in an iterative manner. The aim is to simultaneously improve the classification performance and reduce the manual labeling effort. In this article, a novel subpixel-pixel-superpixel-based multiview AL (MAL) (SPS-MAL) method is proposed for hyperspectral image (HSI) classification. Here, the multiple views are generated via extracting the subpixel-level, pixel-level, and superpixel-level information. The multiple views can reflect various characteristics of HSI, i.e., spectral mixture, spectral discrimination, and spectral-spatial structure. Therefore, the joint use of diverse and complementary information in multiple views will contribute to a better identification ability of different classes. In addition, a coarse-to-fine MAL algorithm is introduced to effectively select the most representative samples with the most uncertainty. Specifically, a disagreement analysis on multiple views and joint posterior probability estimation is used to query unlabeled samples. Along with the expansion of training samples, view-specific confidence scores are estimated to adaptively integrate the classification results of multiple views, according to their discrimination performance. In this way, the classification accuracy will be further boosted while the number of necessary training samples can be significantly reduced. The experimental classification results on three well-known HSIs demonstrate the effectiveness of the proposed SPS-MAL method.

In this article, a novel hyperspectral anomaly detection method with kernel Isolation Forest (iForest) is proposed. The method is based on an assumption that anomalies rather than background can be more susceptible to isolation in the kernel space. Based on this idea, the proposed method detects anomalies as follows. First, the hyperspectral data are mapped into the kernel space, and the first K principal components are used. Then, the isolation samples in the image are detected with the iForest constructed using randomly selected samples in the principal components. Finally, the initial anomaly detection map is iteratively refined with locally constructed iForest in connected regions with large areas. Experimental results on several real hyperspectral data sets demonstrate that the proposed method outperforms other state-of-the-art methods.

Hyperspectral unmixing (HU) is an active research topic in the remote-sensing community. It aims at modeling mixed pixels using a collection of pure constituent materials (endmembers) weighted by their corresponding fractional abundances. Among existing unmixing schemes, nonnegative matrix factorization (NMF) has drawn significant attention due to its unsupervised nature, as well as its capacity to obtain both endmembers and fractional abundances simultaneously. In this article, we present a new blind unmixing method based on the generalized morphological component analysis (GMCA) framework, in which an additional constraint is introduced into the standard NMF model to represent the sparsity and morphological diversity of the abundance maps associated with each endmember. More specifically, we take into account the fact that different ground categories in a hyperspectral scene generally exhibit various spatial distributions and morphological characteristics. As a result, when providing a specific dictionary basis for these categories, their corresponding abundance maps (referred to as sources) can be sparsely represented. In addition, due to the low correlation between different sources, their sparse representations will not share the same most significant coefficients. With this observation in mind, we can further promote source discrimination and separation in the unmixing process. Moreover, in order to obtain a stable solution of the involved optimization problem, we adopt an alternate iterative constrained algorithm with a threshold descent strategy. Our experiments, carried out on both synthetic and real hyperspectral scenes, reveal that our newly developed GMCA-based unmixing method obtains very promising results with fast convergence speed and requiring significantly less parameter tuning. This confirms the advantage of the proposed spatial morphological component approach for HU purposes.

A common approach to high dynamic range (HDR) imaging is to capture multiple images of different exposures followed by multi-exposure image fusion (MEF) in either radiance or intensity domain. A predominant problem of this approach is the introduction of the ghosting artifacts in dynamic scenes with camera and object motion. While many MEF methods (often referred to as deghosting algorithms) have been proposed for reduced ghosting artifacts and improved visual quality, little work has been dedicated to perceptual evaluation of their deghosting results. Here we first construct a database that contains 20 multiexposure sequences of dynamic scenes and their corresponding fused images by nine MEF algorithms. We then carry out a subjective experiment to evaluate fused image quality, and find that none of existing objective quality models for MEF provides accurate quality predictions. Motivated by this, we develop an objective quality model for MEF of dynamic scenes. Specifically, we divide the test image into static and dynamic regions, measure structural similarity between the image and the corresponding sequence in the two regions separately, and combine quality measurements of the two regions into an overall quality score. Experimental results show that the proposed method significantly outperforms the state-of-the-art. In addition, we demonstrate the promise of the proposed model in parameter tuning of MEF methods.1.

Hyperspectral images (HSIs) are often degraded by a mixture of various types of noise during the imaging process, including Gaussian noise, impulse noise, and stripes. Such complex noise could plague the subsequent HSIs processing. Generally, most HSI denoising methods formulate sparsity optimization problems with convex norm constraints, which over-penalize large entries of vectors, and may result in a biased solution. In this paper, a nonconvex regularized low-rank and sparse matrix decomposition (NonRLRS) method is proposed for HSI denoising, which can simultaneously remove the Gaussian noise, impulse noise, dead lines, and stripes. The NonRLRS aims to decompose the degraded HSI, expressed in a matrix form, into low-rank and sparse components with a robust formulation. To enhance the sparsity in both the intrinsic low-rank structure and the sparse corruptions, a novel nonconvex regularizer named as normalized ε -penalty, is presented, which can adaptively shrink each entry. In addition, an effective algorithm based on the majorization minimization (MM) is developed to solve the resulting nonconvex optimization problem. Specifically, the MM algorithm first substitutes the nonconvex objective function with the surrogate upper-bound in each iteration, and then minimizes the constructed surrogate function, which enables the nonconvex problem to be solved in the framework of reweighted technique. Experimental results on both simulated and real data demonstrate the effectiveness of the proposed method.

This paper proposes a novel end-to-end learning model, called skip-connected covariance (SCCov) network, for remote sensing scene classification (RSSC). The innovative contribution of this paper is to embed two novel modules into the traditional convolutional neural network (CNN) model, i.e., skip connections and covariance pooling. The advantages of newly developed SCCov are twofold. First, by means of the skip connections, the multi-resolution feature maps produced by the CNN are combined together, which provides important benefits to address the presence of large-scale variance in RSSC data sets. Second, by using covariance pooling, we can fully exploit the second-order information contained in such multi-resolution feature maps. This allows the CNN to achieve more representative feature learning when dealing with RSSC problems. Experimental results, conducted using three large-scale benchmark data sets, demonstrate that our newly proposed SCCov network exhibits very competitive or superior classification performance when compared with the current state-of-the-art RSSC techniques, using a much lower amount of parameters. Specifically, our SCCov only needs 10% of the parameters used by its counterparts.

Most of the existing face image super-resolution methods are designed for grayscale images. For color images, these methods just treat each color channel individually or considered the illumination part only, ignoring the relationships among different color channels. To address this concern, in this paper we present a color face image super-resolution method by learning the quaternion graph (LQG) representation. Instead of spatial domain, the color image is represented in the quaternionic domain, which encourages the proposed model to well preserve the correlations among different color channels. Besides, a graph regularization is learned in the quaternion space to ensure the smoothness of encoding feature space. More specifically, by utilizing the graph Laplacian, we present to promote the smoothness of representations by forcing similar training samples to share similar encoding coefficients. This not only helps to stabilize the linear system but also makes the model more robust to noise. Experimental results demonstrated the efficiency of the proposed method in super-resolving color face images.

Recently, the locality-constrained linear coding (LLC) based techniques have been widely exploited for face hallucination. However, for the color face image, the conventional LLC model ignores the neighbor self-similarity prior as well as the relevance of different color channels, resulting in unsatisfactory representations. This paper presents a novel Neighbor locality Representation and inter-Channel Correlation (NRCC) model for color face hallucination. Compared with conventional LLC, NRCC makes full use of neighbor self-similarity prior and takes advantage of the co-manifold structure among RGB channels. The neighbor self-similarity prior and co-manifold structure can make the reconstruction results of eyes and lips generated from the proposed method better than those from other methods. The experimental results in some public face databases indicated the superiority of the proposed method over the prior art face hallucination methods.

Fusing a hyperspectral image (HSI) with a multispectral image (MSI) of the same scene has become a popular way to increase the spatial resolution of HSI. In this paper, we propose a novel HSI and MSI fusion method (termed as the SSS), which is based on spectral low rank and non-local spatial similarities. Firstly, to exploit the high spectral correlations of the desired high spatial resolution HSI, we formulate the fusion problem as the estimation of low-dimensional spectral subspace and coefficients. Since the HSI preserves most of spectral information, the spectral subspace is estimated from HSI via singular value decomposition. With the spectral subspace known, we plug a state-of-the-art denoising algorithm, weighted nuclear norm minimization, into the alternating direction method of multipliers to estimate the coefficients, which can effectively promote the non-local similarities of desired high spatial resolution HSI. Experiments demonstrate that our method is competitive to the state-of-the-art approaches.

In this paper, a novel deep learning based visualization method is proposed for natural color visualization of hyperspectral images, which consists of the following steps. First, the spectral bands of the hyperspectral image are divided into two groups, i.e., the red, green, and blue (RGB) bands and the remaining bands. Then, a pretrained convolutional neural network (CNN) model, i.e., VGG-19, is explored to fuse the remaining bands so as to obtain a fused band with rich details. Next, the intensity-hue-saturation (IHS) transform is performed on the averaged red, green, and blue bands to obtain three different components, i.e., intensity (I), hue (H), saturation (S). Finally, the fused band is utilized to replace the intensity component followed by an inverse IHS transform. Experiments performed on two hyperspectral data sets demonstrate that the proposed method cannot only obtain a natural color resulting image, but also well preserving image details with respect to several state-of-the-art methods.

The number and quality of training samples have a big influence on hyperspectral image classification performance. However, it is often difficult to manually annotate a large number of accurate training samples because the annotation requires a lot of manpower and resources. In this paper, we first propose a multi-labeling method to label the training samples efficiently. Instead of giving the exact label for each training pixel, we just precisely label a small number of pixels (called single-label samples), and annotate a large number of pixels in certain regions together (called multi-label samples) with multiple labels. Furthermore, a superpixel segmentation and recursive filtering based sample enhancing method is proposed to make full use of multi-label training samples for classification, which consists of the following major steps: IFRF based feature extraction, superpixels based classification, and spatial-spectral similarity based inaccurate samples removal. Experimental results demonstrate that the proposed method can improve the classification accuracy of multiple classifiers with multi-label training samples.

This paper introduces a novel morphological random walker (MRW) technique for hyperspectral anomaly detection. The proposed MRW detector introduces a morphology-based objective function into a random walker (RW) optimization model, which can fully exploit the morphological property of anomalies for detection. Specifically, the proposed algorithm consists of two main steps. First, the extended morphological profiles and differential operations are employed to exploit the morphological property of anomalies. Then, based on the morphological property, a morphology-based objective function is constructed, and this objective function is incorporated into the RW-based optimization model for anomaly detection. Experimental results show that the proposed detector outperforms several state-of-the-art anomaly detectors.

Sea land segmentation is an important technique in many remote sensing applications, such as coastline surveillance and near-shore ship detection. High resolution satellite optical imaging is able to capture the details in the coastal regions, introducing intra-class variance and interferences like waves, shadow and forestry regions. To overcome such variance and interferences, the high resolution optical satellite image is first over-segmented into super-pixels, i.e., homogeneous regions. Then the conditional random fields (CRFs) are adopted to model the relations of the super-pixels. The optimal labels of all the superpixels are determined by performing the loopy belief propagation on the CRFs. The final segmentation result is obtained by refining the pixelwise probability conditioned on the estimated superpixel labels with edge preserving filtering. Experimental results on Google Earth data of different coastal cities show the effectiveness of the proposed sea land segmentation method.

Hyperspectral compressive sensing (HCS) is considered for reconstructing the hyperspectral image (HSI) from a few random sampled measurements. HCS is crucial for the onboard imaging systems to cut down the acquisition time and data storage volume, and simultaneously maintain image quality. In this paper, a spatial-spectral total variation (SSTV) regularized low-rank tensor decomposition (LRTD) method is proposed for HCS. Specifically, for the HSI, the tensor nuclear norm based LRTD is utilized to characterize the global correlation among all bands, and an anisotropic SSTV regularization is explored to describe the local spatial smooth structure and spectral correlation of adjacent bands. In addition, an efficient algorithm based on the alternative direction multiplier method is developed to solve the resulting optimization problem. Experimental results demonstrate that the proposed method is superior to the existing state-of-the-art ones.

A novel anomaly detection method based on Isolation Forest is proposed for hyperspectral images. The method is directly based on a concept that anomalies rather than background can be more susceptible to isolation in the feature space. Based on this idea, the proposed method detects anomalies by directly isolating anomaly pixels from background. First, the probabilities of the anomalies in HSI is first evaluated by global Isolation Forest to obtain an initial anomaly detection map. Then, the initial anomaly detection map is recursively refined with local Isolation Forest so as to make full use of the spatial information. Experimental results on two real hyperspectral data sets demonstrate that the proposed method is superior over several widely used anomaly detection approaches.

In this letter, a coarse-to-fine unsupervised method is proposed for cloud detection in remote sensing images. First, the color, texture, and statistical features of the remote sensing images are extracted with the color transform, dark channel estimation, Gabor filtering, and local statistical analysis methods. Then, an initial cloud detection map can be obtained by performing the support vector machines (SVM) on the stacked features, in which the SVM is trained with a set of samples automatically labeled by processing the dark channel of the original image with several thresholding and morphological operations. Finally, guided filtering is used to refine the boundaries in the initial detection map, which further improves the cloud detection accuracy. Experiments performed on several real remote sensing images demonstrate that the proposed method show better detection performances with respect to several recently proposed cloud detection methods in terms of both quantitative and visual comparisons.

In recent years, the collaborative representation (CR)-based techniques have been widely employed for face hallucination. However, the conventional CR model becomes less efficient in handling noisy low-resolution face images. In this paper, an iterative relaxed CR (iRCR) model with adaptive weights learning is presented to enhance the resolution of face images corrupted by noise. The core idea of iRCR is that a diagonal weight matrix is incorporated into the objective function, which helps to debase the influence of noise in representation. Different from existing collaborative methods with reweighting strategy where the weights require manually tuning, the weights in iRCR are adaptively learned to stay more consistent with the model error. Moreover, considering the local manifold structure property and nonlocal prior of small patches, the locality regularization and collaborative regularization are incorporated into a unified framework. This enables the proposed iRCR not only to capture the true topology structure of patch manifold but also to exploit the meaningful patterns among the whole training samples for reconstruction. Experimental results on both face dataset and real-world images demonstrate the superiority of our proposed method over several state-of-the-art face hallucination methods.

Tensor completion (TC), aiming to recover original high-order data from its degraded observations, has recently drawn much attention in hyperspectral images (HSIs) domain. Generally, the widely used TC methods formulate the rank minimization problem with a convex trace norm penalty, which shrinks all singular values equally, and may generate a much biased solution. Besides, these TC methods assume the whole high-order data is of low-rank, which may fail to recover the detail information in high-order data with diverse and complex structures. In this paper, a novel nonlocal low-rank regularization-based TC (NLRR-TC) method is proposed for HSIs, which includes two main steps. In the first step, an initial completion result is generated by the proposed low-rank regularization-based TC (LRR-TC) model, which combines the logarithm of the determinant with the tensor trace norm. This model can more effectively approximate the tensor rank, since the logarithm function values can be adaptively tuned for each input. In the second step, the nonlocal spatial-spectral similarity is integrated into the LRR-TC model, to obtain the final completion result. Specifically, the initial completion result is first divided into groups of nonlocal similar cubes (each group forms a 3-D tensor), and then the LRR-TC is applied to each group. Since similar cubes within each group contain similar structures, each 3-D tensor should have low-rank property, and thus further improves the completion result. Experimental results demonstrate that the proposed NLRR-TC method outperforms state-of-the-art HSIs completion techniques.

In this article, a novel hyperspectral image (HSI) classification method based on fusing multiple edge-preserving operations (EPOs) is proposed, which consists of the following steps. First, the edge-preserving features are obtained by performing different types of EPOs, i.e., local edge-preserving filtering and global edge-preserving smoothing on the dimension-reduced HSI. Then, with the assistance of a superpixel segmentation method, the edge-preserving features are further improved by considering the inter and intra spectral properties of superpixels. Finally, the spectral and edge-preserving features are fused to form one composite kernel, which is fed into the support vector machine (SVM) followed by a majority voting fusion scheme. Experimental results on three data sets demonstrate the superiority of the proposed method over several state-of-the-art classification approaches, especially when the training sample size is limited. Furthermore, 21 well-known methods, including mathematical morphology-based approaches, sparse representation models, and deep learning-based classifiers, are adopted to be compared with the proposed method on Houston data set with standard sets of training and test samples released during 2013 Data Fusion Contest, which also shows the effectiveness of the proposed method.

A convolutional neural network (CNN) has recently demonstrated its outstanding capability for the classification of hyperspectral images (HSIs). Typical CNN-based methods usually adopt image patches as inputs to the network. However, a fixed-size image patch in HSI with complex spatial contexts may contain multiple ground objects of different classes, which will deteriorate the classification performance of the CNN. In addition, traditional convolutional layers adopted in the CNN have a huge amount of parameters needed to be tuned, which will cause high computational cost. To address the above-mentioned issues, a novel squeeze multibias network (SMBN) is proposed for HSI classification. Specifically, the proposed SMBN first introduces the multibias module (MBM), which incorporates multibias into the rectified linear unit layers. The MBM can decouple the feature maps of input patches into multiple response maps (corresponding to different ground objects) and adaptively select the meaningful maps for classification. Furthermore, the proposed SMBN replaces the traditional convolutional layer with a squeeze convolution module, which can greatly reduce the number of parameters in the network, thus saving the running time, while still maintaining high classification accuracy. Experimental results on three real HSIs demonstrate the superiority of the proposed SMBN method over several state-of-the-art classification approaches.

Recently, researchers have shown the powerful ability of deep methods with multilayers to extract high-level features and to obtain better performance for hyperspectral image classification. However, a common problem of traditional deep models is that the learned deep models might be suboptimal because of the limited number of training samples, especially for the image with large intraclass variance and low interclass variance. In this paper, novel convolutional neural networks (CNNs) with multiscale convolution (MS-CNNs) are proposed to address this problem by extracting deep multiscale features from the hyperspectral image. Moreover, deep metrics usually accompany with MS-CNNs to improve the representational ability for the hyperspectral image. However, the usual metric learning would make the metric parameters in the learned model tend to behave similarly. This similarity leads to obvious model's redundancy and, thus, shows negative effects on the description ability of the deep metrics. Traditionally, determinantal point process (DPP) priors, which encourage the learned factors to repulse from one another, can be imposed over these factors to diversify them. Taking advantage of both the MS-CNNs and DPP-based diversity-promoting deep metrics, this paper develops a CNN with multiscale convolution and diversified metric to obtain discriminative features for hyperspectral image classification. Experiments are conducted over four real-world hyperspectral image data sets to show the effectiveness and applicability of the proposed method. Experimental results show that our method is better than original deep models and can produce comparable or even better classification performance in different hyperspectral image data sets with respect to spectral and spectral-spatial features.

The classification of hyperspectral images (HSIs) using convolutional neural networks (CNNs) has recently drawn significant attention. However, it is important to address the potential overfitting problems that CNN-based methods suffer when dealing with HSIs. Unlike common natural images, HSIs are essentially three-order tensors which contain two spatial dimensions and one spectral dimension. As a result, exploiting both spatial and spectral information is very important for HSI classification. This paper proposes a new hand-crafted feature extraction method, based on multiscale covariance maps (MCMs), that is specifically aimed at improving the classification of HSIs using CNNs. The proposed method has the following distinctive advantages. First, with the use of covariance maps, the spatial and spectral information of the HSI can be jointly exploited. Each entry in the covariance map stands for the covariance between two different spectral bands within a local spatial window, which can absorb and integrate the two kinds of information (spatial and spectral) in a natural way. Second, by means of our multiscale strategy, each sample can be enhanced with spatial information from different scales, increasing the information conveyed by training samples significantly. To verify the effectiveness of our proposed method, we conduct comprehensive experiments on three widely used hyperspectral data sets, using a classical 2-D CNN (2DCNN) model. Our experimental results demonstrate that the proposed method can indeed increase the robustness of the CNN model. Moreover, the proposed MCMs+2DCNN method exhibits better classification performance than other CNN-based classification strategies and several standard techniques for spectral-spatial classification of HSIs.

Most of the existing hyperspectral anomaly detectors only consider the difference between anomaly pixels and background pixels. These methods may mistakenly detect some regions in a complex background that contains various ground covers, since some background regions and anomaly objects may have similar spectral or spatial characteristics. Therefore, with a new perspective, this paper introduces a novel manifold ranking-based detection framework (MRDF). In addition to capturing the difference between anomaly pixels and background pixels, this detection framework exploits the similarity between anomaly pixels for detection. Specifically, the proposed detection framework comprises three main steps. First, the Reed-Xiaoli method is applied to capture the spectral difference between anomaly pixels and background pixels, and an initial detection map can be obtained. A set of anomaly queries are obtained automatically by employing the binary segmentation to the initial detection map. Then, we construct a closed-loop graph to characterize the spatial similarity between adjoining nodes where each node is a superpixel. Finally, a manifold ranking technique is employed to estimate the ranking value of every node based on the similarity between the test node and anomaly queries. By normalizing the ranking value of each node, a final detection map is generated. Abundant experiments are conducted on four real hyperspectral data sets. It is found that the proposed detection framework obtains a better detection performance than the current state-of-the-art detectors.

Hyperspectral image (HSI) classification has become a hot topic in the field of remote sensing. In general, the complex characteristics of hyperspectral data make the accurate classification of such data challenging for traditional machine learning methods. In addition, hyperspectral imaging often deals with an inherently nonlinear relation between the captured spectral information and the corresponding materials. In recent years, deep learning has been recognized as a powerful feature-extraction tool to effectively address nonlinear problems and widely used in a number of image processing tasks. Motivated by those successful applications, deep learning has also been introduced to classify HSIs and demonstrated good performance. This survey paper presents a systematic review of deep learning-based HSI classification literatures and compares several strategies for this topic. Specifically, we first summarize the main challenges of HSI classification which cannot be effectively overcome by traditional machine learning methods, and also introduce the advantages of deep learning to handle these problems. Then, we build a framework that divides the corresponding works into spectral-feature networks, spatial-feature networks, and spectral-spatial-feature networks to systematically review the recent achievements in deep learning-based HSI classification. In addition, considering the fact that available training samples in the remote sensing field are usually very limited and training deep networks require a large number of samples, we include some strategies to improve classification performance, which can provide some guidelines for future studies on this topic. Finally, several representative deep learning-based classification methods are conducted on real HSIs in our experiments.

Mislabeled training samples may have a negative effect on the performance of hyperspectral image classification. In order to solve this problem, a new density peak (DP) clustering-based noisy label detection method is proposed, which consists of the following steps. First, the distances among the training samples of each class are calculated using four representative distance metrics, i.e., the Euclidean distance (ED), orthogonal projection divergence (OPD), spectral information divergence (SID), and correlation coefficient (CC). Then, the local density of each training sample can be obtained using the DP clustering algorithm. Finally, a local density-based decision function is used to detect the noisy labels. The effectiveness of the proposed method is evaluated using the support vector machines on several real hyperspectral data sets. Experimental results demonstrate that the proposed noisy label detection method indeed helps in improving the classification performance.

Land-cover classification with hyperspectral imagery has been an active topic in the remote sensing community. It aims at relating a unique class label to each pixel in the scene, so that it can be well defined by a given land cover type. In this paper, we explore the intrinsic characteristics of hyperspectral imagery from a subpixel-level perspective and propose a new subpixel component analysis (SCA) approach for feature extraction and land-cover classification. The core idea of SCA is that we extract a subpixel attribute component feature from the abundance maps. Compared with the abundance maps, the extracted subpixel feature image shows higher signal-to-noise level and clearer spatial distribution details. In order to deal with spectral variability, as well as obtain representative image endmember signatures and their corresponding abundance maps, we adopt a regional clustering-based spatial preprocessing (RCSPP) strategy for endmember identification, and a partial unmixing model based on mixture tuned matched filtering (MTMF) for abundance estimation. Furthermore, to highlight the spatial distribution details as well as eliminate the noise disturbance in the derived abundance maps, we perform sparse image decomposition on the obtained abundance maps, thus achieving a new subpixel feature representation for classification. Our experimental results reveal that the proposed SCA approach can obtain feature representation with explicit physical meaning, clear spatial distribution details, and better noise robustness, leading to state-of-the-art classification results.

Natural images often contain patches with high similarity. In this paper, to effectively utilize the local and nonlocal self-similarity for low-rank models, we propose a novel weighted tensor rank-1 decomposition method (terms as WTR1) for nonlocal image denoising. Although the low-rank approximation problem has been well studied for matrices, it remains elusive of the theoretically extension to tensors due to the NPhard tensor decomposition. To tackle this problem, the proposed WTR1 method designs a new efficient CANDECOMP/PARAFAC (CP) decomposition algorithm and constructs a straightforward low-rank tensor approximation strategy. This is achieved by elegantly manipulating the CP-rank, called intrinsic low-rank tensor approximation. Specifically, the WTR1 method first groups similar patches into a 3-D stack and converts the stack into a finite sum of rank-1 products. Then, we deploy the intrinsic low-rank tensor approximation to produce the final denoised image. The proposed WTR1 method can jointly exploit the local and nonlocal self-similarity, thus improving the nonlocal image denoising quality. Experimental results have shown that the proposed WTR1 outperforms several state-of-the-art denoising methods.

Recently, combining a low spatial resolution hyperspectral image (LR-HSI) with a high spatial resolution multispectral image (HR-MSI) into an HR-HSI has become a popular scheme to enhance the spatial resolution of HSI. We propose a novel subspace-based low tensor multi-rank regularization method for the fusion, which fully exploits the spectral correlations and non-local similarities in the HR-HSI. To make use of high spectral correlations, the HR-HSI is approximated by spectral subspace and coefficients. We first learn the spectral subspace from the LR-HSI via singular value decomposition, and then estimate the coefficients via the low tensor multi-rank prior. More specifically, based on the learned cluster structure in the HR-MSI, the patches in coefficients are grouped. We collect the coefficients in the same cluster into a three-dimensional tensor and impose the low tensor multi-rank prior on these collected tensors, which fully model the non-local self-similarities in the HR-HSI. The coefficients optimization is solved by the alternating direction method of multipliers. Experiments on two public HSI datasets demonstrate the advantages of tour method.

Automatic and accurate classification of retinal optical coherence tomography (OCT) images is essential to assist ophthalmologist in the diagnosis and grading of macular diseases. Clinically, ophthalmologists usually diagnose macular diseases according to the structures of macular lesions, whose morphologies, size, and numbers are important criteria. In this paper, we propose a novel lesion-aware convolutional neural network (LACNN) method for retinal OCT image classification, in which retinal lesions within OCT images are utilized to guide the CNN to achieve more accurate classification. The LACNN simulates the ophthalmologists' diagnosis that focuses on local lesion-related regions when analyzing the OCT image. Specifically, we first design a lesion detection network to generate a soft attention map from the whole OCT image. The attention map is then incorporated into a classification network to weight the contributions of local convolutional representations. Guided by the lesion attention map, the classification network can utilize the information from local lesion-related regions to further accelerate the network training process and improve the OCT classification. Our experimental results on two clinically acquired OCT datasets demonstrate the effectiveness and efficiency of the proposed LACNN method for retinal OCT image classification.

Hyperspectral images (HSIs) with high spectral resolution only have the low spatial resolution. On the contrary, multispectral images (MSIs) with much lower spectral resolution can be obtained with higher spatial resolution. Therefore, fusing the high-spatial-resolution MSI (HR-MSI) with low-spatial-resolution HSI of the same scene has become the very popular HSI super-resolution scheme. In this paper, a novel low tensor-train (TT) rank (LTTR)-based HSI super-resolution method is proposed, where an LTTR prior is designed to learn the correlations among the spatial, spectral, and nonlocal modes of the nonlocal similar high-spatial-resolution HSI (HR-HSI) cubes. First, we cluster the HR-MSI cubes as many groups based on their similarities, and the HR-HSI cubes are also clustered according to the learned cluster structure in the HR-MSI cubes. The HR-HSI cubes in each group are much similar to each other and can constitute a 4-D tensor, whose four modes are highly correlated. Therefore, we impose the LTTR constraint on these 4-D tensors, which can effectively learn the correlations among the spatial, spectral, and nonlocal modes because of the well-balanced matricization scheme of TT rank. We formulate the super-resolution problem as TT rank regularized optimization problem, which is solved via the scheme of alternating direction method of multipliers. Experiments on HSI data sets indicate the effectiveness of the LTTR-based method.

In visual recognition and surveillance system, human face is one of the most important factors. Unfortunately, due to the low-cost imaging sensors and the complexity imaging environment, the captured face images are always low-resolution (LR) and corrupted by noise. The noisy LR face images possess limited useful information, which will extremely degrade the performance of face recognition system. To address this issue, in this paper we presented a K-nearest neighbor (K-NN) Regularized Collaborative Representation (K-RCR) method to simultaneously enhance the resolution of face images and suppress the noise. The proposed K-RCR breaks the bottlenecks of patch based face super-resolution methods, which makes it to be a reality that denoising and super-resolution can be achieved in a unified framework. Specifically, the K-NN selection strategy is employed to use the most important K nearest neighbors in the training dataset to collaboratively represent the test patch, leading to a unique and stable solution for the least squares problem. Moreover, a diagonal weight matrix is incorporated into the objective function to equip it more robust to noise. Experimental results on the standard test face dataset, i.e., FEI, demonstrate the superiority of our proposed method over several state-of-the-art face image super-resolution methods.

This paper presents a subpixel spatial-spectral feature mining approach for hyperspectral image classification. First, a regional clustering-based spatial preprocessing (RCSPP) strategy is introduced to identify the endmember signatures from the original image. Then, a partial unmixing model of mixture tuned matched filtering (MTMF) is adopted to estimate the abundance maps. Finally, the morphological component analysis (MCA) is adopted to decompose the abundance map into different spatial morphological components, and the smoothness components are chosen for classification. The experimental results reveal that the obtained subpixel spatial-spectral feature can lead to very good classification accuracies.

Compressed sensing (CS) technique contributes to reduce the burden of storage and transmission for hyperspectral images (HSIs) which are large 3D data cubes. Previous CS methods usually adopt sensing matrixes whose column number is equal to the length of the signal to sample data. As the length of the signal increases, the sensing matrix can be very large, especially for high-dimensional hyperspectral data. To overcome the drawback, a new semi-tensor based CS (ST-CS) method is proposed to for HSIs. In the sampling model, we construct a semi-tensor sensing matrix whose column number is much smaller than the length of spectral pixel. Then, the semi-tensor product, which breaks the dimension matching condition for matrix multiplication, is applied to the data sampling. The lower-dimensional sensing matrixes can reduce the data storage in the onboard system. In the sparse coding and reconstruction model, the spatial correlation of spectral pixels is exploited by introducing a regularization. The regularization tries to push reconstructed neighboring pixels to be similar. As a consequence, some unsuccessfully reconstructed pixels may be corrected by the use of neighbor information. Furthermore, the spatial structure of the HSI can be better reconstructed. Experimental results show the effectiveness of the proposed method.

Remotely sensed hyperspectral images (HSIs) usually have high spectral resolution but low spatial resolution. A way to increase the spatial resolution of HSIs is to solve a fusion inverse problem, which fuses a low spatial resolution HSI (LR-HSI) with a high spatial resolution multispectral image (HR-MSI) of the same scene. In this paper, we propose a novel HSI super-resolution approach (called LRSR), which formulates the fusion problem as the estimation of a spectral dictionary from the LR-HSI and the respective regression coefficients from both images. The regression coefficients are estimated by formulating a variational regularization problem which promotes local (in the spatial sense) low-rank and sparse regression coefficients. The local regions, where the spectral vectors are low-rank, are estimated by segmenting the HR-MSI. The formulated convex optimization is solved with SALSA. Experiments provide evidence that LRSR is competitive with respect to the state-of-the-art methods.

In this paper, a novel multi-scale structure extraction based spectral-spatial hyperspectral image classification method is proposed, which consists of the following steps. First, the spectral dimension of the hyperspectral image is reduced by averaging adjacent spectral bands. Then, in order to extract the multi-scale significant structural features (MSFs) which are insensitive to image noise and texture, a relative total variation based structure extraction method is applied on the dimension reduced hyperspectral image. Finally, the MSFs are fused together with the kernel principal component analysis (KPCA), so as to obtain the kernel PCA fused multi-scale structural features (KPCA-MSFs) for classification. Experiments conducted on a real hyperspectral image demonstrate the outstanding performance of the proposed approach over several state-of-the-art spectral-spatial classifiers, especially when the image is corrupted by serious scene noise.

In this paper, the effect of ground truths on performance evaluation of hyperspectral image classification is studied. The purpose is to investigate whether the accuracies in terms of three representative accuracy indexes, i.e., the overall accuracy (OA), the average accuracy (AA), and the Kappa coefficient, can be completely responsible when the ground truth is insufficient. The major contribution of this work is designing several experiments so as to subjectively and objectively analysis the influences of ground truths on performance evaluation. Furthermore, four evaluation metrics, i.e., the Pearson linear correlation coefficient (PLCC), root mean square error (RMSE), Spearmans rank correlation coefficient (SR-CC), and Kendalls rank correlation coefficient (KRCC) have been adopted to measure the robustness of different classification methods to ground truths containing different numbers of labeled pixels and the location of ground truth in the image. Based on the designed experiments, a conclusion is obtained that insufficient ground truths may affect the performance of existing accuracy indexes. This underlines that overoptimistic performance evaluations may exist when the ground truth contains a small number of labeled pixels.

In this paper, a covariance matrix based feature fusion (CMF-F) framework is proposed to combine two low-level visual features i.e., the Gabor feature and color feature for scene classification. Generally, the proposed method consists of following three steps. Firstly, the Gabor feature and color feature are extracted from original image and stacked together. Then, a covariance matrix is extracted to fuse these two low-level visual features. Each nondiagonal entry in the covariance matrix stands for the correlation of two different feature dimensions. Finally, the obtained covariance matrix is handled by a kernel linear discriminative analysis algorithm followed with nearest neighboring classifier for label assignment. The proposed method is tested on a public 21-classes UC Merced land use data set and compared with mid-level visual feature oriented method and the high-level feature oriented methods. The experimental results demonstrate that the proposed CMFF framework can not only improve the classification performance of the low-level visual feature (the Gabor feature and the color feature), but also can outperform the conventional mid-level visual feature oriented methods.

In this paper, a novel fusion framework, which is called as subpixel, pixel and superpixel fusion based anomaly detection (SPSF-AD), is proposed for hyperspectral anomaly detection. Most existing methods are based on single pixel-level data representation, which fail to fully exploit spectral-spatial information in hyperspectral image for anomaly detection. For the proposed SPSF-AD method, the first step is to separately extract subpixel, pixel, and superpixel features from HSI via the spectral unmixing, morphological operation, and super-pixel segmentation technique. Then, three sparse anomalies detection results based on different features can be generated via low-rank decomposition (LRD) technique. Finally, since the different features contain highly complementary information, an effective decision fusion technique, i.e., average operator, is employed to estimate the final anomaly detection result. Experiments on three real HSI data sets can demonstrate the effectiveness of the proposed method.

In this paper, a novel supervised band selection (BS) method based on pair-wise constraint and band-wise correlation information is proposed for the dimension reduction of hyperspectral images. On the one hand, the band-wise correlation information, is used for selecting band-subset with lower redundancy and higher representation. This process is achieved by first partitioning all spectral bands into continuous groups and then calculate a band-wise correlation matrix within each group, which is used later for selecting bands of more representation and lower redundancy. On the other hand, pair-wise supervised information (i.e., whether a pair of labeled samples are from the same class) is exploited for selecting band-subsets to better discriminate different classes. That is, a few bands are adaptively chosen for each pair of labeled samples according to spectral-similarity, to ensure that the distance between samples from different classes is far and keep sample-pair from same class close. By the joint use of both pair-wise constraint information and band-wise correlation information, the proposed BS method can lead to select optimal band-subsets with low-redundancy, high-representation and high-discrimination. Experimental results demonstrate the effectiveness of the proposed BS method.

Recently, deep learning has been introduced to extract hierarchical features of hyperspectral images (HSls) and achieved good classification performance. However, the previous deep learning based methods only consider the semantic information of individual pixel, which cannot effectively deal with the complex spectral-spatial characteristic of HSls. In this paper, we propose a novel deep learning based framework to learn the similarity-preserving deep features (SPDF) for HSI classification. Specifically, we firstly introduce a deep network that can take pairs of image patches as training samples, and then a loss function is elaborately designed to minimize the feature distance of similar pairs and maximize the feature distance of dissimilar pairs in feature space. Once the deep network is well trained, the SPDF can be obtained by propagating the samples through the trained network. Finally, these features are fed into the support vector machines (SVM) for HSI classification. Experimental results demonstrate the pro-nosed method outperforms other competitive methods.

Recently, robust principal component analysis (RPCA) based methods have been used for hyperspectral images (HSIs) restoration to simultaneously remove several types of noise, including Gaussian noise, impulse noise, stripes, and so on. However, most of these RPCA methods formulate the optimization problem with a convex l1-norm penalty, which over-penalizes large entries of vectors, and results in a biased solution. In this paper, a novel nonconvex sparsity regularizer (NonSR) for measuring the clean HSI low rank structure and noise sparsity structure is proposed, which can effectively approximate rank function and noise sparsity instead of the convex l1-norm. By embedding the sparsity regularizer into the RPCA framework, we formulate a new model, which enhance the capability in simultaneously removing several types of noise. In addition, an iterative algorithm based on the alternative direction multiplier method (ADMM) is developed to effectively solve the proposed model. Experimental results demonstrate that the proposed NonSR method outperforms state-of-the-art HSIs restoration techniques.

In this letter, a novel anomaly detection method is proposed, which can effectively fuse the multiscale information extracted by attribute and edge-preserving filters. The proposed method consists of the following steps. First, multiscale attribute and edge-preserving filters are utilized to obtain multiscale anomaly detection maps. Then, the multiscale detection maps are fused via an averaging approach, and the training samples of the anomalies and background are selected from the fused detection map. Next, the support vector machine classification is performed on the hyperspectral image to obtain an anomaly probability map. Finally, the detection result is obtained by multiplying the fused detection map and the anomaly probability map, followed by an edge-preserving filtering-based postprocessing. Experiments performed on four real hyperspectral data sets demonstrate that the proposed method shows a better detection performance with respect to several state-of-the-art hyperspectral anomaly detection methods.

In recent years, the sparse coding-based techniques have been widely used for image denoising. However, most of the sparse coding-based mixed noise reduction methods fail to take full advantage of the geometric structure of data samples. In other words, they neglect the common information shared by the similar patches in sparse coding. To address this concern, in this paper, we propose a robust constrained sparse representation (RCSR) method to remove mixed noise. By using the center coefficient of similar patches as the guider which is approximated by the coefficient of query patch in sparse coding, the geometric structure of data can be well preserved. Moreover, different from most existing two-stage mixed noise reduction methods that use explicit detectors to restrain impulse noise, the proposed RCSR adaptively adjusts the contribution of each pixel in the loss function to eliminate the influences of outliers. Experiments on the reconstruction of synthetic data and the removal of mixed noise in real images demonstrate the effectiveness of our proposed method.

Recently, locality-constrained linear coding (LLC) has been drawn great attentions and been widely used in image processing and computer vision tasks. However, the conventional LLC model is always fragile to outliers. In this paper, we present a robust locality-constrained bi-layer representation model to simultaneously hallucinate the face images and suppress noise and outliers with the assistant of a group of training samples. The proposed scheme is not only able to capture the nonlinear manifold structure but also robust to outliers by incorporating a weight vector into the objective function to subtly tune the contribution of each pixel offered in the objective. Furthermore, a high-resolution (HR) layer is employed to compensate the missed information in the low-resolution (LR) space for coding. The use of two layers (the LR layer and the HR layer) is expected to expose the complicated correlation between the LR and HR patch spaces, which helps to obtain the desirable coefficients to reconstruct the final HR face. The experimental results demonstrate that the proposed method outperforms the state-of-the-art image super-resolution methods in terms of both quantitative measurements and visual effects.

Recently, the locality linear coding (LLC) has attracted more and more attentions in the areas of image processing and computer vision. However, the conventional LLC with real setting is just designed for the grayscale image. For the color image, it usually treats each color channel individually or encodes the monochrome image by concatenating all the color channels, which ignores the correlations among different channels. In this paper, we propose a quaternion-based locality-constrained coding (QLC) model for color face hallucination in the quaternion space. In QLC, the face images are represented as quaternion matrices. By transforming the channel images into an orthogonal feature space and encoding the coefficients in the quaternion domain, the proposed QLC is expected to learn the advantages of both quaternion algebra and locality coding scheme. Hence, the QLC cannot only expose the true topology of image patch manifold but also preserve the inherent correlations among different color channels. Experimental results demonstrated that our proposed QLC method achieved superior performance in color face hallucination compared with other state-of-the-art methods.

An extinction profile (EP) is an effective spatial-spectral feature extraction method for hyperspectral images (HSIs), which has recently drawn much attention. However, the existing methods utilize the EPs in a stacking way, which is hard to fully explore the information in EPs for HSI classification. In this paper, a novel fusion framework termed EPs-fusion (EPs-F) is proposed to exploit the information within and among EPs for HSI classification. In general, EPs-F includes the following two stages. In the first stage, by extracting the EPs from three independent components of an HSI, three complementary groups of EPs can be constructed. For each EP, an adaptive superpixel-based composite kernel strategy is proposed to explore the spatial information within an EP. The weights to create the composite kernel and the number of superpixels are automatically determined based on the spatial information of each EP. In the second stage, since the different EPs contain highly complementary information, a simple yet effective decision fusion method is further applied to obtain the final classification result. Experiments on three real HSI data sets verify the qualitative and quantitative superiority of the proposed EPs-F method over several state-of-the-art HSI classifiers.

In this paper, a novel local covariance matrix (CM) representation method is proposed to fully characterize the correlation among different spectral bands and the spatial-contextual information in the scene when conducting feature extraction (FE) from hyperspectral images (HSIs). Specifically, our method first projects the HSI into a subspace, using the maximum noise fraction method. Then, for each test pixel in the subspace, its most similar neighboring pixels (within a local spatial window) are clustered using the cosine distance measurement. The test pixel and its neighbors are used to calculate a local CM for FE purposes. Each nondiagonal entry in the matrix characterizes the correlation between different spectral bands. Finally, these matrices are used as spatial-spectral features and fed to a support vector machine for classification purposes. The proposed method offers a new strategy to characterize the spatial-spectral information in the HSI prior to classification. Experimental results have been conducted using three publicly available hyperspectral data sets for classification, indicating that the proposed method can outperform several state-of-the-art techniques, especially when the training samples available are limited.

Sparse representation (SR) has been successfully used in the classification of hyperspectral images (HSIs) by representing HSI pixels over a dictionary and yielding discriminative sparse coefficients. Most of SR-based classification methods construct the dictionary by directly using some labeled pixels as atoms. Such dictionary can lead to inefficient SR for large-sized HSIs, and may be incomplete when the number of labeled pixels is less than the number of spectral bands. This paper proposes a contextual online dictionary learning (DL) method for HSIs classification, which learns a dictionary over the whole image rather than few labeled pixels. The proposed method can effectively and efficiently improve the adaptive representation capability of different pixels with an online learning mechanism. Specifically, the contextual characteristics of the HSI are integrated with discriminative spectral information for online DL, i.e., pushing similar pixels in neighborhood to share similar sparse coefficients with respect to the well-learned dictionary. By this way, the obtained sparse coefficients are structured and discriminative. Finally, a traditional classifier, i.e., the linear support vector machine, is applied to the sparse coefficients, and the final classification results are obtained. Experimental results on real HSIs show the effectiveness of the proposed method.

This paper proposes a new method, called multilayer stacked covariance pooling (MSCP), for remote sensing scene classification. The innovative contribution of the proposed method is that it is able to naturally combine multilayer feature maps, obtained by pretrained convolutional neural network (CNN) models. Specifically, the proposed MSCP-based classification framework consists of the following three steps. First, a pretrained CNN model is used to extract multilayer feature maps. Then, the feature maps are stacked together, and a covariance matrix is calculated for the stacked features. Each entry of the resulting covariance matrix stands for the covariance of two different feature maps, which provides a natural and innovative way to exploit the complementary information provided by feature maps coming from different layers. Finally, the extracted covariance matrices are used as features for classification by a support vector machine. The experimental results, conducted on three challenging data sets, demonstrate that the proposed MSCP method can not only consistently outperform the corresponding single-layer model but also achieve better classification performance than other pretrained CNN-based scene classification methods.

Imaging spectroscopy, also known as hyperspectral imaging, has been transformed in the last four decades from being a sparse research tool into a commodity product available to a broad user community. Specially, in the last 10 years, a large number of new techniques able to take into account the special properties of hyperspectral data have been introduced for hyperspectral data processing, where hyperspectral image classification, as one of the most active topics, has drawn massive attentions. Spectral-spatial hyperspectral image classification can achieve better classification performance than its pixel-wise counterpart, since the former utilizes not only the information of spectral signature but also that from spatial domain. In this paper, we provide a comprehensive overview on the methods belonging to the category of spectral-spatial classification in a relatively unified context. First, we develop a concept of spatial dependency system that involves pixel dependency and label dependency, with two main factors: neighborhood covering and neighborhood importance. In terms of the way that the neighborhood information is used, the spatial dependency systems can be classified into fixed, adaptive, and global systems, which can accommodate various kinds of existing spectral-spatial methods. Based on such, the categorizations of single-dependency, bilayer-dependency, and multiple-dependency systems are further introduced. Second, we categorize the performings of existing spectral-spatial methods into four paradigms according to the different fusion stages wherein spatial information takes effect, i.e., preprocessing-based, integrated, postprocessing-based, and hybrid classifications. Then, typical methodologies are outlined. Finally, several representative spectral-spatial classification methods are applied on real-world hyperspectral data in our experiments.

Image decolorization is known to be an effective way in transferring a color image into a gray one while well preserving the major information of all three bands. In this paper, a simple yet effective hyperspectral image visualization framework based on decolorization, named decolorization based hyperspectral visualization, is proposed, which enables us to fully exploit the benefits of decolorization technique. The proposed framework consists of the following two main steps. First, the hyperspectral image is partitioned into nine subsets of adjacent hyperspectral bands and the averaged band of each subset is calculated. Then, the dimension reduced image is further divided into three groups of adjacent bands, and the bands in each group are fused by using an image decolorization method. The main contribution of this paper is that the strong correlations in two different fields, i.e., image decolorization and hyperspectral image visualization, are first built. Experiments performed on several real hyperspectral data sets demonstrate that the proposed framework can obtain outstanding visualization performance in terms of both subjective and objective evaluations.

In this paper, a novel method is introduced to detect and correct mislabeled training samples for hyperspectral image classification. First, domain transform recursive filtering-based feature extraction is used to improve the separability of the training samples. Then, constrained energy minimization-based object detection is performed on the training set with each training sample serving as the object spectrum. Finally, the label of each training sample is verified or corrected based on the averaged detection probabilities of different classes. Experiments performed on real hyperspectral data sets demonstrate the effectiveness of the proposed method in improving classification performance with respect to the classifier trained with the original training set that contains a number of mislabeled samples.

The existence of shadows in very high resolution satellite images obstructs image interpretation and the following applications, such as target detection and recognition. Traditional shadow detection methods consider only the pixel-level properties, such as color and intensity of image pixels, and thus, may produce errors around object boundaries. To overcome this problem, a novel shadow detection algorithm based on extended random walker (ERW) is proposed by jointly integrating both shadow property and spatial correlations among adjacent pixels. First, a set of training samples is automatically generated via an improved Otsu-based thresholding method. Then, the support vector machine is applied to obtain an initial detection map, which categorizes all the pixels in the scene into shadow and nonshadow. Finally, the initial detection map is refined with the ERW model, which can simultaneously characterize the shadow property and spatial information in satellite images to further improve shadow detection accuracy. Experiments performed on five real remote sensing images demonstrate the superiority of the proposed method over several state-of-the-art methods in terms of detection accuracy.

In the field of hyperspectral image classification, a widely used way for objective performance evaluation of different classification methods is calculating three accuracy indexes, i.e., the overall accuracy, the average accuracy, and the Kappa coefficient. These accuracy indexes are obtained by comparing the classification results with the ground truth, i.e., a reference classification map labeled by human experts. In this paper, the effect of ground truths on the objective performance evaluation of hyperspectral image classification is studied. The purpose is to investigate, if the ground truth is insufficient, whether the above accuracy indexes can be completely responsible. Furthermore, in order to measure the robustness of different classification methods to those insufficient ground truths, four evaluation metrics, i.e., the Pearson linear correlation coefficient, root-mean-square error, Spearman's rank correlation coefficient, and Kendall's rank correlation coefficient have been adopted for further analysis. Based on these experiments, an interesting conclusion can be obtained that insufficient ground truths may limit the assessment capability of existing accuracy indexes. This underlines that overoptimistic performance evaluations may exist and stresses the demand of designing more appropriate accuracy indexes for objective performance evaluation with insufficient ground truths.

Recently, deep learning has been introduced to classify hyperspectral images (HSIs) and achieved good performance. In general, deep models adopt a large number of hierarchical layers to extract features. However, excessively increasing network depth will result in some negative effects (e.g., overfitting, gradient vanishing, and accuracy degrading) for conventional convolutional neural networks. In addition, the previous networks used in HSI classification do not consider the strong complementary yet correlated information among different hierarchical layers. To address the above two issues, a deep feature fusion network (DFFN) is proposed for HSI classification. On the one hand, the residual learning is introduced to optimize several convolutional layers as the identity mapping, which can ease the training of deep network and benefit from increasing depth. As a result, we can build a very deep network to extract more discriminative features of HSIs. On the other hand, the proposed DFFN model fuses the outputs of different hierarchical layers, which can further improve the classification accuracy. Experimental results on three real HSIs demonstrate that the proposed method outperforms other competitive classifiers.

In remote sensing image classification, active learning aims to learn a good classifier as best as possible by choosing the most valuable (informative and representative) training samples. Multiview is a concept that regards analyzing the same object from multiple different views. Generally, these views show diversity and complementarity of features. In this paper, we propose a new multiview active learning (MVAL) framework for hyperspectral image classification. First, we generate multiple views by extracting different attribute components from the same image data. Specifically, we adopt the multiple morphological component analysis to decompose the original image into multiple pairs of attribute components, including content, coarseness, contrast, and directionality, and the smooth component from each pair is chosen as one single view. Second, we construct two multiview intensity-based query strategies for active learning. On the one hand, we exploit the intensity differences of multiple views along with the samples' uncertainty to choose the most informative candidates. On the other hand, we consider the clustering distribution of all unlabeled samples, and query the most representative candidates in addition to the highly informative ones. Our experiments are performed on four benchmark hyperspectral image data sets. The obtained results show that the proposed MVAL framework can lead to better classification performance than the traditional, single-view active learning schemes. In addition, compared with the conventional disagree-based MVAL scheme, the proposed query selection strategies show competitive classification accuracy.

Fusing a low spatial resolution hyperspectral image (LR-HSI) with a high spatial resolution multispectral image (HR-MSI) to obtain a high spatial resolution hyperspectral image (HR-HSI) has attracted increasing interest in recent years. In this paper, we propose a coupled sparse tensor factorization (CSTF) based approach for fusing such images. In the proposed CSTF method, we consider an HR-HSI as a three-dimensional tensor and redefine the fusion problem as the estimation of a core tensor and dictionaries of the three modes. The high spatial-spectral correlations in the HR-HSI are modeled by incorporating a regularizer which promotes sparse core tensors. The estimation of the dictionaries and the core tensor are formulated as a coupled tensor factorization of the LR-HSI and of the HR-MSI. Experiments on two remotely sensed HSIs demonstrate the superiority of the proposed CSTF algorithm over current state-of-the-art HSI-MSI fusion approaches.

Hyperspectral image (HSI) sharpening, which aims at fusing an observable low spatial resolution (LR) HSI (LR-HSI) with a high spatial resolution (HR) multispectral image (HR-MSI) of the same scene to acquire an HR-HSI, has recently attracted much attention. Most of the recent HSI sharpening approaches are based on image priors modeling, which are usually sensitive to the parameters selection and time-consuming. This paper presents a deep HSI sharpening method (named DHSIS) for the fusion of an LR-HSI with an HR-MSI, which directly learns the image priors via deep convolutional neural network-based residual learning. The DHSIS method incorporates the learned deep priors into the LR-HSI and HR-MSI fusion framework. Specifically, we first initialize the HR-HSI from the fusion framework via solving a Sylvester equation. Then, we map the initialized HR-HSI to the reference HR-HSI via deep residual learning to learn the image priors. Finally, the learned image priors are returned to the fusion framework to reconstruct the final HR-HSI. Experimental results demonstrate the superiority of the DHSIS approach over existing state-of-the-art HSI sharpening approaches in terms of reconstruction accuracy and running time.

Hyperspectral image (HSI) super-resolution, which fuses a low-resolution (LR) HSI with a high-resolution (HR) multispectral image (MSI), has recently attracted much attention. Most of the current HSI super-resolution approaches are based on matrix factorization, which unfolds the three-dimensional HSI as a matrix before processing. In general, the matrix data representation obtained after the matrix unfolding operation makes it hard to fully exploit the inherent HSI spatial-spectral structures. In this paper, a novel HSI super-resolution method based on non-local sparse tensor factorization (called as the NLSTF) is proposed. The sparse tensor factorization can directly decompose each cube of the HSI as a sparse core tensor and dictionaries of three modes, which reformulates the HSI super-resolution problem as the estimation of sparse core tensor and dictionaries for each cube. To further exploit the non-local spatial self-similarities of the HSI, similar cubes are grouped together, and they are assumed to share the same dictionaries. The dictionaries are learned from the LR-HSI and HR-MSI for each group, and corresponding sparse core tensors are estimated by spare coding on the learned dictionaries for each cube. Experimental results demonstrate the superiority of the proposed NLSTF approach over several state-of-the-art HSI super-resolution approaches.

Sparse representation (SR) based hyperspectral image (HSI) classification is a rapidly evolving research topic. How to construct an optimized dictionary to better characterize spectral-spatial features of HSI is an important problem. In this paper, a novel spectral-spatial online dictionary learning (SSODL) method for HSI classification is proposed. The main idea is to learn a complete and discriminative dictionary by exploiting both spatial and spectral information all over the whole image. Rather than only using training samples for dictionary construction, the online dictionary learning (ODL) mechanism can effectively improve the adaptive representation capability of different pixels. Specifically, the contextual characteristics of HSI are integrated with discriminative spectral information for the ODL, i.e., pushing similar pixels in neighborhood to share similar sparse coefficients w.r.t. the well learnt dictionary. By this way, the yielding sparse coefficients are structured and discriminative. Finally, a traditional classifier, i.e., linear support vector mechine (SVM), is applied to the sparse coefficients and the final classification results are obtained. Experimental results on real HSIs show the effectiveness of the proposed method.

Extinction profile (EP) is an effective feature extraction method which can well preserve the geometrical characteristics of a hyperspectral image (HSI) and by extracting the EP from first three independent components (ICs) of an HSI, three correlated and complementary groups of EP features can be constructed. In this paper, an EPs fusion (EPs-F) strategy is proposed for HSI classification by exploring spatial-spectral information within and among three EP features. In general, the EPs-F method includes two stages. In the first stage, within each EP feature, a superpixel-based composite kernel strategy is proposed to adaptively fuse the spatial information of EP and the spectral feature of HSI. Then, the obtained adaptive composite kernel is used to create a classification map for each EP. In the second stage, decision fusion is further applied on different classification maps to create the final classification result. Experiments on two real HSIs verify the effectiveness of the proposed EPs-F algorithm.

In this paper, a novel spectral-spatial very high resolution images shadow detection algorithm based on random walker is proposed. First, a set of training samples is obtained by an improved Otsu based thresholding method automatically. Then, a widely used pixel-wise classifier, i.e., the Support Vector Machine (SVM), is applied to obtain an initial binary classification map. Finally, the initial classification map is refined with the extended random walker model, which can jointly integrating both the spectral characteristics and spatial-correlation among adjacent pixels to further improve shadow detection accuracy. Experimental results performed on real data sets demonstrate the superiority of the proposed method over several state-of-the-art methods.

Recently, low-rank matrix recovery has been demonstrated to be an effective tool in hyperspectral images (HSIs) denoising. However, the previous low-rank matrix recovery method with a window of the fixed-shape cannot adaptively exploit spatial structure information and nonlocal similarity. In this paper, multiscale low-rank matrix recovery (MC-LRMR) is proposed to recover HSI corrupted by different kinds of noise. The proposed method contains three main steps. First, HSI is transformed by the principal component analysis (PCA) algorithm and multiscale superpixel segmentation is applied to the first principal component, to segment the HSI into non-overlapping homogeneous regions. Then, the mixed noises, including Gaussian noise, impulse noise, dead lines noise, stripes noise, are removed by the low-rank matrix recovery (LRMR) in a superpixel-by-superpixel manner. Finally, a fusion rule, i.e., average operator, is adopted to combine denoising results of various scales, to acquire a fused noise-free estimation. Experiments on simulated and real HSI data sets can demonstrate the effectiveness of the proposed method.

Hyperspectral image classification, an astonishing tool to distinguish the land covers in remote sensed hyperspectral images, has been investigated by multiple disciplines such as geoscience, environmental science, mathematics, and computer vision. Following early machine learning (e.g., support vector machines and neural networks) and feature extraction theories (e.g., principal component analysis), hundreds of hyperspectral image classification algorithms have been proposed in order to further improve the classification accuracies. However, it is still unclear what are the real improvements of the newly proposed methods in this field or we are just fitting models to some specific data sets? To address this problem, this paper aims at discussing the major motivations and ideas in conducting a comprehensive benchmark analysis for hyperspectral image classification. The benchmark should not only allows researchers to compare their models with other algorithms but also helps identify the chief factors affecting the performance of their classification methods.

In this paper, a novel model of Gabor Filtering based Deep Network (GFDN) for hyperspectral image classification is proposed. First, spatial features are extracted via Gabor filtering from the three principal components. Gabor filter can capture physical structures of hyperspectral images, such as specific orientation information. Then, the Gabor features and spectral features are simply staked to form combined features. Finally, high-level features are learnt by a stacked sparse auto-encoder deep network. Since the limited training samples negatively affect the classification performance in deep learning, here, an effective way is designed to simulate more training samples. By using both the real and virtual samples, the parameters of deep network can be better learnt and updated, leading to more robust and accurate classification results. Experiments on the real hyperspectral data set reveal the superior performance of the proposed method over some well-known classification methods.

In this paper, a novel iterative clustering based active learning (ICAL) method for hyperspectral image classification is proposed. On the one hand, the extreme learning machine is combined with the Markov random field (ELM-MRF) for label assignment, to exploit both spectral and spatial information to boost classification result. On the other hand, an iterative clustering based sample selection strategy is introduced to optimally choose the most informative training sample set. This strategy first selects a candidate set of samples, according to the differential map that is obtained by comparing the ELM-MRF based classification results in adjacent iterations. Then, all the pixels in the candidate set are clustered according to spectral characteristics. Finally, from each cluster, the one sample with the highest uncertainty is added to the new training sample set. By this sample selection strategy, the diversity and uncertainty of training samples can be maximized, which can further contribute to the improvement of classification performance. Experimental results show that the proposed ICAL method can achieve competitive classification results even with a limited number of labeled samples.

Recently, deep learning has been introduced to classify hyperspectral images (HSIs) and achieved effective performance. In general, the previous networks are not enough deep, which might not extract very discriminant features for classification. In addition, they do not consider strong correlations among different hierarchical layers. Due to the two problems, a hybrid deep residual network is presented for HSIs classification in this paper. The proposed method firstly employs deep residual network (DRN) to extract very deep and discriminant features of HSIs. The DRN can help to overcome the decrease of classification accuracy that is caused by the increasing network depth and limited available training samples. Moreover, by incorporating different hierarchical features of network with a hybrid mechanism, the classification results can be further improved. Experimental results on a real hyperspectral image demonstrate that the proposed method outperforms other competitive methods.

Remote sensing is a powerful technology for Earth observation (EO), and it plays an essential role in many applications, including environmental monitoring, precision agriculture, resource managing, urban characterization, disaster and emergency response, etc. However, due to limitations in the spectral, spatial, and temporal resolution of EO sensors, there are many situations in which remote sensing data cannot be fully exploited, particularly in the context of emergency response (i.e., applications in which real/near-real-time response is needed). Recently, with the rapid development and availability of social media data, new opportunities have become available to complement and fill the gaps in remote sensing data for emergency response. In this paper, we provide an overview on the integration of social media and remote sensing in time-critical applications. First, we revisit the most recent advances in the integration of social media and remote sensing data. Then, we describe several practical case studies and examples addressing the use of social media data to improve remote sensing data and/or techniques for emergency response.

Sparse representation (SR) can transform spectral signatures of hyperspectral pixels into sparse coefficients with very few nonzero entries, which can efficiently be used for compression. In this paper, a spectral-spatial adaptive SR (SSASR) method is proposed for hyperspectral image (HSI) compression by taking advantage of the spectral and spatial information of HSIs. First, we construct superpixels, i.e., homogeneous regions with adaptive sizes and shapes, to describe HSIs. Since homogeneous regions usually consist of similar pixels, pixels within each superpixel will be similar and share similar spectral signatures. Then, the spectral signatures of each superpixel can be simultaneously coded in the SR model to exploit their joint sparsity. Since different superpixels generally have different performances of SR, their rate-distortion performances in the sparse coding will be different. To achieve the best possible overall rate-distortion performance, an adaptive coding scheme is introduced to adaptively assign distortions to superpixels. Finally, the obtained sparse coefficients are quantized and entropy coded and constitute the final bitstream with the coded superpixel map. The experimental results over several HSIs show that the proposed SSASR method outperforms some state-of-the-art HSI compression methods in terms of the rate-distortion and spectral fidelity performances.

Edge-preserving features (EPFs) obtained by the application of edge-preserving filters to hyperspectral images (HSIs) have been found very effective in characterizing significant spectral and spatial structures of objects in a scene. However, a direct use of the EPFs can be insufficient to provide a complete characterization of spatial information when objects of different scales are present in the considered images. Furthermore, the edge-preserving smoothing operation unavoidably decreases the spectral differences among objects of different classes, which may affect the following classification. To overcome these problems, in this paper, a novel principal component analysis (PCA)-based EPFs (PCA-EPFs) method for HSI classification is proposed, which consists of the following steps. First, the standard EPFs are constructed by applying edge-preserving filters with different parameter settings to the considered image, and the resulting EPFs are stacked together. Next, the spectral dimension of the stacked EPFs is reduced with the PCA, which not only can represent the EPFs in the mean square sense but also highlight the separability of pixels in the EPFs. Finally, the resulting PCA-EPFs are classified by a support vector machine (SVM) classifier. Experiments performed on several real hyperspectral data sets show the effectiveness of the proposed PCA-EPFs, which sharply improves the accuracy of the SVM classifier with respect to the standard edge-preserving filtering-based feature extraction method, and other widely used spectral-spatial classifiers.

A novel method for anomaly detection in hyperspectral images is proposed. The method is based on two ideas. First, compared with the surrounding background, objects with anomalies usually appear with small areas and distinct spectral signatures. Second, for both the background and the objects with anomalies, pixels in the same class are usually highly correlated in the spatial domain. In this paper, the pixels with specific area property and distinct spectral signatures are first detected with attribute filtering and a Boolean map-based fusion approach in order to obtain an initial pixel-wise detection result. Then, the initial detection result is refined with edge-preserving filtering to make full use of the spatial correlations among adjacent pixels. Compared with other widely used anomaly detection methods, the experimental results obtained on real hyperspectral data sets including airport, beach, and urban scenes demonstrate that the performance of the proposed method is quite competitive in terms of computing time and detection accuracy.

Supervised classification of hyperspectral images (HSI) is a very challenging task due to the existence of noisy and mixed spectral characteristics. Recently, the widely developed spectral unmixing techniques offer the possibility to extract spectral mixture information at a subpixel level, which can contribute to the categorization of seriously mixed spectral pixels. Besides, it has been demonstrated that the discrimination between different materials will be improved by integrating the geometry and structure information, which can be derived from the variance between neighboring pixels. Furthermore, by incorporating the spatial context, the superpixel-based spectral-spatial similarity information can be used to smooth classification results in homogeneous regions. Therefore, a novel fusion framework for HSI classification that combines subpixel, pixel, and superpixel-based complementary information is proposed in this paper. Here, both feature fusion and decision fusion schemes are introduced. For the feature fusion scheme, the first step is to extract subpixel-level, pixel-level, and superpixel-level features from HSI, respectively. Then, the multiple feature-induced kernels are fused to form one composite kernel, which is incorporated with a support vector machine (SVM) classifier for label assignment. For the decision fusion scheme, class probabilities based on three different features are estimated by the probabilistic SVM classifier first. Then, the class probabilities are adaptively fused to form a probabilistic decision rule for classification. Experimental results tested on different real HSI images can demonstrate the effectiveness of the proposed fusion schemes in improving discrimination capability, when compared with the classification results relied on each individual feature.

Hyperspectral imaging (HSI) classification has become a popular research topic in recent years, and effective feature extraction is an important step before the classification task. Traditionally, spectral feature extraction techniques are applied to the HSI data cube directly. This paper presents a novel algorithm for HSI feature extraction by exploiting the curvelet-transformed domain via a relatively new spectral feature processing technique—singular spectrum analysis (SSA). Although the wavelet transform has been widely applied for HSI data analysis, the curvelet transform is employed in this paper since it is able to separate image geometric details and background noise effectively. Using the support vector machine classifier, experimental results have shown that features extracted by SSA on curvelet coefficients have better performance in terms of classification accuracy over features extracted on wavelet coefficients. Since the proposed approach mainly relies on SSA for feature extraction on the spectral dimension, it actually belongs to the spectral feature extraction category. Therefore, the proposed method has also been compared with some state-of-the-art spectral feature extraction techniques to show its efficacy. In addition, it has been proven that the proposed method is able to remove the undesirable artifacts introduced during the data acquisition process. By adding an extra spatial postprocessing step to the classified map achieved using the proposed approach, we have shown that the classification performance is comparable with several recent spectral–spatial classification methods.

Active learning (AL) and semisupervised learning (SSL) are both promising solutions to hyperspectral image classification. Given a few initial labeled samples, this work combines AL and SSL in a novel manner, aiming to obtain more manually labeled and pseudolabeled samples and use them together with the initial labeled samples to improve the classification performance. First, based on a comparison of the segmentation and spectral-spatial classification results obtained by random walker (RW) and extended RW (ERW) algorithms, the unlabeled samples are separated into two different sets, i.e., low- and high-confidence unlabeled data sets. For the high-confidence unlabeled data, pseudolabeling is performed, which can ensure the correctness and informativeness of the pseudolabeled samples. For the low-confidence unlabeled data, AL is used to select samples. In this way, the samples which are more effective for improvement of classification performance can be labeled in only a few iterations. Finally, with the learned training set and the original hyperspectral image as inputs, the ERW classifier is used to obtain the final classification result. Experiments performed on three real hyperspectral data sets show that the proposed method can achieve competitive classification accuracy even with a very limited number of manually labeled samples.

In the literature of remote sensing, deep models with multiple layers have demonstrated their potentials in learning the abstract and invariant features for better representation and classification of hyperspectral images. The usual supervised deep models, such as convolutional neural networks, need a large number of labeled training samples to learn their model parameters. However, the real-world hyperspectral image classification task provides only a limited number of training samples. This paper adopts another popular deep model, i.e., deep belief networks (DBNs), to deal with this problem. The DBNs allow unsupervised pretraining over unlabeled samples at first and then a supervised fine-tuning over labeled samples. But the usual pretraining and fine-tuning method would make many hidden units in the learned DBNs tend to behave very similarly or perform as “dead” (never responding) or “potential over-tolerant” (always responding) latent factors. These results could negatively affect description ability and thus classification performance of DBNs. To further improve DBN's performance, this paper develops a new diversified DBN through regularizing pretraining and fine-tuning procedures by a diversity promoting prior over latent factors. Moreover, the regularized pretraining and fine-tuning can be efficiently implemented through usual recursive greedy and back-propagation learning framework. The experiments over real-world hyperspectral images demonstrated that the diversity promoting prior in both pretraining and fine-tuning procedure lead to the learned DBNs with more diverse latent factors, which directly make the diversified DBNs obtain much better results than original DBNs and comparable or even better performances compared with other recent hyperspectral image classification methods.

We demonstrate the usefulness of utilizing a segmentation step for improving the performance of sparsity based image reconstruction algorithms. In specific, we will focus on retinal optical coherence tomography (OCT) reconstruction and propose a novel segmentation based reconstruction framework with sparse representation, termed segmentation based sparse reconstruction (SSR). The SSR method uses automatically segmented retinal layer information to construct layer-specific structural dictionaries. In addition, the SSR method efficiently exploits patch similarities within each segmented layer to enhance the reconstruction performance. Our experimental results on clinical-grade retinal OCT images demonstrate the effectiveness and efficiency of the proposed SSR method for both denoising and interpolation of OCT images.

In this paper, a non-local based sparse representation (called as the NLSR) is proposed for the super-resolution of hyperspectral image. Specifically, the NLSR firstly uses the non-local Kmeans to partition pixels of low spatial resolution hyperspectral image into several classes. The non-local Kmeans can exploit the similar patterns and structures of the low spatial resolution image to enhance the efficiency of the sparse solution. Then, the sparse representation is independently applied on each class of the low resolution hyperspectral image and high spatial resolution multispectral image to obtain the high resolution hyperspectral image. Experimental results demonstrate the superiority of the proposed NLSR method over several well-known super-resolution methods.

In this paper, a decision fusion of pixel-level and superpixel-level classifiers (DFPSC) for the HSI is proposed. First, the support vector machine based classification probability combined with the local spatial information is introduced to classify the HSI in a pixel-by-pixel manner. Then, the HSI is over-segmented into non-overlapping superpixels. Each superpixel contains spatially-connected and spectrally-similar pixels, which are assigned to the same label via joint sparse regularization. Finally, a guided map is generated based on the edge map and superpixel map, which is used to guide the fusion of both pixel-level and superpixel-level classification results. With the proposed decision fusion scheme, the classification results in homogeneous and structural areas can be better balanced, leading to the improvement of the overall classification accuracy. The experimental results demonstrate the superiority of the proposed method over some well-known classification methods.

Traditional joint sparse representation based hyperspectral classification methods define a local region for each pixel. Through representing the pixels within the local region simultaneously, the class of the central pixel is able to be decided. A common limitation of this kind of methods is that only local pixels are considered in such methods, and thus, non-local information will be ignored. In order to improve the classification accuracy with the non-local information of hyperspectral images, a novel hyperspectral image classification based on K nearest neighbors (KNN) sparse representation is proposed in this paper. First, a feature space is defined based on the first principal components of the hyperspectral image and the spatial coordinates of different pixels. Then, in the defined feature space, K non-local neighborhoods of each pixel are found by using the KNN searching scheme. At last, through jointly representing the K pixels with the joint sparse model and comparing the representation residuals, the label of each pixel can be determined. Experiments performed on a widely used real HSI data set show that the proposed method obtain better classification performances when compared with the traditional joint sparse representation method and other recently proposed hyperspectral image classification methods.

Recently, the superpixel segmentation is introduced into the hyperspectral image (HSI) classification to exploit the spatial information. However, the size of superpixel is hard to determine since small superpixels lack enough spatial information and large superpixels usually result in error segmentation. Therefore, a multiscale superpixels based sparse representation (MSSR) algorithm is proposed to utilize the spatial-spectral information of multiscale superpixels for the HSI classification. Specifically, multiscale superpixels of a HSI are generated firstly. Then, the joint sparse representation classification (JSRC) is used to obtain the class labels of superpixels of different scales. Finally, the majority voting is applied on the labels of different scales to create the final class label for each pixel. Experimental results show that the proposed MSSR algorithm outperforms several well-known classification algorithms.

A novel hyperspectral image (HSI) classification method by the probabilistic fusion of pixel-level and superpixel-level classifiers is proposed. Generally, pixel-level classifiers based on spectral information only may generate “salt and pepper” result in the classification map since spatial correlation is not considered. By incorporating spatial information in homogeneous regions, the superpixel-level classifiers can effectively eliminate the noisy appearance. However, the classification accuracy will be deteriorated if undersegmentation cannot be fully avoided in superpixel-based approaches. Therefore, it is proposed to adaptively combine both the pixel-level and superpixel-level classifiers, to improve the classification performance in both homogenous and structural areas. In the proposed method, a support vector machine classifier is first applied to estimate the pixel-level class probabilities. Then, superpixel-level class probabilities are estimated based on a joint sparse representation. Finally, the two levels of class probabilities are adaptively combined in a maximum a posteriori estimation model, and the classification map is obtained by solving the maximum optimization problem. Experimental results on real HSI images demonstrate the superiority of the proposed method over several well-known classification approaches in terms of classification accuracy.

A novel set-to-set distance-based spectral-spatial classification method for hyperspectral images (HSIs) is proposed. In HSIs, the spatially connected and spectrally similar pixels within each homogeneous region can be considered as one set of test samples, i.e., a test set, which should belong to the same class. In addition, each class of labeled pixels can be regarded as one set of training samples, i.e., a training set. Therefore, it is a natural consideration in the proposed method to measure the similarity between test and training sets via specific set-based distance criteria and then decide the classification label for each test set, accordingly. Specifically, the superpixel algorithm-based oversegmentation technique jointly exploits both the spatial similarity and structural information to first divide the HSI into multiple perceptually uniform regions. As a result, each segmented region corresponds to one test set. Then, each test/training set is represented with an affine hull (AH) model, which utilizes both the similarity and variance of pixels within each set to adaptively characterize the set. Finally, the class label for each test set is determined based on the closest geometry distance between test and training AHs. Experimental results on real HSI data sets demonstrate the superiority of the proposed algorithm over several well-known classification approaches, in terms of classification accuracy and computational speed.

In this paper, a novel spectral–spatial adaptive sparse representation (SSASR) method is proposed for hyperspectral image (HSI) denoising. The proposed SSASR method aims at improving noise-free estimation for noisy HSI by making full use of highly correlated spectral information and highly similar spatial information via sparse representation, which consists of the following three steps. First, according to spectral correlation across bands, the HSI is partitioned into several nonoverlapping band subsets. Each band subset contains multiple continuous bands with highly similar spectral characteristics. Then, within each band subset, shape-adaptive local regions consisting of spatially similar pixels are searched in spatial domain. This way, spectral–spatial similar pixels can be grouped. Finally, the highly correlated and similar spectral–spatial information in each group is effectively used via the joint sparse coding, in order to generate better noise-free estimation. The proposed SSASR method is evaluated by different objective metrics in both real and simulated experiments. The numerical and visual comparison results demonstrate the effectiveness and superiority of the proposed method.

Scanned images of historical documents often suffer from bleed-through, which refers to the ink on one side seeping through the paper and appearing on the other side. In this paper, a new conditional random field (CRF)-based method is proposed to remove the bleed-through from the scanned images of historical images. The proposed method only requires the scanned image of one side, referred as a blind method. In general, the scanned historical document image is composed of three components: foreground, bleed-through, and background. By assuming Gaussian distributions of the three components, the proposed method establishes conditional probability distribution (CPD) models of the three components first. The parameters of the component CPD models are estimated based on an initial segmentation of the input image. Then, CRFs are used to capture the relations between observed pixels in the scanned image and the corresponding labels as well as the spatial relation between the adjacent labels. The belief propagation algorithm is used to calculate the probabilities of different labels for each pixel. Once the labeling is completed by choosing the most possible label for each pixel, the bleed-through component is removed from the input historical image by a random-filling inpainting algorithm. Experimental results on the real data set show that the proposed method preserves the foreground component very well and removes the bleed-through effectively.

Due to the quality of paper and long-time preservation, the ink on one side of the historical documents often seeps through and appears on the other side. In this paper, a new blind ink bleed-through removal method is proposed to deal with the scanned historical document images. The scanned historical document image generally consists of three components: foreground, bleed-through and background. In the proposed method, conditional probability distribution (CPD) models of the three components are firstly established by statistics. Then, conditional random fields (CRFs) are used to model the observed scanned image and the corresponding labels. For each input scanned image, parameters of the component-wise CPD models are estimated and belief propagation is performed on the CRFs model to determine the most possible labels. Once the bleed-through component is found, an inpainting algorithm is proposed to remove the ink bleed-through from the input historical image. Experimental results show that the proposed method preserves the foreground component very well and removes the bleed-through effectively.

We propose a superpixel-based composite kernel framework for hyperspectral image (HSI) classification. Composite kernel methods can utilize both the spectral and the spatial information for the HSI classification. However, setting the optimal spatial neighborhood for different spatial structures is a non-trivial issue. In order to adaptively exploit the spatial contextual information, we utilize superpixel to obtain spatial information. A superpixel can be regarded as a local neighborhood, whose size and shape can be adaptively adjusted according to the spatial structures in the HSI. Then, the spatial features are extracted by computing the mean of the spectral pixels within each superpixel. Finally, composite kernel with support vector machine is implemented on real HSI. Experiments on two real HSIs demonstrate the outstanding performance of the proposed method.

Recently, the superpixel segmentation is introduced into the hyperspectral image (HSI) classification to exploit the spatial information. However, the size of superpixels influences the classification significantly because small superpixels can not provide enough spatial information and large superpixels generally result in error segmentation. The error segmentation is irreversible and intolerable, so the size of superpixels tends to be small. This paper proposes a hyperspectral unmixing based superpixel merging criterion to merge small su-perpixels and thus make use of the spatial information. The spatial information is then incorporated into the joint sparsity model for the spectral-spatial classification. Experimental results demonstrate the superiority of the proposed method over some widely used classification methods.

The 2014 data fusion contest organized by IGARSS 2014 has shown an example that dual images with quite different spatial resolutions may have quite different ground coverages. Specifically, due to the technology limit of optical sensor and the incomplete flight path, the high spatial resolution visible image used in the data fusion contest can only provide a sparse ground coverage. To fill the missing area in this visible image, a simple yet effective image completion method is introduced in this paper which consists of the following two steps: First, through performing patch matching on the hyperspectral image, the most common recurring offsets between patches in the known region and those in the missing region are estimated. Second, the pixels in the missing region is completed by fusing pixels of the shifted visible images (obtained using the above estimated offsets). Experimental results show that the proposed method performs much better than multiple general image completion methods for this data set.

Hyperspectral Unmixing aims at getting the endmember signature and their corresponding abundance maps from highly mixed Hyperspectral image. Nonnegative Matrix Factorization (NMF) is a widely used method for spectral unmixing because it can obtain better performance while there is no pure pixels in the hyperspectral image. However, many methods based on nonnegative matrix factorization seldom consider the spatial information both on local and nonlocal. To combine the spatial and spectral information together to improve the unmixing accuracy, an adaptive endmember extraction based sparse nonnegative matrix factorization with spatial local information (ASNMF) is proposed in this paper. A superpixel segmentation is to obtain many meaningful regions which are spectral similar and spatial adjacent. Endmember is adaptively extracted on each superpixel to generate endmember set. Initialing the endmember set, ASNMF could adaptively obtain the final endmembers with the sparse nonnegative matrix factorization. Both the experiments on synthetic and real scene images show the effectiveness of the proposed method for hyperspectral unmixing.

In this paper, a gradient-guided sparse representation method (GGSR) for the hyperspectral image denoising is proposed. In the context of the hyperspectral image, neighbourhood spectral bands always have highly similar spatial and structural characteristics, which can be jointly used to improve the image quality. On the one hand, the sparse representation, as one powerful image processing tool, is introduced to jointly sparsely code similar image patches from different spectral bands. By this way, the redundant spatial similarity can be effectively exploited. On the other hand, the reference gradient is incorporated with the sparse representation model, in order to exploit the redundant structural information to better preserve the structure/texture characteristics. Practically, the gradient reference can be estimated from the neighbouring structural similar spectral bands. Experimental results demonstrate the effectiveness of the proposed method in removing noise as well as preserving structures.

For the classification of hyperspectral images (HSIs), this paper presents a novel framework to effectively utilize the spectral-spatial information of superpixels via multiple kernels, which is termed as superpixel-based classification via multiple kernels (SC-MK). In the HSI, each superpixel can be regarded as a shape-adaptive region, which consists of a number of spatial neighboring pixels with very similar spectral characteristics. First, the proposed SC-MK method adopts an oversegmentation algorithm to cluster the HSI into many superpixels. Then, three kernels are separately employed for the utilization of the spectral information, as well as spatial information, within and among superpixels. Finally, the three kernels are combined together and incorporated into a support vector machine classifier. Experimental results on three widely used real HSIs indicate that the proposed SC-MK approach outperforms several well-known classification methods.

A novel superpixel-based discriminative sparse model (SBDSM) for spectral-spatial classification of hyperspectral images (HSIs) is proposed. Here, a superpixel in a HSI is considered as a small spatial region whose size and shape can be adaptively adjusted for different spatial structures. In the proposed approach, the SBDSM first clusters the HSI into many superpixels using an efficient oversegmentation method. Then, pixels within each superpixel are jointly represented by a set of common atoms from a dictionary via a joint sparse regularization. The recovered sparse coefficients are utilized to determine the class label of the superpixel. In addition, instead of directly using a large number of sampled pixels as dictionary atoms, the SBDSM applies a discriminative K-SVD learning algorithm to simultaneously train a compact representation dictionary, as well as a discriminative classifier. Furthermore, by utilizing the class label information of training pixels and dictionary atoms, a class-labeled orthogonal matching pursuit is proposed to accelerate the K-SVD algorithm while still enforcing high discriminability on sparse coefficients when training the classifier. Experimental results on four real HSI datasets demonstrate the superiority of the proposed SBDSM algorithm over several well-known classification approaches in terms of both classification accuracies and computational speed.

In this paper, a novel feature extraction method based on intrinsic image decomposition (IID) is proposed for hyperspectral image classification. The proposed method consists of the following steps. First, the spectral dimension of the hyperspectral image is reduced with averaging-based image fusion. Then, the dimension reduced image is partitioned into several subsets of adjacent bands. Next, the reflectance and shading components of each subset are estimated with an optimization-based IID technique. Finally, pixel-wise classification is performed only on the reflectance components, which reflect the material-dependent properties of different objects. Experimental results show that, with the proposed feature extraction method, the support vector machine classifier is able to obtain much higher classification accuracy even when the number of training samples is quite small. This demonstrates that IID is indeed an effective way for feature extraction of hyperspectral images.

This paper introduces a novel spectral-spatial classification method for hyperspectral images based on extended random walkers (ERWs), which consists of two main steps. First, a widely used pixelwise classifier, i.e., the support vector machine (SVM), is adopted to obtain classification probability maps for a hyperspectral image, which reflect the probabilities that each hyperspectral pixel belongs to different classes. Then, the obtained pixelwise probability maps are optimized with the ERW algorithm that encodes the spatial information of the hyperspectral image in a weighted graph. Specifically, the class of a test pixel is determined based on three factors, i.e., the pixelwise statistics information learned by a SVM classifier, the spatial correlation among adjacent pixels modeled by the weights of graph edges, and the connectedness between the training and test samples modeled by random walkers. Since the three factors are all well considered in the ERW-based global optimization framework, the proposed method shows very good classification performances for three widely used real hyperspectral data sets even when the number of training samples is relatively small.

Pansharpening aims to synthesize a high-spatial-resolution multispectral (MS) image by fusing a panchromatic (PAN) image and a low-resolution MS image. The multiresolution analysis (MRA)-based methods are a popular group of pansharpening methods. However, in the MRA-based methods, spatial distortions may occur in the pansharpened product due to the misalignment of PAN and MS data. To address the spatial distortion issue in MRA-based methods, this paper proposes a two-step approach, which consists of the coarse step and the refined step. The coarse step produces a preliminary result using the traditional details injection model. Then, the preliminary product is refined with a second details injection operation in the refined step. Moreover, in our proposed two-step approach, a novel multiscale decomposition based on a normalized nonlocal means (NNLM) filter is developed to extract the spatial detail. Compared with the original nonlocal means filter, the designed NNLM makes the similarity measure more robust and accurate by exploiting the normalized intensity value and the mean value jointly. The experimental results on various satellite data demonstrate the superiority of the proposed pansharpening scheme by comparing with ten well-known methods.

Feature extraction is of high importance for effective data classification in hyperspectral imaging (HSI). Considering the high correlation among band images, spectral-domain feature extraction is widely employed. For effective spatial information extraction, a 2-D extension to singular spectrum analysis (2D-SSA), which is a recent technique for generic data mining and temporal signal analysis, is proposed. With 2D-SSA applied to HSI, each band image is decomposed into varying trends, oscillations, and noise. Using the trend and the selected oscillations as features, the reconstructed signal, with noise highly suppressed, becomes more robust and effective for data classification. Three publicly available data sets for HSI remote sensing data classification are used in our experiments. Comprehensive results using a support vector machine classifier have quantitatively evaluated the efficacy of the proposed approach. Benchmarked with several state-of-the-art methods including 2-D empirical mode decomposition (2D-EMD), it is found that our proposed 2D-SSA approach generates the best results in most cases. Unlike 2D-EMD that requires sequential transforms to obtain detailed decomposition, 2D-SSA extracts all components simultaneously. As a result, the execution time in feature extraction can be also dramatically reduced. The superiority in terms of enhanced discrimination ability from 2D-SSA is further validated when a relatively weak classifier, i.e., the k-nearest neighbor, is used for data classification. In addition, the combination of 2D-SSA with 1-D principal component analysis (2D-SSA-PCA) has generated the best results among several other approaches, demonstrating the great potential in combining 2D-SSA with other approaches for effective spatial-spectral feature extraction and dimension reduction in HSI.

We present a novel general-purpose compression method for tomographic images, termed 3D adaptive sparse representation based compression (3D-ASRC). In this paper, we focus on applications of 3D-ASRC for the compression of ophthalmic 3D optical coherence tomography (OCT) images. The 3D-ASRC algorithm exploits correlations among adjacent OCT images to improve compression performance, yet is sensitive to preserving their differences. Due to the inherent denoising mechanism of the sparsity based 3D-ASRC, the quality of the compressed images are often better than the raw images they are based on. Experiments on clinical-grade retinal OCT images demonstrate the superiority of the proposed 3D-ASRC over other well-known compression methods.

From a single low resolution image, a real-time document image super-resolution algorithm is proposed to obtain high resolution document image with sharp text boundaries. First, a highly efficient document image matting algorithm based on local linear modeling is designed to decompose the input image into text, foreground and background layers, which contain the text edge information, the color information of the foreground and background respectively. Then the text layer is up-sampled with Teager filter to increase the sharpness of the text. For efficiency, the foreground and background layers are simply up-sampled through the bi-cubic interpolation. Finally, these three high resolution layers are composed to obtain the high-resolution image. Experiments on real scanned document images demonstrate the effectiveness of the proposed method in both visual perception and OCR performance

This paper proposes a new spectral-spatial hyperspectral classification method named the shape-adaptive sparse representation (SASR). The fixed window is not suitable for all pixels of hyperspectral image (HSI) to search local similar regions. In order to overcome the drawback, we propose to apply the shape-adaptive algorithm to exploit the contextual spatial information of HSI. Furthermore, the hyperspectral classification is implemented by incorporating the spatial contextual information of HSI into the sparse representation classification model. Experimental results demonstrate the superiority of the proposed SASR method over both classical and state-of-the-art approaches.

A novel spectral-spatial hyperspectral image classification is proposed based on extended random walkers. First, a widely used pixel-wise classifier, i.e., the support vector machine (SVM), is adopted to obtain probability maps for a hyper-psectral image, which measure the probabilities that a pixel belongs to different classes. Then, the initial probabilities are optimized with the extended random walkers. Finally, by assigning each pixel with the label for which the greatest probability is obtained, the classification result is obtained. Experiments show the outstanding performance of the proposed method in terms of classification accuracy especially when the number of training samples is relatively small.

To select informative features on extremely high-dimensional problems, in this paper, a sampling scheme is proposed to enhance the efficiency of recently developed feature generating machines (FGMs). Note that in FGMs O(mlogr) time complexity should be taken to order the features by their scores; the entire computational cost of feature ordering will become unbearable when m is very large, for example, m > 10(11) , where m is the feature dimensionality and r is the size of the selected feature subset. To solve this problem, in this paper, we propose a feature generating sampling method, which can reduce this computational complexity to O(Gslog(G)+G(G+log(G))) while preserving the most informative features in a feature buffer, where Gs is the maximum number of nonzero features for each instance and G is the buffer size. Moreover, we show that our proposed sampling scheme can be deemed as the birth-death process based on random processes theory, which guarantees to include most of the informative features for feature selections. Empirical studies on real-world datasets show the effectiveness of the proposed sampling method.

Sparse representation has been demonstrated to be a powerful tool in classification of hyperspectral images (HSIs). The spatial context of an HSI can be exploited by first defining a local region for each test pixel and then jointly representing pixels within each region by a set of common training atoms (samples). However, the selection of the optimal region scale (size) for different HSIs with different types of structures is a nontrivial task. In this paper, considering that regions of different scales incorporate the complementary yet correlated information for classification, a multiscale adaptive sparse representation (MASR) model is proposed. The MASR effectively exploits spatial information at multiple scales via an adaptive sparse strategy. The adaptive sparse strategy not only restricts pixels from different scales to be represented by training atoms from a particular class but also allows the selected atoms for these pixels to be varied, thus providing an improved representation. Experiments on several real HSI data sets demonstrate the qualitative and quantitative superiority of the proposed MASR algorithm when compared to several well-known classifiers.

The integration of spatial context in the classification of hyperspectral images is known to be an effective way in improving classification accuracy. In this paper, a novel spectral-spatial classification framework based on edge-preserving filtering is proposed. The proposed framework consists of the following three steps. First, the hyperspectral image is classified using a pixelwise classifier, e.g., the support vector machine classifier. Then, the resulting classification map is represented as multiple probability maps, and edge-preserving filtering is conducted on each probability map, with the first principal component or the first three principal components of the hyperspectral image serving as the gray or color guidance image. Finally, according to the filtered probability maps, the class of each pixel is selected based on the maximum probability. Experimental results demonstrate that the proposed edge-preserving filtering based classification method can improve the classification accuracy significantly in a very short time. Thus, it can be easily applied in real applications.

Feature extraction is known to be an effective way in both reducing computational complexity and increasing accuracy of hyperspectral image classification. In this paper, a simple yet quite powerful feature extraction method based on image fusion and recursive filtering (IFRF) is proposed. First, the hyperspectral image is partitioned into multiple subsets of adjacent hyperspectral bands. Then, the bands in each subset are fused together by averaging, which is one of the simplest image fusion methods. Finally, the fused bands are processed with transform domain recursive filtering to get the resulting features for classification. Experiments are performed on different hyperspectral images, with the support vector machines (SVMs) serving as the classifier. By using the proposed method, the accuracy of the SVM classifier can be improved significantly. Furthermore, compared with other hyperspectral classification methods, the proposed IFRF method shows outstanding performance in terms of classification accuracy and computational efficiency.

Pansharpening aims at creating a fused image of high spatial and spectral resolutions through merging a panchromatic (PAN) image with a multispectral (MS) image. Component substitution is the most widely used pansharpening method. However, most research in this field focuses on improving the existing component substitution-based pansharpening methods, e.g., principal component substitution and intensity hue saturation transform. The major contribution of this paper is a novel component substitution framework based on an image matting model. The matting model refers to an MS image that can be decomposed into three components, i.e., alpha channel, spectral foreground, and background. Through substituting the alpha channel of the MS image with the PAN image, the high-resolution MS image is able to be reconstructed perfectly. Experiments performed on different data sets demonstrate that the proposed method outperforms several state-of-the-art pansharpening methods in terms of subjective and objective evaluation.

Currently, most electrophotographic printers use halftoning technique to print continuous tone images, so scanned images obtained from such hard copies are usually corrupted by screen like artifacts. In this paper, a new model of scanned halftone image is proposed to consider both printing distortions and halftone patterns. Based on this model, an adaptive filtering based descreening method is proposed to recover high quality contone images from the scanned images. Image redundancy based denoising algorithm is first adopted to reduce printing noise and attenuate distortions. Then, screen frequency of the scanned image and local gradient features are used for adaptive filtering. Basic contone estimate is obtained by filtering the denoised scanned image with an anisotropic Gaussian kernel, whose parameters are automatically adjusted with the screen frequency and local gradient information. Finally, an edge-preserving filter is used to further enhance the sharpness of edges to recover a high quality contone image. Experiments on real scanned images demonstrate that the proposed method can recover high quality contone images from the scanned images. Compared with the state-of-the-art methods, the proposed method produces very sharp edges and much cleaner smooth regions.

A novel matting model based pansharpening (MMP) method is proposed for creating a fused image of high spatial and spectral resolutions through merging a panchromatic (PAN) image with a multi-spectral (MS) image. Matting model refers to that an MS image can be decomposed into three components, i.e., alpha channel, spectral foreground and background. Through substituting the alpha channel of the MS image with the PAN image, the edge information from the PAN image can be transferred into the MS image without causing spectral distortion. Experiments demonstrate the superiority of the proposed method by subjective and objective evaluation.

Remote sensing image fusion can integrate the spatial detail of panchromatic (PAN) image and the spectral information of a low-resolution multispectral (MS) image to produce a fused MS image with high spatial resolution. In this paper, a remote sensing image fusion method is proposed with sparse representations over learned dictionaries. The dictionaries for PAN image and low-resolution MS image are learned from the source images adaptively. Furthermore, a novel strategy is designed to construct the dictionary for unknown high-resolution MS images without training set, which can make our proposed method more practical. The sparse coefficients of the PAN image and low-resolution MS image are sought by the orthogonal matching pursuit algorithm. Then, the fused high-resolution MS image is calculated by combining the obtained sparse coefficients and the dictionary for the high-resolution MS image. By comparing with six well-known methods in terms of several universal quality evaluation indexes with or without references, the simulated and real experimental results on QuickBird and IKONOS images demonstrate the superiority of our method.

A fast and effective image fusion method is proposed for creating a highly informative fused image through merging multiple images. The proposed method is based on a two-scale decomposition of an image into a base layer containing large scale variations in intensity, and a detail layer capturing small scale details. A novel guided filtering-based weighted average technique is proposed to make full use of spatial consistency for fusion of the base and detail layers. Experimental results demonstrate that the proposed method can obtain state-of-the-art performance for fusion of multispectral, multifocus, multimodal, and multiexposure images.

In this paper, we present a novel technique, based on compressive sensing principles, for reconstruction and enhancement of multi-dimensional image data. Our method is a major improvement and generalization of the multi-scale sparsity based tomographic denoising (MSBTD) algorithm we recently introduced for reducing speckle noise. Our new technique exhibits several advantages over MSBTD, including its capability to simultaneously reduce noise and interpolate missing data. Unlike MSBTD, our new method does not require an a priori high-quality image from the target imaging subject and thus offers the potential to shorten clinical imaging sessions. This novel image restoration method, which we termed sparsity based simultaneous denoising and interpolation (SBSDI), utilizes sparse representation dictionaries constructed from previously collected datasets. We tested the SBSDI algorithm on retinal spectral domain optical coherence tomography images captured in the clinic. Experiments showed that the SBSDI algorithm qualitatively and quantitatively outperforms other state-of-the-art methods.

In this letter, we propose an unsupervised approach for change detection in multitemporal satellite images based on a novel detail-enhancing algorithm. The multitemporal source images are first used to generate the difference image, which is decomposed into low-pass approximation and high-pass directional subbands by the nonsubsampled contourlet transform. The coefficients from the directional subbands are fused at intrascale and interscale to extract the meaningful details of the difference image. After that, the extracted details are injected into one base image selected from the approximation subbands, which results in a detail-enhanced difference image. For each pixel in the enhanced difference image, a dimension-reduced feature vector is created using the principal component analysis (PCA). The final change detection map is achieved by clustering the feature vectors using a PCA-guidedk-means algorithm into “changed” and “unchanged” classes. Experimental results demonstrate the superior performance of the proposed approach compared with several well-known change detection techniques.

In this paper we introduce the nonsubsampled shear let transform for multi-focus image fusion. In the proposed method, source images are decomposed by nonsubsampled shear let transform firstly. Then the decomposition coefficients are merged according to the given fusion rule. Finally the fused image is reconstructed by inverse nonsubsampled shear let transform. The experimental results over five pairs of registered multi-focus images and one pair of mis-registered multi-focus images demonstrate the superiority of the proposed method.

Vascular diseases are major public heath problem around the world. Vessel segmentation has been widely concerned because it is a key step for diagnosis and surgical planning. Among past strategies, multi-scale line filters are very popular detectors. However, multi-scale integration results in undesirable diffusion when two vessels are closely located. To avoid this problem, we use gradient vector flow as vector field and introduce a vesselness measure to detect vessel which gives high and homogeneous output for line structure so that it is more suitable for segmentation over Frangi's vesselness measure. Level set method is applied to perform vessel segmentation. Our model is tested on real images. Experimental results demonstrate that our approach can successfully separate closely adjacent vessels and address the problems of low contrast and varying vessel width. It shows better performance than multi-scale approach. Furthermore, gradient vector flow makes the contour moving into boundary concavities.

In near infrared face recognition systems, situations including head rotation, motion blur, darkness, eyes closed, mouth opened and the small face region will deteriorate the recognition accuracy. Thus, it is necessary to design a quality assessment system to select the best frame from the input video sequence before face recognition or saving it to database. In this paper we present a scoring evaluation system based on five features including sharpness, brightness, resolution, head pose and expression. Firstly, the score of each feature is computed independently, and then the final quality score is obtained by combining the scores of five features with weights. Center for Biometrics and Security Research (CBSR) Near Infrared Face Dataset is used to test the system. The experiment results demonstrate the effectiveness of the proposed quality assessment.

Person re-identification is an important problem in computer vision, which involves matching appearance of individuals between non-overlapping camera views. In this paper we present a novel appearance-based method for person re-identification problem. Color feature, Gabor, local binary pattern (LBP) are utilized to form a covariance descriptor to handle the difficulties such as varying illumination, viewpoint angle and non-rigid body, then distances of these features are computed to match these individuals. Experimental results over the challenging dataset VIPeR demonstrate that our method obtains competitive performance.

In this paper, we propose a novel feature vector clustering method for unsupervised change detection in multitemporal satellite images. A feature vector for each pixel is extracted using the compressed sparse representation of the difference image which is obtained by comparing a pair of co-registered images acquired at different times on the same area. The compressed sparse representation is achieved by taking two stages: compressed sampling and sparse representation. The compressed sampling is first employed in order to reduce the dimensionality of the feature vectors. Then, the sparse representation is applied to extract the meaningful change information and to combat the noise interference. The final change detection is obtained by clustering the extracted feature vectors using k-means algorithm into “changed” and “unchanged” classes. Experimental results clearly show that the proposed approach consistently yields superior performance compared to several well-known change detection techniques on both noise-free and noisy satellite images.

This paper presents a novel method based on the developed multiscale dual bilateral filter to fuse high spatial resolution panchromatic image and high spectral resolution multispectral image. Compared with traditional multi-resolution based methods, the process of detail extraction considers the characteristics of panchromatic image and multispectral image simultaneously. The low resolution multispectral image is resampled to the same size of the high resolution panchromatic image and sharpened through injecting the extracted details. The proposed fusion method is tested over QuickBird and IKONOS images and compared with three popular methods. The experimental results demonstrate that our method outperforms conventional methods.

Image super resolution is a challenging highly ill-posed inverse problem. In this paper, we proposed a texture constrained sparse representation for single image super resolution. Firstly, the low resolution observed image is segmented into different texture regions. Through preprepared texture databases, the low resolution regions are classified into different texture categories using the designed texture classifier. Then, the high resolution segments are reconstructed by sparse representation with relevant texture dictionaries. Integrating all segments, the high resolution result is obtained. The proposed method is compared with sparse representation method and some existing methods. The experimental results show that our method achieves better results in visual inspection and quantitative analysis.

This paper addresses the remote sensing image pan-sharpening problem from the perspective of compressed sensing (CS) theory which ensures that with the sparsity regularization, a compressible signal can be correctly recovered from the global linear sampled data. First, the degradation model from a high- to low-resolution multispectral (MS) image and high-resolution panchromatic (PAN) image is constructed as a linear sampling process which is formulated as a matrix. Then, the model matrix is considered as the measurement matrix in CS, so pan-sharpening is converted into signal restoration problem with sparsity regularization. Finally, the basis pursuit (BP) algorithm is used to resolve the restoration problem, which can recover the high-resolution MS image effectively. The QuickBird and IKONOS satellite images are used to test the proposed method. The experimental results show that the proposed method can well preserve spectral and spatial details of the source images. The pan-sharpened high-resolution MS image by the proposed method is competitive or even superior to those images fused by other well-known methods.

Particle swarm optimizer (PSO) is a powerful optimization algorithm that has been applied to a variety of problems. It can, however, suffer from premature convergence and slow convergence rate. Motivated by these two problems, a hybrid global optimization strategy combining PSOs with a modified Broyden-Fletcher-Goldfarb-Shanno (BFGS) method is presented in this paper. The modified BFGS method is integrated into the context of the PSOs to improve the particles' local search ability. In addition, in conjunction with the territory technique, a reposition technique to maintain the diversity of particles is proposed to improve the global search ability of PSOs. One advantage of the hybrid strategy is that it can effectively find multiple local solutions or global solutions to the multimodal functions in a box-constrained space. Based on these local solutions, a reconstruction technique can be adopted to further estimate better solutions. The proposed method is compared with several recently developed optimization algorithms on a set of 20 standard benchmark problems. Experimental results demonstrate that the proposed approach can obtain high-quality solutions on multimodal function optimization problems.

This paper proposes a new learned dictionary for sparse representation of given data and suggests a way to apply it to non-local denoising. First, a quad-tree structure is efficiently embedded into a sparse dictionary model. This enables the dictionary to discover the complex structures in the given data and to be easily employed to high dimensional data. Besides, we propose a joint 3-D operation to exploit the correlations among the similar blocks, as the non-local denoising model assumes that there exist mutually similar blocks in nature images. This 3-D operation is achieved by a simple concatenation of the similar patches to a single vector and training the proposed dictionary on it. The experimental results indicate that our approach is competitive with several well known denoising techniques in terms of both PSNR and visual quality.

This paper presents a novel adaptive synthetic (ADASYN) sampling approach for learning from imbalanced data sets. The essential idea of ADASYN is to use a weighted distribution for different minority class examples according to their level of difficulty in learning, where more synthetic data is generated for minority class examples that are harder to learn compared to those minority examples that are easier to learn. As a result, the ADASYN approach improves learning with respect to the data distributions in two ways: (1) reducing the bias introduced by the class imbalance, and (2) adaptively shifting the classification decision boundary toward the difficult examples. Simulation analyses on several machine learning data sets show the effectiveness of this method across five evaluation metrics.

Microarray data typically have thousands of genes, and thus feature extraction is a critical problem for accurate cancer classification. In this paper, a feature extraction method based on the discrete wavelet transform (DWT) is proposed. The approximation coefficients of DWT, together with some useful features from the high-frequency coefficients selected by the maximum modulus method, are used as features. The combined coefficients are then forwarded to a SVM classifier. Experiments are performed on two standard benchmark data sets: ALL/AML Leukemia and Colon tumor. Experimental results show that the proposed method can achieve state-of-the-art performance on cancer classification.

In this paper, we show that one-class SVMs can also utilize data covariance in a robust manner to improve performance. Furthermore, by constraining the desired kernel function as a convex combination of base kernels, we show that the weighting coefficients can be learned via quadratically constrained quadratic programming (QCQP) or second order cone programming (SOCP) methods. Performance on both toy and real-world data sets show promising results. This paper thus offers another demonstration of the synergy between convex optimization and kernel methods.

Many vision-related processing tasks, such as edge detection, image segmentation and stereo matching, can be performed more easily when all objects in the scene are in good focus. However, in practice, this may not be always feasible as optical lenses, especially those with long focal lengths, only have a limited depth of field. One common approach to recover an everywhere-in-focus image is to use wavelet-based image fusion. First, several source images with different focuses of the same scene are taken and processed with the discrete wavelet transform (DWT). Among these wavelet decompositions, the wavelet coefficient with the largest magnitude is selected at each pixel location. Finally, the fused image can be recovered by performing the inverse DWT. In this paper, we improve this fusion procedure by applying the discrete wavelet frame transform (DWFT) and the support vector machines (SVM). Unlike DWT, DWFT yields a translation-invariant signal representation. Using features extracted from the DWFT coefficients, a SVM is trained to select the source image that has the best focus at each pixel location, and the corresponding DWFT coefficients are then incorporated into the composite wavelet representation. Experimental results show that the proposed method outperforms the traditional approach both visually and quantitatively.