Semantic understanding of large-scale aerial scenes represents a critical challenge in 3D computer vision, hindered by the prohibitive cost of dense annotation. This paper introduces EvoPropGS, a novel approach for the semantic segmentation of 3D Gaussian Splatting models that requires only minimal supervision. Our core insight is to leverage the inherent structural repetitions within aerial environments to propagate semantic information from a sparse set of annotations across the entire 3D scene. Our approach constructs a prompt library by pairing SAM-generated mask candidates with DINOv2 feature embeddings from annotated views. For unannotated regions, we generate pseudo-labels by matching region proposals with these featured prompts via cosine similarity. We then formulate optimal prompt selection as a discrete optimization problem solved via evolutionary search, guided by our novel fitness function that evaluates both 3D consistency and 2D semantic coherence. Extensive experiments demonstrate that EvoPropGS achieves accurate segmentation with only 2 percent annotated pixels.

The iterative search process of evolutionary algorithms (EAs) encapsulates optimization knowledge within historical populations and fitness evaluations. Effective utilization of this knowledge is crucial for facilitating knowledge transfer and online adaptation. However, current research typically addresses these goals in isolation and faces distinct limitations: evolutionary sequential transfer optimization often suffers from incomplete utilization of prior knowledge, while adaptive strategies, utilizing real-time knowledge, are limited to tailoring specific evolutionary operators. To simultaneously achieve these two capabilities, we introduce the Optimization Knowledge Adaptation Evolutionary Model (OKAEM), a unified learnable evolutionary framework capable of adaptively updating parameters based on available optimization knowledge. By parameterizing evolutionary operators via attention mechanisms, OKAEM enables learnable update rules that facilitate the utilization of optimization knowledge via two phases: pre-training to integrate extensive prior knowledge for efficient transfer, and adaptive optimization to dynamically update parameters based on real-time knowledge. Experimental results confirm that OKAEM significantly outperforms state-of-the-art sequential transfer methods across 12 transfer scenarios via pre-training, and surpasses advanced learnable EAs solely through its self-tuning mechanism in prior-free settings. Beyond demonstrating practical utility in prompt tuning for vision-language models, ablation studies validate the necessity of the learnable components, while visualization analyses reveal the model's capacity to autonomously discover interpretable evolutionary principles.

Matrix factorization is a fundamental characterization model in machine learning and is usually solved using mathematical decomposition reconstruction loss. However, matrix factorization is a data-driven model whose results depend on data quality, making it susceptible to noise. Inspired by physics, the law of conservation of energy is used to introduce physical laws into matrix factorization, which is called Physics-informed Matrix Factorization operator (PiMF). The PiMF operator uses the heat conduction equation to construct the energy objective function for matrix factorization, thereby retaining the mathematical model's decomposition meaning and satisfying the interpretability of physics. The PiMF follows the physical laws, thereby suppressing irregular or sudden noise signals that violate these physical principles. The solutions of the PiMF operator include more comprehensive knowledge of mathematics and physics, which improves the ability to generalize complex data, especially for noisy data. We demonstrate the consistency of the energy objective function and the mathematical model, which verifies the feasibility of matrix factorization using physical energy laws. In addition, the physical interpretability of the PiMF operator is proved from the perspective of energy decline. This study proposes two practical algorithms for PiMF in classification and clustering tasks, enhancing the practicability of matrix factorization by incorporating task-specific prior information constraints. The experimental results of PiMF for classification and clustering demonstrate the advantages of the proposed operator. The importance of physics-informed matrix factorization is verified, especially for noisy data.

Domain generalization-based hyperspectral image classification methods have achieved promising results in recent years. However, these studies seldom consider the issue of small sample in the source domain. In practical applications, manually annotating hyperspectral images is difficult, so labeled samples in the source domain may be scarce. Existing models have limited feature extraction capability and poor generalization performance in scenarios with limited labeled samples. To address the limitations of existing methods on small sample data of the source domain, a novel approach, Progressive Multiscale Generator for Domain Generalization (PMGDG), is proposed in this paper. The PMGDG employs a progressive multiscale generator comprising a series of sub-generators with paired sub-discriminators. The channel dimension of generated samples grows gradually from the first layer to the last layer. Then, the Classifier network is trained on both the original samples and the generated samples with different distributions to enhance its generalization performance. Additionally, we introduce a hierarchical optimization approach to stabilize the training process. Extensive experiments are conducted on three public hyperspectral image cross-domain datasets:Houston, Pavia, and HyRANK. The experimental results demonstrate that, compared to existing domain generalization methods for hyperspectral image classification, the proposed approach significantly improves classification performance under small sample. The code is available from the website: https://github.com/adwfdawd/PMGDG.

Textural details are useful for image super-resolution, but massive CNN methods ignored the high-frequency components and generated over-smoothed outputs. The knowledge-based contourlet inference network is proposed in this paper. Different from other CNN-based methods that are directly infer high-resolution (HR) images, our model learns to reconstruct the HR image through the series of corresponding contourlet coefficients. Specifically, first, we consider the low-pass subbands of the contourlet as the corresponding low-resolution (LR) image. Then, feed it to the embedding net with residual blocks to provide adequate information for the contourlet coefficients prediction. Finally, we innovatively convert the estimation of contourlet coefficients into the estimation of the generalized gaussian distribution (GGD) parameters, and design the corresponding loss function to ensure training stability, which explores the smoothness of the contour effectively and guarantees the general structure and details of images. Experiments on four remote sensing datasets, four natural scenes and human-made content datasets, and the outdoor dataset demonstrate the superiority of the proposed model quantitatively and qualitatively.

With the Transformer architecture achieving impressive results in the vision domain. It has become a current popular research to explore more potentials of Transformer mixed architectures and explore more suitable mixed combinations. In this paper, Transformer classification network is designed and explored by multi-task architecture search algorithm. A new paradigm for multi-task architecture is designed by combining convolution and Transformer. The designed search architecture can combine the respective advantages of convolution and Transformer and can obtain better performance. At the same time, corresponding knowledge-aware multi-task genetic operators are designed to generate offspring individuals. Inter-task and inter-experience knowledge-aware is utilised to facilitate evolutionary convergence. During the search process, the reference evaluation method is utilised to reduce the redundant computation and time during the search process. In the experimental section, the search results are compared with state-of-the-art architectures and search algorithms. The experimental results confirm the effectiveness and high generalisation of the searched architectures. The ablation experimental part proves the effectiveness of the proposed architectural paradigm, genetic operators and reference evaluation.

The multi-scale geometric analysis is a great representation tool. It can be used to improve the feature representation and learning process of deep networks. In addition to extracting features, the multi-scale geometric prior knowledge can also be used for the structure improvement of deep networks. In this paper, we propose a multi-scale scattering representation learning network, abbreviated as MSRLN, for image classification tasks. The exploration of structure improvement can be made with multi-scale scattering operations. In this way, the better singularity representation learning process for networks can be achieved. Firstly, the filter banks and multi-scale scattering operator are introduced for non-linear and singularity representation. Secondly, the novel multi-scale scattering representation learning network structure is designed. The scaling- wise scattering process is deployed in the shallow layer as a non-linear layer. This structure essentially supplements deep networks with geometric prior knowledge. It can further improve the non-linear activation and singularity representation process. Thirdly, we put forward the multi-stage scattering representation strategy and the prior knowledge weakening mechanism. With flexible scaling factors and learning rates, the stepwise approximation and learning process of networks can be achieved. In sum, MSRLN is a kind of structural innovative, and the scattering singularity representation structure can be extended to other backbones or tasks. Extensive experimental results show that MSRLN can achieve better image classification accuracy. Finally, necessary convergence, insight, and adaptability analyses are provided in evaluation experiments.

Remote sensing (RS) images are prone to various degradations, which poses challenges to downstream tasks. Although existing single-task remote sensing image restoration methods are effective, they lack generalizability across tasks. All-in-one methods can handle multiple degradation tasks, but they usually focus on spatial information, ignoring the physical properties of the degradation information. To address the above limitations, we propose a Multiscale Spatial-Frequency Degradation Decoupling framework for All-in-One remote sensing image restoration (SFD$^{2}$IR), which decouples degradation features across different tasks to guide the model in performing task-specific image restoration. Specifically, a task-specific instruction generator (TIG) is proposed first to transform degradation features into task-specific prompts. Then, a multi-scale multi-frequency enhancement (MME) module is designed to decouple degradation effects from both spatial and frequency perspectives, thus enhancing the model's adaptability to various degradation types. Finally, a prompt feature refinement (PFR) module is developed to further refine the model's response to degraded tasks. Extensive experiments demonstrate that the proposed method achieves excellent performance on different RSIR tasks, including cloud removal, deblurring, dehazing, and super-resolution. The source code will be publicly available at SFD$^{2}$IR.

Existing diffusion-based methods have achieved impressive results in human motion editing. However, these methods often exhibit significant ghosting and body distortion in unseen in-the-wild cases. In this paper, we introduce Edit-Your-Motion, a video motion editing method that tackles these challenges through one-shot fine-tuning on unseen cases. Specifically, firstly, we utilized DDIM inversion to initialize the noise, preserving the appearance of the source video and designed a lightweight motion attention adapter module to enhance motion fidelity. DDIM inversion aims to obtain the implicit representations by estimating the prediction noise from the source video, which serves as a starting point for the sampling process, ensuring the appearance consistency between the source and edited videos. The Motion Attention Module (MA) enhances the model's motion editing ability by resolving the conflict between the skeleton features and the appearance features. Secondly, to effectively decouple motion and appearance of source video, we design a spatio-temporal two-stage learning strategy (STL). In the first stage, we focus on learning temporal features of human motion and propose recurrent causal attention (RCA) to ensure consistency between video frames. In the second stage, we shift focus on learning the appearance features of the source video. With Edit-Your-Motion, users can edit the motion of humans in the source video, creating more engaging and diverse content. Extensive qualitative and quantitative experiments, along with user preference studies, show that Edit-Your-Motion outperforms other methods.

In feature learning (FL), structural information shows advantages in retaining information and maintaining stability. Graph diffusion, a graph learning method that can focus on neighborhood structure and transmit information, has great research potential. In this study, a novel dynamic diffusion graph nonnegative matrix factorization (DGNMF) method is proposed, which uses a diffusion graph to improve the performance of FL and further enhances the effectiveness and stability of downstream classification tasks. DGNMF aims to mine and retain structural information more deeply in FL to build a more powerful and stable FL method. First, the model embeds graph learning into FL to obtain features containing structural information. Second, dynamic diffusion graph learning is used to mine deeper and more global structural information. Finally, we construct an updateable indicator matrix to enhance the discriminability of features. The classification experimental results of DGNMF on six databases demonstrate its advantages, verify its effectiveness and stability, and prove the importance of diffusion graph in improving FL.

Matrix factorization (MF) is a fundamental problem in machine learning, which is usually used as a feature learning method in various fields. For complex data involving spatiotemporal interactions, MF that only handles 2-D data will disrupt spatial dependence or temporal dynamics, failing to effectively couple spatial information with temporal factors. According to Markov chain principle, the spatial information of the present time is related to the spatial state of the previous time. We propose a spatial-temporal diffusion model for MF (STDMF), which uses graph diffusion to couple spatial-temporal information. Then, MF is used to learn the joint feature of data and spatial-temporal diffusion graph. Specifically, STDMF utilizes the graph diffusion with physical laws to generate spatial-temporal structure information. It obtains the underlying core structure of complex systems from a global perspective, which enhances the generalization ability of MF in noisy time-series data. To learn the lowest rank subspace of MF in time-series data, STDMF uses structural learning to constrain the rank of the learned features. Finally, STDMF is applied to clustering and anomaly detection of dynamic graph. The effectiveness of this method is verified by sufficient experiments, especially for noisy data.

Specific emitter identification (SEI) is a crucial task in various applications such as wireless communications and radar systems. The low-pass nature of vanilla Transformers hinders the extraction of high-frequency fingerprint features, resulting in poor SEI performance. Moreover, the introduction of additional high-frequency sensing structures can increase the computational efficiency of the already computationally intensive Transformer. To address these issues, we propose a high-frequency enhanced and low-complexity Transformer named HET. The framework integrates a multihead low-complexity self-attention (MLSA) module, a high-frequency enhanced connection, and a multihead high-frequency enhanced low-complexity self-attention (MESA) module. The MLSA module reduces the computational complexity by key and value mapping. The MESA and high-frequency enhanced connection module capture high-frequency information by reconstructing the low-frequency and high-frequency components of the features. We construct three HET variants, namely, $\text {HET}_{n}$ , $\text {HET}_{u}$ , and $\text {HET}_{m}$ , based on different enhancement methods and positions using $\text {MESA}_{n}$ , $\text {MESA}_{u}$ , and $\text {MESA}_{m}$ , respectively. Extensive experiments are conducted on the XSRP, ADS-B, and Wi-Fi datasets to evaluate the proposed models, demonstrating their competitive accuracy and faster throughput compared with popular methods. Theoretical proofs of high-frequency suppression and frequency response results confirm that the proposed framework has more gain for high-frequency information in SEI. Code is available at: https://github.com/zhailei-zl/HETmodel.

This paper studies grid-free point process modeling under varying fluid parameters. Existing methods rely on grid-based approaches or fixed parameters, making it challenging to handle complex nonlinear dynamics and out-of-distribution (OOD) scenarios. To address this, we propose Adaptive Perturbation Graph ODE (AGODE), a novel framework that integrates three key innovations: (1) an adaptive conditioning mechanism for physical parameter adaptation(2) a continuous graph neural ODE for spatiotemporal evolution modeling, and (3) a perturbation module with mutual information maximization for uncertainty quantification. AGODE employs graph neural networks to encode unstructured point cloud data into latent dynamics governed by neural ODEs, where physical parameters are injected through context-aware conditioning vectors. The perturbation module generates diverse trajectory samples by introducing stochastic noise during ODE integration, while contrastive learning aligns predictions with physical contexts to filter implausible outcomes.

Applying self-supervised deep learning improves the processing speed and accuracy of automatic modulation recognition (AMR). It reduces the dependence of previous deep networks on many labeled samples. However, affected by an incomplete signal representation modes set, previous models do not fully utilize the multiview property of signals in self-supervised learning. To deal with this issue, a hybrid-view contrastive model for AMR is proposed in this article based on self-supervised learning framework. First, star video is proposed to complete the set of signal representation modes. Next, a self-supervised learning framework based on hybrid-view contrastive learning, hybrid-view self-supervised framework (HVSF), is established to fully extract the signal features, where signals are augmented across views, including the discrete sequence, image, and video format. Considering the view-exclusive information loss and the model complexity, a weakly contrastive strategy and a Transformer-based view-shared feature extractor are finally constructed. Evaluation on four standard datasets demonstrates that the proposed model, HVSF, outperforms both the self-supervised models and supervised models, affirming its superior performance and stability.

Automatic modulation classification (AMC) has a wide range of applications in both civilian and military fields, such as industrial Internet of Things (IIoT) security, communication spectrum management, and military electronic countermeasures. However, label mislabeling often occurs in practical scenarios, significantly impacting the performance and robustness of deep neural networks (DNNs). In this article, we propose a meta-learning guided label noise distillation method to enhance the robustness of AMC models against label noise or errors. Specifically, we propose a teacher-student heterogeneous network (TSHN) to discriminate and distill label noise. Following the notion that labels represent information, a teacher network, utilizing trusted few-shot labeled samples, reevaluates and corrects labels for a considerable number of untrusted labeled samples through meta-learning. By dividing and conquering untrusted labeled samples according to their confidence levels, the student network learns more effectively. Additionally, we propose a multiview signal (MVS) method to further enhance the performance of hard-to-classify categories with few-shot trusted labeled samples. Extensive experiments on the RadioML2016 and HisarMod2019.1 data sets demonstrate that our methods significantly improve accuracy and robustness in signal AMC across diverse label noise scenarios, including symmetric, asymmetric, and mixed label noise. For example, compared to the baseline convolutional neural network with the cross-entropy loss, our proposed TSHN achieves a remarkable 1.26% to 36.84% accuracy improvement under symmetric label noise and 0.12% to 38.59% accuracy improvement under mixed label noise. Moreover, TSHN exhibits greater robustness to varying label noise rates compared to existing methods.

Building fair deep neural networks (DNNs) is a crucial step towards achieving trustworthy artificial intelligence. Delving into deeper factors that affect the fairness of DNNs is paramount and serves as the foundation for mitigating model biases. However, current methods are limited in accurately predicting DNN biases, relying solely on the number of training samples and lacking more precise measurement tools. Here, we establish a geometric perspective for analyzing the fairness of DNNs, comprehensively exploring how DNNs internally shape the intrinsic geometric characteristics of datasets-the intrinsic dimensions (IDs) of perceptual manifolds, and the impact of IDs on the fairness of DNNs. Based on multiple findings, we propose Intrinsic Dimension Regularization (IDR), which enhances the fairness and performance of models by promoting the learning of concise and ID-balanced class perceptual manifolds. In various image recognition benchmark tests, IDR significantly mitigates model bias while improving its performance.

To address the challenges of long-tailed classification, researchers have proposed several approaches to reduce model bias, most of which assume that classes with few samples are weak classes. However, recent studies have shown that tail classes are not always hard to learn, and model bias has been observed on sample-balanced datasets, suggesting the existence of other factors that affect model bias. In this work, we first establish a geometric perspective for analyzing model fairness and then systematically propose a series of geometric measurements for perceptual manifolds in deep neural networks. Subsequently, we comprehensively explore the effect of the geometric characteristics of perceptual manifolds on classification difficulty and how learning shapes the geometric characteristics of perceptual manifolds. An unanticipated finding is that the correlation between the class accuracy and the separation degree of perceptual manifolds gradually decreases during training, while the negative correlation with the curvature gradually increases, implying that curvature imbalance leads to model bias. We thoroughly validate this finding across multiple networks and datasets, providing a solid experimental foundation for future research. We also investigate the convergence consistency between the loss function and curvature imbalance, demonstrating the lack of curvature constraints in existing optimization objectives. Building upon these observations, we propose curvature regularization to facilitate the model to learn curvature-balanced and flatter perceptual manifolds. Evaluations on multiple long-tailed and non-long-tailed datasets show the excellent performance and exciting generality of our approach, especially in achieving significant performance improvements based on current state-of-the-art techniques. Our work opens up a geometric analysis perspective on model bias and reminds researchers to pay attention to model bias on non-long-tailed and even sample-balanced datasets.

Hyperspectral image (HSI) provides detailed spectral and spatial information, essential for precise earth observation and various applications. Deep learning has advanced HSI classification, but the scarcity of labeled data and large model parameters necessitate semi-supervised methods to enhance performance and generalization. In this paper, we propose a novel semi-supervised framework dubbed Knowledge-Aware Geometric Contourlet Semantic Learning (KGCSL), aiming to achieve high-precision HSI classification with limited samples leveraging geometric and semantic knowledge. Specifically, to fully leverage geometric knowledge, KGCSL incorporates multi-scale and multi-directional representations of the contourlet transform within the neural network, enhancing the robustness of feature extraction and interpretability. Furthermore, to fully utilize semantic knowledge, an entropy-weighted prototype loss function is designed that exploits the attribute relationships between labeled and unlabeled samples to guide the optimization of unlabeled samples, promoting comprehensive semantic learning. Comprehensive evaluations of the proposed KGCSL framework on three public HSI datasets show that it outperforms existing state-of-the-art HSI classification methods and exhibits excellent generalization capabilities in limited-sample scenarios. The source code is available athttps://github.com/ShirlySmile/KGCSL.

Deep neural networks (DNNs) have been successfully applied in the remote sensing semantic segmentation. However, training DNNs requires a large number of densely labeled samples, which is laborious and time-consuming. Sparsely supervised semantic segmentation (SSSS) can train deep segmentation networks using only sparse annotations. In this paper, we propose a negative class guided spatial consistency network (NCG-SCNet) for semantic segmentation with sparse annotations. Specifically, we introduce a spatial consistency enhancement module (SCEM) to enhance network features by non-linearly combining spatially similar features. Thus, it could provide better representations of the boundaries and the shape of the target. Additionally, a channel compression module (CCM) is proposed to reduce channel redundancy while preserving the network’s feature extraction capability. A negative class guided loss function (NCG Loss) is constructed to provide extra supervisory information, where the negative classes are defined as the classes with lower probability in the prediction. Extensive experiments on two widely used remote sensing datasets show that the proposed NCG-SCNet outperforms the comparison methods.

The diversity of contextual information is of great importance for accurate semantic segmentation. However, most methods focus on single spatial contextual information, which results in an overlap of the semantic content of categories and a loss of contour information of objects. In this article, we propose a novel contour knowledge-aware perception learning network (CKPL-Net) to capture diverse contextual information by space-category aggregation module (SCAM) and contour-aware calibration module (CACM). First, SCAM is introduced to enhance intraclass consistency and interclass differentiation of features. By integrating space-aware and category-aware attention, SCAM reduces the redundancy of features from a categorical perspective while maintaining spatial correlation of pixels, substantially avoiding the overlap of the semantic content in categories. Second, CACM is designed to maintain the integrity of objects by perceiving contour contextual information. It develops a novel contour-aware knowledge and adaptively transforms the grid structure of convolutions for boundary pixels, which effectively calibrates the representation of features near boundaries. Finally, the quantitative and qualitative analyses on the three public datasets: ISPRS Potsdam dataset, ISPRS Vaihingen dataset, and WHDLD dataset, demonstrate that the proposed CKPL-Net achieves superior performance compared with prevalent methods, which indicates diverse contextual information is beneficial for accurate segmentation.

Cross-scene hyperspectral image (HSI) classification faces spectral-spatial feature distribution shifts resulting from cross-domain heterogeneity, which has become a critical challenge that urgently needs resolution in the field of remote sensing intelligent interpretation. To address this distribution shift, the mainstream approach is Domain Generalization (DG). However, existing HSI DG methods primarily focus on inter-class separability, while paying relatively less attention to cross-domain transferability. To overcome the limitation that existing methods mainly focus on inter-class separability, this study proposes a cross-scene HSI classification network, termed DPSKDnet. By synergistically employing dynamic perturbation-based destylization and self-knowledge distillation modeling mechanisms, DPSKDnet constructs domain-invariant representations with strong generalization capabilities in the feature space. Specifically, this study first builds a generator based on dynamic perturbation destylization to mine source domain (SD) invariant features and generate extended domain (ED) samples. Subsequently, a Fourier Augmentation Module is utilized to optimize the frequency domain representations of the SD, ED, and their combination-generated intermediate domain, obtaining frequency-enhanced representations. To effectively improve the model’s ability to capture domain-invariant features, a sample pair distillation loss is devised. This loss, informed by multi-domain mixed data input, guides the discriminator in online self-supervised learning. The overall accuracy of this method on Loukia, Houston2018, and Pavia Center increased by 0.48%, 0.81%, and 1.5%, respectively, compared to state-of-the-art methods. The code is available on the website: https://github.com/Yuhang-Hong/TGRS_DPSKDnet.

In the field of multimodal remote sensing image (MRSI) classification, self-supervised learning (SSL) algorithms have demonstrated significant advantages, particularly in scenarios with limited labeled samples. Existing SSL methods typically use auxiliary tasks within either contrastive or generative frameworks, focusing on discriminative or structural information separately. In this article, we propose a novel hybrid SSL paradigm, discriminative autoencoder with Gaussian selective patch (DAE-GSP) for MRSI classification. The DAE framework integrates contrastive learning with the masked image modeling (MIM) technique, allowing for simultaneous learning of structural information and discriminative representations from images. Furthermore, a cross-attention-based data-level fusion strategy is introduced during pretraining stage to enhance intermodal interactions, thereby improving the effectiveness of modality fusion. In addition, we propose a novel Gaussian selective patch (GSP) strategy, addressing the limitations of traditional square patch selection methods. Combined with self-supervised auxiliary tasks, this strategy facilitates the improved integration of multiple modalities and encourages the model to capture essential semantic information. Extensive experiments conducted on three public datasets (Houston2013, Augsburg, and Berlin) demonstrate the effectiveness of the proposed approach. With only ten labeled training samples per class, the proposed method achieves overall accuracy (OA) of 90.15%, 82.64%, and 71.03% on the Houston2013, Augsburg, and Berlin datasets, respectively, indicating improvements of 1.31%, 1.22%, and 1.48% over state-of-the-art methods.

This study introduces a novel cross-modal spatial-spectral interaction Mamba (CMS2I-Mamba) for remote sensing image fusion classification. Unlike convolution-based models focusing on local details and Transformer-based models with high computational complexity, CMS2I-Mamba efficiently models global long-range dependencies in a linear complexity manner. First, multispectral (MS) and panchromatic (PAN) images each have unique advantages in the spectral and spatial attributes. Given this, this paper innovatively designs the multi-path selective-scan mechanism (MPS2M), which applies different path scanning strategies to deeply capture the global features from both spectral and spatial dimensions, aiming to enhance the robustness and complementarity of spatial-spectral features. Secondly, to overcome the characterization differences between images acquired by different sensors, this paper further introduces the channel interaction alignment module (CIAM). This module employs efficient former-last and oddeven channel interaction strategies to achieve precise semantic alignment of deep features between modalities. Finally, to leverage the shared fusion features to guide the unique singular features, this paper proposes a semantic-aware calibration module (SACM), which accurately constraints and calibrates the same semantic information in deep features. This not only enhances the model’s ability to understand scene semantics, but also promotes the deep fusion and utilization of information between different modalities. Through experimental verification on multiple datasets, the CMS2I-Mamba proposed in this paper shows excellent recognition performance and computational efficiency (parameter quantity and running speed) in fusion classification tasks. The code for CMS2I-Mamba is available at: https://github.com/ru-willow/CMSI-Mamba.

Deep neural networks often inherit biases from training data, compromising generalization. In visual recognition, distinguishing foreground from background is insufficient, as models tend to rely on spurious correlations rather than learning essential causal patterns. To address this issue, this paper proposes a novel transformer architecture, termed dual causal-aware detection transformer (DCDT), specifically designed for object detection in optical remote sensing images from a causal perspective. Specifically, we begin by constructing a structural causal model to intuitively analyze the causal effects inherent in the overall visual patterns. Building on this foundation, DCDT introduces causal constraints at the attention level by embedding dynamic multi-scale causal prototypes into the attention mechanism. The derived causal priors are subsequently used to enhance features at the representation level, thereby enforcing feature-level causal modulation. This dual causal-aware strategy enables the precise extraction and reinforcement of causally relevant features, improving both robustness and discriminative capability in complex detection scenarios. In addition, a sparse kernel-region mask is incorporated to decouple local information from global representations, effectively strengthening the modeling of fine-grained structures. Extensive experiments conducted on two challenging public datasets, DIOR and HRRSD, demonstrate that DCDT consistently outperforms existing methods and baselines. These results validate the effectiveness of DCDT in capturing both global causal semantics and local fine-grained features, highlighting its practicality in complex remote sensing scenarios.

In the field of remote sensing self-supervised learning, the self-supervised learning paradigm based on masked image modeling (MIM) effectively promotes the learning of structural and contextual semantic information in images. However, the random masking strategies employed in past studies have not effectively utilized the distribution of semantic information within images. Furthermore, overly simple reconstruction tasks cannot effectively improve self-supervised learning performance. To address these challenges, this paper introduces a gradient-guided activation-aware autoencoder (GA-MAE). Specifically, this approach computes and captures the spatial spectral activation map (SSAM) with semantic distribution information during the process of backpropagation of spatial-spectral reconstruction loss in the pre-training phase, thereby enabling masking operations on similar visual patches with higher reconstruction loss. Additionally, we introduce an Activation-Aware Self-Attention mechanism (ASA) that adjusts the self-attention dependencies by utilizing the weight information of visual patches provided by SSAM. Extensive experiments conducted on three public datasets (Houston2013, Augsburg, and Berlin) demonstrate the effectiveness of the proposed approach. With only ten labeled training samples per class, the proposed method achieves anoverall accuracy (OA) of 91.24%, 84.69%, and 73.43% on the Houston2013, Augsburg, and Berlin datasets, respectively, indicating improvements of 1.09%, 2.05%, and 2.40% over state-of-the-art methods.

In recent years, superpixel-based graph convolutional networks (GCNs) have drawn increasing attention within the hyperspectral image (HSI) classification community. Due to the high-dimensional property of HSI, establishing a high-quality and accurate initial graph is still a great challenge for the superpixel-based GCN methods. In addition, the lack of high-level semantics within the superpixel-based node features leads to poor classification performance of the model, especially in scenarios with limited labeled samples. To tackle these problems, we propose a novel approach called the dual-semantic graph convolution network (DSGCN) for HSI classification in this article. Specifically, our method employs superpixel segmentation to construct graph nodes with semantic structure information, treating each superpixel in the HSI as a node within the graph. We design a superpixel-level autoencoder that integrates with the initial graph to update the edge weights. With the learnable edge weights, our model can adaptively learn robust spatial semantic (SS) information from HSI. Additionally, we introduce a spectrum-flow (SF) module to extract global spectral semantic variation information. To further enhance the nonlinearity capability of GCN, we replace the traditional linear layer with a novel network layer referred to as Kolmogorov Arnold networks (KANs) during the node representation phase. In addition, we develop a memory-efficient residual spectral attention (MERSA) module that adapts to the full-batch training manner in the convolutional neural network (CNN) branch to supplement fine-grained pixel-level features. Extensive experiments conducted on four benchmark datasets demonstrate that our proposed DSGCN significantly outperforms several state-of-the-art methods, particularly when using a small amount of labeled data.

Remote sensing image super-resolution (RSISR) techniques are crucial in practice as an economical approach to enhancing the resolution of remote sensing images (RSIs). The scale of structural information and the richness of texture details in RSIs far exceed those in natural images. Therefore, accurately restoring and preserving edge and detail information are a critical challenge in the super-resolution (SR) process. Currently, convolutional neural network (CNN)-based methods primarily rely on local feature extraction, which fails to effectively capture and integrate global contextual information. Generative adversarial network (GAN)-based methods, while improving the visual quality, often suffer from artifacts and training instability, adversely affecting image quality. Moreover, these approaches struggle to accurately represent high-frequency features, leading to blurriness or distortion when reconstructing fine details and edges. To address these limitations, we introduce the channel–spatial coherent transformer (CSCT). The core of CSCT includes the channel–spatial coherent attention (CSCA) and the frequency-gated feed-forward network (FGFN), which work synergistically to enhance edge and detail preservation while significantly improving overall image clarity. CSCA efficiently aggregates channel and spatial information, while FGFN adaptively adjusts frequency information to enhance high-frequency details and suppress low-frequency noise. Moreover, this article leverages advanced data augmentation methods that markedly boost RSISR performance, offering new avenues for further exploration. The empirical analysis across several remote sensing SR benchmark datasets reveals that our approach excels in detail restoration, effectively reduces artifacts and noise, and significantly enhances the quality of SR images.

Recently, deep neural networks (DNNs) based emitter recognition or identification has received increasing interest. However, most of them are purely data-driven and require a large number of labeled instances. In this paper, a new Knowledge Driven Signal Transformer (KDSiT) is proposed, which introduces the knowledge graph (KG) into a signal Transformer (ST) model for accurate emitter recognition in real-world scenarios. On the one hand, KDSiT use a unified multimodal Transformer structure to explore the latent long-range dependencies in signals, and capture the subtle differences of emitters. On the other hand, KDSiT introduces domain knowledge, such as relationships and attributes between emitters, by constructing an emitter knowledge graph. By combining the powerful feature learning capability of DNNs with the rich semantic information in KG, KDSiT can extract more discriminative features of emitters from multimodal learning, to improve the identification accuracy in degraded environments. Extensive experiments are conducted, and the results prove the superiority of KDSiT over its counterparts, especially in the case of low signal-to-noise ratio (SNR), incomplete signals, and a limited number of labeled instances.

The proliferation of Internet of Things (IoT) devices generates substantial data that supports deep learning, significantly advancing intelligent specific emitter identification (SEI) technology. However, challenges such as labeling costs and privacy concerns limit the availability of labeled samples, thereby constraining deep model training. To address this problem, this paper focuses on enhancing the data manifold structure through deep feature information, proposing a semi-supervised SEI method named SSME. A well-structured manifold makes the model capture underlying patterns and relationships within the data more effectively, leading to more accurate and generalizable classification boundaries. First, to maximize the use of supervision information from limited labeled samples, we design a supervised cross-class contrastive (SCCC) loss, which increases the feature distance between anchor samples and cross-class samples based on their labels, achieving better manifold separation of different categories. Second, we propose an instance neighborhood matching regularization (INMR) loss that captures the neighborhood of weakly and strongly augmented samples of unlabeled instances within the feature space. By aligning these neighborhood representations, neighborhood-to-neighborhood consistency learning is achieved, enhancing the structural consistency and smoothness of local manifolds. Evaluated on ADS-B and XSRP datasets across diverse settings, our method demonstrates superior performance over existing approaches. Notably, even with only five labeled samples per class, it surpasses supervised baselines by 24.82% and 12.55% on the respective datasets.

Recently, contrastive clustering (CC) has exhibited considerable advantages for unsupervised specific emitter identification (USEI). However, emitter signals consist of connotative information, fingerprint information, and noise. As meaningful information transmitted by signals, connotation information hinders the extraction of discriminative fingerprints from emitter signals. In this paper, a novel Similarity Connotation Masking guided Contrastive Clustering (SC3) method is proposed for USEI. First, SC3 generates pairs of emitter signal samples with connotative exclusivity through the connotation masking module (CMM). Second, a translation-invariant multi-scale fingerprint extractor (TIMFE) with a wide receptive field to efficiently extract and decouple fingerprints. By separating the connotation information from the sample pairs in CC and learning robust features via TIMFE, SC3 could obtain accurate radio frequency fingerprints (RFFs) from degraded emitter signals. Extensive experiments are conducted on several datasets, including CBRS, Wi-Fi and XSRP datasets. The numerical results indicate that the proposed SC3 method consistently outperforms state-of-the-art algorithms regarding four clustering indicators. Code available: https://github.com/2017212073/SC3.

With the rapid development of wireless communications, industrial electromagnetic environments are facing challenges in terms of spectrum scarcity and cyberspace threats. Moreover, the coexistence of various types of radio signals within the same frequency band may cause signal distortion and degrade the quality and efficiency of communication. To effectively address these challenges, a novel temporal–spectral feature fusion network (TSFFN) for radio signal classification (RSC) is proposed. TSFFN adopts a Cutmix-based temporal–spectral fusion and an attention-based multiview feature fusion mechanism. These mechanisms automatically learn and merge spectrogram, temporal–spectral, and time-domain features by effectively combining temporal and spectral information into high-dimensional representations. This augmentation enhances the network's ability to discriminate different radio signals, enabling accurate spectrum sensing and signal identification for effective spectrum management. Experimental results on five datasets demonstrate the effectiveness of our approach in enhancing RSC performance across diverse industrial scenarios.

Cracks are critical defects in concrete structures, traditionally identified through human inspection. However, computer vision techniques, especially convolutional neural networks (CNNs), offer promising solutions for automated detection. Driven by this trend, this study proposesCrackVisionX, a state-of-the-art deep learning framework for classifying binary concrete cracks.CrackVisionXlies in its integration of advanced CNN architectures, ResNet50, MobileNet_v3_large, DenseNet121, and EfficientNetB0, with extensive hyper-parameter tuning. This integration optimizes crack detection accuracy while maintaining low model complexity and reducing bias, making it suitable for real-time applications. Furthermore, the framework introduces a robust data augmentation strategy that effectively addresses dataset imbalances, enhancing model generalization across diverse domains. Additionally,CrackVisionXemploys comprehensive preprocessing on the METU and SDNET2018 datasets to create six domains: Bridge Deck, Wall, Pavement, SDNET2018, METU, and METU & SDNET2018. The framework’s performance is thoroughly evaluated and benchmarked against state-of-the-art methods, utilizing diverse metrics to improve the detection of cracks in concrete structures. EfficientNetB0, a core component of the framework, demonstrated superior performance with exceptional test accuracies of up to 99.71%, 99.78%, 99.55%, 99.89%, 99.98%, and 99.92% for Bridge Deck, Wall, Pavement, SDNET2018, METU, and METU & SDNET2018, respectively. Moreover, we evaluated the robustness ofCrackVisionXusing images contaminated with different types and intensities of noise, demonstrating its reliability and effectiveness. This balance between high accuracy and computational efficiency confirms the framework’s potential for practical deployment. The experimental results emphasize the transformative potential of deep learning in construction safety and structural health monitoring.

View synthesis of aerial scenes has gained attention in the recent development of applications such as urban planning, navigation, and disaster assessment. This development is closely connected to the recent advancement of the Neural Radiance Field (NeRF). However, when autonomousaerial vehicles(AAVs) encounter constraints such as limited perspectives or energy limitations, NeRF degrades with sparsely sampled views in complex aerial scenes. On this basis, we aim to solve this problem in a few-shot manner. In this paper, we propose Uncertainty Guided Perception NeRF (UPNeRF), an uncertainty-guided perceptual learning framework that focuses on applying and improving NeRF in few-shot aerial view synthesis (FSAVS). First, simply optimizing NeRF in complex aerial scenes with sparse input can lead to overfitting in training views, resulting in a collapsed model. To address this, we propose a progressive learning strategy that utilizes the uncertainty present in sparsely sampled views, enabling a gradual transition from easy to hard learning. Second, to take advantage of the inherent inductive bias in the data, we introduce an uncertainty-aware discriminator. This discriminator leverages convolutional capabilities to capture intricate patterns in the rendered patches associated with uncertainty. Third, direct optimization of NeRF lacks prior knowledge of the scene. This, coupled with a reduction in training views, can result in unrealistic rendering. To overcome this, we present a perceptual regularizer that incorporates prior knowledge through prompt tuning of a self-supervised pre-trained vision transformer. In addition, we adopt a sampled scene annealing strategy to enhance training stability. Finally, we conducted experiments with two public datasets, and the positive results indicate our method is effective.

Visual transformers have achieved great success in representation learning. This is mainly due to efficient token dependency modeling via self-attention. However, the computational burden increases sharply as the input pixels increase. Although recent Fourier-based global frequency-domain mixing methods attempt to improve the efficiency of transformers for high-resolution image inputs, the Fourier operator has limited ability to capture the local geometric structure. Complex wavelets can perform local attention in both the spatial domain and the frequency domain. Therefore, we propose the complex wavelet informed transformer operator that uses the real and imaginary wavelets of the dual-tree complex wavelet transform to simulate the interaction in the attention kernel. In order to further reduce the computational burden of operators, we introduce an adaptive local block shared attention mechanism in the channel domain for our wavelet informed operators. Further, we construct the deep multi-head operator network consisting of a hybrid stack of complex wavelet informed transformer operators and self-attention layers. This enables the Transformer to more sparsely capture multi-scale and multi-directional structured features in the process of learning dependencies. Extensive experimental results show that our adaptive complex wavelet informed transformer operator under the Transformer architecture achieves highly competitive accuracy performance on multiple image classification benchmark datasets. And the proposed operators can be flexibly and effectively migrated to vision tasks in dynamic video scenarios.

Deep learning based remote sensing (RS) image segmentation significantly impacts several real application scenarios. Behind its success, massive labeled data plays an important role. However, annotating high-resolution RS images requires time-consuming and relevant expertise efforts. To address it, many works dive into semi-supervised learning which utilizes raw information embedded in unlabeled data to improve the segmentation model. Nevertheless, previous studies ignore the integrity and effectiveness of the potential context information hidden in RS data. In this work, we propose an uncertainty-aware masked consistency learning (U-MCL) framework that contains an uncertainty-aware masked denoising (U-MD) module and an uncertainty-aware masked image consistency (U-MIC) module. U-MCL initially generates a patch-wise uncertainty map for each unlabeled image during each training iteration, which is then used to derive an adaptive mask ratio for pseudo-label denoising in U-MD. Simultaneously, the uncertainty map is adopted to model a masked unlabeled image for reasoning unseen areas in U-MIC. Consequently, U-MCL is capable of enhancing model performance by engaging in accurate and stable consistency learning while preserving the integrity of the context and employing the context to infer the predictions of the masked regions safely. Extensive experiments on six RS datasets, i.e., ISPRS Vaihingen, FloodNet, MiniFrance, LoveDA, MER, and MSL, demonstrate the superiority of our U-MCL over recent most advanced methods, achieving new state-of-the-art performance under all benchmarks.

As imaging sensor technology in remote sensing has advanced quickly, multimodal fusion classification has become an important research direction in land cover and urban planning classification tasks. While generative models and image classification have greatly benefited from diffusion models, the present ones primarily concentrate on single-modality-driven diffusion processes. Therefore, this paper presents a 3D self-awareness diffusion network (3DSA-DiffNet) for multispectral (MS) and panchromatic (PAN) image fusion classification, which would make it easier to classify heterogeneous data from various sensors. First, in order to model the relationship between multi-channel spectra and multi-pixel spatial distributions as well as samples, respectively, a spatial-spectral joint denoising network (S$^{2}$JD-Net) is proposed. It can incorporate the diffusion process into the neural network to enhance the quality of diffusion features. Secondly, to imitate the brain's spatial-spectral coexistence learning mechanism, this work offers a 3D self-awareness module (3DSA-Module) that can learn the weight of each pixel in 3D space, resulting in extraordinarily high feature representation capabilities. Finally, experimental verification demonstrates that the 3D self-awareness diffusion fusion network driven by brain inspiration outperforms more sophisticated approaches on the Xi'an, Huhhot, and Muufl datasets.

In satellite video object tracking, the individual frame analysis method is usually used for target localization, ignoring informative cues of the dynamic scene. Temporal information could contribute to identifying the target from distractors. In this work, a novel dynamic scene learning reasoning tracker is proposed for satellite videos, which reasons over temporal dynamic information to derive the target location. It is inspired by the tracking pattern through human perception and reasoning. First, static-dynamic united analysis is designed to construct dynamic scenes by concatenating the static searching results along the temporal dimension. Second, the information of each response object is aggregated by wavelet transforms. Meanwhile, these scenes are projected into low-frequency and high-frequency subspaces, which could imitate different levels of perceptions of humans for scenes. Third, an object-aware reasoning transformer is proposed to utilize the temporal dynamics of input response objects. In each subspace, it models the mutual interactions between dynamic objects and further learns the intrinsic property of each object for target reasoning. Finally, to obtain the current reasoning result, inverse wavelet transforms are utilized to integrate the results of low-frequency and high-frequency subspaces. The effectiveness of the proposed method is validated on three public satellite video datasets, including SV248S, SkySat, and VISO. Qualitative and quantitative experimental results show that the proposed tracker outperforms 22 popular approaches in seven challenging tracking satellite scenarios.

The preservation and enhancement of texture information is crucial for the fusion of visible and infrared images. However, most current deep neural network (DNN)-based methods ignore the differences between texture and content, leading to unsatisfactory fusion results. To further enhance the quality of fused images, we propose a texture-content dual guided (TCDG-Net) network, which produces the fused image by the guidance inferred from source images. Specifically, a texture map is first estimated jointly by combining the gradient information of visible and infrared images. Then, the features learned by the shallow feature extraction (SFE) module are enhanced with the guidance of the texture map. To effectively model the texture information in the long-range dependencies, we design the texture-guided enhancement (TGE) module, in which the texture-guided attention mechanism is utilized to capture the global similarity of the texture regions in source images. Meanwhile, we employ the content-guided enhancement (CGE) module to refine the content regions in the fused result by utilizing the complement of the texture map. Finally, the fused image is generated by adaptively integrating the enhanced texture and content information. Extensive experiments on three benchmark datasets demonstrate the effectiveness of the proposed TCDG-Net in terms of qualitative and quantitative evaluations. Besides, the fused images generated by our proposed TCDG-Net also show better performance in downstream tasks, such as objection detection and semantic segmentation.

Multisource optical remote sensing (RS) image classification has obtained extensive research interest with demonstrated superiority. Existing approaches mainly improve classification performance by exploiting complementary information from multisource data. However, these approaches are insufficient in effectively extracting data features and utilizing correlations of multisource optical RS images. For this purpose, this article proposes a generalized spatial-spectral relation-guided fusion network (S2RGF-Net) for multisource optical RS image classification. First, we elaborate on spatial- and spectral-domain-specific feature encoders based on data characteristics to explore the rich feature information of optical RS data deeply. Subsequently, two relation-guided fusion strategies are proposed at the dual-level (intradomain and interdomain) to integrate multisource image information effectively. In the intradomain feature fusion, an adaptive de-redundancy fusion module (ADRF) is introduced to eliminate redundancy so that the spatial and spectral features are complete and compact, respectively. In interdomain feature fusion, we construct a spatial-spectral joint attention module (SSJA) based on interdomain relationships to sufficiently enhance the complementary features, so as to facilitate later fusion. Experiments on various multisource optical RS datasets demonstrate that S2RGF-Net outperforms other state-of-the-art (SOTA) methods.

The progress of brain cognition and learning mechanisms has provided new inspiration for the next generation of artificial intelligence (AI) and provided the biological basis for the establishment of new models and methods. Brain science can effectively improve the intelligence of existing models and systems. Compared with other reviews, this article provides a comprehensive review of brain-inspired deep learning algorithms for learning, perception, and cognition from microscopic, mesoscopic, macroscopic, and super-macroscopic perspectives. First, this article introduces the brain cognition mechanism. Then, it summarizes the existing studies on brain-inspired learning and modeling from the perspectives of neural structure, cognitive module, learning mechanism, and behavioral characteristics. Next, this article introduces the potential learning directions of brain-inspired learning from four aspects: perception, cognition, understanding, and decision-making. Finally, the top-ten open problems that brain-inspired learning, perception, and cognition currently face are summarized, and the next generation of AI technology has been prospected. This work intends to provide a quick overview of the research on brain-inspired AI algorithms and to motivate future research by illuminating the latest developments in brain science.

Recently, the multiscale problem in computer vision has gradually attracted people's attention. This article focuses on multiscale representation for object detection and recognition, comprehensively introduces the development of multiscale deep learning, and constructs an easy-to-understand, but powerful knowledge structure. First, we give the definition of scale, explain the multiscale mechanism of human vision, and then lead to the multiscale problem discussed in computer vision. Second, advanced multiscale representation methods are introduced, including pyramid representation, scale-space representation, and multiscale geometric representation. Third, the theory of multiscale deep learning is presented, which mainly discusses the multiscale modeling in convolutional neural networks (CNNs) and Vision Transformers (ViTs). Fourth, we compare the performance of multiple multiscale methods on different tasks, illustrating the effectiveness of different multiscale structural designs. Finally, based on the in-depth understanding of the existing methods, we point out several open issues and future directions for multiscale deep learning.

Attention-based transformer networks have recently played an increasingly important role in computer vision tasks. However, since pixel-by-pixel attention multiplication does not involve constraint assumptions such as spatial invariance, the computational complexity grows quadratically with the increase of input pixels. Therefore, this article proposes a complex pyramid scattering Transformer in dense scale space, which introduces sparse scattering constraints with a small number of wavelet basis parameters. It enhances the Transformer's flexibility and sparsity in multiscale space and, to a certain extent, slows down the increase in computational complexity caused by multiresolution input. In addition, compared with the general single-tree real wavelet transform, the dual-tree complex scattering method improves the aliasing of the scattering attention layer and helps obtain a more robust feature representation. At the same time, the multihead stepwise pyramid scattering coupling mechanism helps increase the abundance of directional priors. We conduct experiments in image classification and video tracking scenarios and verify the reliability and superiority of our dual-tree complex pyramid scattering Transformer for visual tasks with different scale requirements. The performance is better than that of the baseline Transformer and other advanced wavelet scattering networks at the same parameter scale. The code is available at https://github.com/Dawn5786/CPSTFormer.

The Transformer-convolutional neural network (CNN) hybrid learning approach is gaining traction for balancing deep and shallow image features for hierarchical semantic segmentation. However, they are still confronted with a contradiction between comprehensive semantic understanding and meticulous detail extraction. To solve this problem, this article proposes a novel Transformer-CNN hybrid hierarchical network, dubbed contourlet transformer (CoT). In the CoT framework, the semantic representation process of the Transformer is unavoidably peppered with sparsely distributed points that, while not desired, demand finer detail. Therefore, we design a deep detail representation (DDR) structure to investigate their fine-grained features. First, through contourlet transform (CT), we distill the high-frequency directional components from the raw image, yielding localized features that accommodate the inductive bias of CNN. Second, a CNN deep sparse learning (DSL) module takes them as input to represent the underlying detailed features. This memory- and energy-efficient learning method can keep the same sparse pattern between input and output. Finally, the decoder hierarchically fuses the detailed features with the semantic features via an image reconstruction-like fashion. Experiments demonstrate that CoT achieves competitive performance on three benchmark datasets: PASCAL Context [57.21% mean intersection over union (mIoU)], ADE20K (54.16% mIoU), and Cityscapes (84.23% mIoU). Furthermore, we conducted robustness studies to validate its resistance against various sorts of corruption. Our code is available at: https://github.com/yilinshao/CoT-Contourlet-Transformer.

Existing efforts are dedicated to designing many topologies and graph-aware strategies for the graph Transformer, which greatly improve the model's representation capabilities. However, manually determining the suitable Transformer architecture for a specific graph dataset or task requires extensive expert knowledge and laborious trials. This article proposes an evolutionary graph Transformer architecture search (EGTAS) framework to automate the construction of strong graph Transformers. We build a comprehensive graph Transformer search space with the micro-level and macro-level designs. EGTAS evolves graph Transformer topologies at the macro level and graph-aware strategies at the micro level. Furthermore, a surrogate model based on generic architectural coding is proposed to directly predict the performance of graph Transformers, substantially reducing the evaluation cost of evolutionary search. We demonstrate the efficacy of EGTAS across a range of graph-level and node-level tasks, encompassing both small-scale and large-scale graph datasets. Experimental results and ablation studies show that EGTAS can construct high-performance architectures that rival state-of-the-art manual and automated baselines.

Open-set modulation classification (OMC) of signals is a challenging task for handling "unknown" modulation types that are not included in the training dataset. This article proposes an incremental contrastive learning method for OMC, called Open-ICL, to accurately identify unknown modulation types of signals. First, a dual-path 1-D network (DONet) with a classification path (CLP) and a contrast path (COP) is designed to learn discriminative signal features cooperatively. In the COP, the deep features of the input signal are compared with the semantic feature centers (SFCs) of known classes calculated from the network, to infer its signal novelty. An unknown signal bank (USB) is defined to store unknown signals, and a novel moving intersection algorithm (MIA) is proposed to dynamically select reliable unknown signals for the USB. The "unknown" instances, together with SFCs, are continuously optimized and updated, facilitating the process of incremental learning. Furthermore, a dynamic adaptive threshold (DAT) strategy is proposed to enable Open-ICL to adaptively learn changing signal distributions. Extensive experiments are performed on two benchmark datasets, and the results demonstrate the effectiveness of Open-ICL for OMC.

The past decade has witnessed the rapid development of deep neural networks (DNNs) for automatic modulation classification (AMC). However, most of the available works learn signal features from only a single domain via DNNs, which is not reliable enough to work in uncertain and complex electromagnetic environments. In this brief, a new cross-domain signal transformer (CDSiT) is proposed for AMC, to explore the latent association between different domains of signals. By constructing a signal fusion bottleneck (SFB), CDSiT can implicitly fuse and classify signal features with complementary structures in different domains. Extensive experiments are performed on RadioML2016.10A and RadioML2018.01A, and the results show that CDSiT outperforms its counterparts, particularly for some modulation modes that are difficult to classify before. Through ablation experiences, we also verify the effectiveness of each module in CDSiT.

With the advancement of cognitive radio technology (CRT) in radio communication networks, deep learning (DL) has become instrumental in enhancing spectrum efficiency. However, supervised DL methods demand extensive labeled data and incur high manual costs. Consequently, practical applications of CRT increasingly necessitate techniques capable of learning robust representations from large volumes of unlabeled data. Although recent DL advancements have driven the use of self-supervised learning (SSL) in CRT through time-domain contrastive methods, these approaches fall short in extracting high-level spectral representations due to their neglect of time-frequency features. To address these limitations, a generative SSL framework is proposed for CRT applications. First, SSL pretraining is conducted in the time-frequency domain by reconstructing masked spectrograms using a Masked Autoencoder. Then, to recover the spectrogram under extreme radio conditions, mutual information maximization is employed to extract high-level spectral information obscured by noise patterns. Additionally, an attention-based channel-spectrum fusion module is designed to automatically extract and integrate features from the channel and spectral domains. The feasibility of the proposed framework is evaluated across multiple downstream tasks on four public datasets. Experimental results demonstrate that the proposed framework significantly outperforms existing methods in various downstream tasks.

Most existing wideband signal detection and recognition (WSDR) methods rely on diverse, large-scale, and well-labeled training data, which are often difficult to obtain in practical application scenarios such as non-cooperative environments and novel signaling regimes. In this article, we propose a method for constructing a virtual signal large model (VSLM) and applying it to tackle the WSDR challenge under few-shot or even cross-domain few-shot scenarios. Firstly, we design two plug-and-play modules, virtual sample generation (VSG) and virtual category generation (VCG), for VSLM, respectively. VSG simulates the local and overall relationship between the burst signal and the constant signal, which is mainly completed by extracting time-frequency meta-block and data enhancement. Based on VSG and the multi-label concept, we further create virtual novel categories by injecting customizable semantic information into meta-blocks. Then, we further propose a dual decoupled network (DDN) to train the VSLM. DDN enhances signal details by decoupling low gray values (DLGV) in time-frequency representation, and alleviates conflicts during multi-task joint optimization by decoupling spectrum localization and signal classification. Finally, based on the wideband spectrogram dataset, extensive experiments have validated that our proposed methods can significantly improve the performance of WSDR under few-shot conditions.

Deep learning methods have achieved outstanding performance in various signal tasks. However, due to degraded signals in real electromagnetic environment, it is crucial to seek methods that can improve the representation of signal features. In this paper, a Singular Value decomposition-based Attention, SVA is proposed to explore structure of signal data for adaptively enhancing intrinsic feature. Using a deep neural network as a base model, SVA performs feature semantic subspace learning through a decomposition layer and combines it with an attention layer to achieve adaptive enhancement of signal features. Moreover, we consider the gradient explosion problem brought by SVA and optimize SVA to improve the stability of training. Extensive experimental results demon-strate that applying SVA to a generalized classification model can significantly improve its ability in representations, making its recognition performance competitive with, or even better than, the state-of-the-art task-specific models.

Fault diagnosis is crucial in mechanical prognostics and health management. However, fault features extracted from single-sensor data are limited in complex operating environments. Extracting complementary and robust fault features from multi-sensor monitoring data is essential, especially under limited labeled samples. Leveraging the advantages of self-supervised learning, we propose a novel cross-sensor self-supervised learning (CSSL) method for rotating machinery fault diagnosis under limited sample conditions. Our method employs contrastive learning across multiple sensors, including both intra-sensor and inter-sensor contrastive learning, to derive robust cross-sensor fault representations. The efficacy of our approach is substantiated on two benchmark datasets, revealing superior classification performance. Furthermore, the experimental results under various operating conditions demonstrate outstanding performance and solid robustness.

Time series analysis finds wide applications in fields such as weather forecasting, anomaly detection, and behavior recognition. Previous methods attempted to model temporal variations directly using 1D time series. However, this has been quite challenging due to the discrete nature of data points in time series and the complexity of periodic variation. In terms of periodicity, taking weather and traffic data as an example, there are multi-periodic variations such as yearly, monthly, weekly, and daily, etc. In order to break through the limitations of the previous methods, we decouple the implied complex periodic variations into inclusion and overlap relationships among different level periodic components based on the observation of the multi-periodicity therein and its inclusion relationships. This explicitly represents the naturally occurring pyramid-like properties in time series, where the top level is the original time series and lower levels consist of periodic components with gradually shorter periods, which we call the periodic pyramid. To further extract complex temporal variations, we introduce self-attention mechanism into the periodic pyramid, capturing complex periodic relationships by computing attention between periodic components based on their inclusion, overlap, and adjacency relationships. Our proposed Peri-midFormer demonstrates outstanding performance in five mainstream time series analysis tasks, including short- and long-term forecasting, imputation, classification, and anomaly detection.

With the continuous growth of Internet of Things (IoT) deployments, various wireless devices and communication technologies coexist in industrial environments resulting in a crowded and dynamic frequency spectrum. Efficient spectrum sensing becomes essential to mitigate interference, enhance the communication reliability, and ensure the seamless coexistence of diverse wireless technologies. However, the extremely dense and varied signals bring challenges for the precise detection, estimation, and recognition of signals in complex and varied signal environments. The residual dilated network (RDN) and horizontal shift attention (HSA) mechanisms presented in this article offer innovative solutions to the challenges posed by this intricate spectrum landscape. Through multiscale dilated convolution and attention mechanisms, our approach aims to capture and locate the signals precisely, enabling enhanced spectrum utilization within IoT applications. Extensive experiments are conducted on the two data sets, and the results show that our proposed method can automatically extract discriminative features of signals, thereby improving the detection accuracy and recall rate in spectrum sensing. In addition, the probability of false negatives and the inference time can also be reduced simultaneously.

In recent years, data-driven deep learning methods have significantly improved the performance of automatic modulation classification (AMC). However, labeling the vast number of signal samples obtained in a complex electromagnetic environment is challenging due to data security concerns and the drain on manpower and material resources. The scarcity of labeled samples constrains the applicability of these methods. In this article, an ensemble SigMatch (ESM) semi-supervised AMC method is proposed to fully leverage the unlabeled modulated signals. First, a SigMatch (SM) semi-supervised AMC framework is proposed, combining pseudo-labeling, consistency regularization, and modulated signal augmentation for direct identification of raw timing signals. Three different types of signal augmentation methods are investigated through mathematical analysis of the signal model. Second, based on SM and multiview learning, the ESM method is proposed to further enhance the performance of semi-supervised AMC through consistency learning of multiple augmentation views of unlabeled signals. A multiview consistency loss is designed in ESM, with additional data augmentation as complementary views. Multiple perturbed views are guided by the same sample to achieve consistent classification through a shared classification model, thus achieving more robust feature representation. Our method demonstrates remarkable performance on data sets RML2016.10A and RML2016.04C, especially with few labeled samples. On RML2016.10A, with only 110 labeled samples, the ESM enhances the overall classification accuracy from 35.77% to 70.44% compared with supervised learning.

In the non-cooperative electromagnetic environment, new radar emitters will emerge unexpectedly during the test phase, which brings the “Open-Set” Radar Emitter Recognition (OS-RER). Conventional classifiers cannot identify new radar emitters that do not exist in the training dataset. Therefore, in this paper, a novel Deep Metric Auto-Encoder (DMAE) is proposed for OS-RER. In DMAE, deep metric learning learns new non-linear mappings in the metric space to measure the similarity between instances. The dual-path deep auto-encoder is designed to reduce the open space risk by learning a low-dimensional manifold and a discriminative representation of known instances. Specifically, DMAE models known classes, and measures class belongingness through the reconstruction error of the AE and the entropy of the classifier. The deep metric network learns a more precise distance metric by minimizing the distance between the known class instances and the corresponding reconstruction. To accurately detect unknown instances, the classifier and the deep metric network are used together to preliminarily detect unknown instances. Finally, the detected unknown instances are used to further train the classifier to recognize the radar emitter in the open-set scenarios. The DMAE learns the discriminative representation through end-to-end learning. Extensive experiments conducted on real radar datasets and simulated radar datasets show that DMAE can identify unknown emitters and significantly outperforms existing open-set classification methods.

This work delves into unveiling the singularity issue latent in global attention-based Transformers. Empirical and theoretical analyses elucidate that interrelationships among token channels lead to singularities, impeding the training of attention weights. Concretely, the similar neighbor pixels within image patches can form intercorrelated channels after being flattened. Images that one color dominates can possess correlated channels . Furthermore, the fixed global connection architecture retains correlation relationships, contributing to the persistence of singularities. High singularity risks reducing Transformers’ performance and robustness. Based on the singularity analysis, we propose the Token Singularity Removal (TSR) strategy. It incorporates the Dual-Tree Complex Wavelet Transform (DTCWT) stem and Feature Decorrelation (FD) loss, aiming to encourage Transformers to learn tokens with unrelated channels and eliminate singularities. Experimental validation across various image classification datasets and corruption image data sets demonstrate improved accuracy and robustness of Transformers utilizing the TSR strategy. Our code is publicly available at https://github.com/wdanc/TSR .

Cross-domain classification of hyperspectral data is a critical challenge in remote sensing, especially when labels are unavailable in the target domain. Deep learning-based domain adaptation (DA) methods have been widely used in recent years. However, curren methods primarily focus on the global domain structure of the source and target domains when considering domain adaptation, neglecting the subdomain structure within each class. Additionally, current methods directly employ predicted outputs without further exploring the confidence level of the target domain samples. These limitations lead to confusion in domain adaptation and hinder effective feature selection in neural networks. In this paper, we propose the Pseudo-Label-Assisted Subdomain Adaptation (PASDA) method, which addresses these limitations by jointly considering the subdomain structure of the source and target domains and adopting a sample selection strategy. PASDA aligns the subdomains while learning domain-invariant features as a foundation. Furthermore, it selects high-quality pseudo-labeled samples from the target domain to enhance the learning of domain-invariant features. For generating pseudo-labels in the target domain, we employ the Reweighted Pruning Label Propagation (RPLPA) strategy to reweight the output of the predicted target domain. Finally, the high-confidence samples with pseudo-labels are selected to finetune the network. The entropy regularized dual classifier constraint is introduced to enhance the discriminative feature extraction ability for the target domain. Extensive experiments on three public HSI cross-domain datasets, Pavia, Houston, and HyRANK, using overall accuracy (OA), average accuracy (AA) and kappa coefficient (Kappa) as the evaluation indicators of classification performance, demonstrate the superiority of our method. Compared with the existing state-of-the-art (SOTA) unsupervised domain adaptation (UDA) methods, our method improves OA by 2% and AA by 4%.

Recently, semi-supervised semantic segmentation methods based on weak-to-strong consistency learning have achieved the most advanced performance. The key to such a technique lies in strong perturbations and multi-objective co-training. However, CutMix, the most commonly used data augmentation in this field, limits the strength of perturbations as it only focuses on single random local context. Besides, complex optimization targets also reduce computational efficiency. In this work, we propose an efficient consistency learning based framework. Specifically, a novel unsupervised data augmentation strategy, EntropyMix, is present for semi-supervised semantic segmentation. Patches of unlabeled data from multi-view augmentations are combined into new training samples based on their prediction entropy, which provides more informative and powerful perturbations for consistency regularization and impels the model to focus on cross-view local context. On this basis, we further propose Self Pseudo Entropy Knowledge Distillation (SPEED) to learn global pixel relations from multi- and cross-view perturbations by optimizing a linear combination of feature-and logit-level distillation loss, enhancing model performance without additional auxiliary segmentation heads or a complex pre-trained teacher model. The collocation of the two ideas above is a plug-and-play technique without additional modification. Extensive experimental results on PASCAL VOC and Cityscapes datasets under various training settings demonstrate the superiority of the proposed data augmentation strategy and self-distillation loss, achieving new state-of-the-art performance. Remarkably, our method reaches mIoU of 75.16% using only 0.87% labeled data on PASCAL VOC and mIoU of 76.98% using only 6.25% labeled data on Cityscapes. The code is available at https://github.com/xiaoqiang-lu/SPEED.

A growing number of earth observation satellites are able to simultaneously gather multimodal images of the same area due to the expanding availability and resolution of satellite remote sensing data. This paper proposes a novel multimodal balanced self-learning interaction network (MBSI-Net) for the classification task. It involves a dual-branch teacher-student network that enables knowledge interaction and transfer between the multimodalities. Firstly, in order to introduce statistical information in addition to local and global structural information, a texture feature equalization module (TFE-Module) is proposed. This can enhance the texture information of features through histogram equalization and further improve the representation ability of features. Secondly, to enable the student network to provide timely feedback questions, the paper proposes a feature fusion module (F2-Module) that models and enhances teacher features through the student network. This helps to raise the classification’s accuracy by incorporating information from multimodal images. Finally, the paper proposes a loss function based on structural similarity analysis to ensure balanced self-learning between the student and the teacher networks. Taking the multispectral (MS) and the panchromatic (PAN) images of the same scene as examples, through experimental verification, the proposed method can achieve good results on multiple datasets compared with other methods. Therefore, it offers an effective method for classifying and fusing multimodal data.

The abundant spectral signatures and spatial contexts are effectively utilized as the key to hyperspectral image (HSI) classification. Existing convolutional neural networks (CNNs), only focus on locally spatial context information and lack the ability to learn global spectral sequence representations, whereas the transformer performs well in learning the global dependence of sequential data. To solve this issue, inspired by the transformer, we propose an interactive global spectral and local spatial feature fusion transformer called ISSFormer. Specifically, we achieve an elegant integration of self-attention and convolution in a parallel design, i.e., the multi-head self-attention mechanism (MHSA) and the local spatial perception mechanism (LSP). ISSFormer can learn both local spatial feature representation and global spectral feature representation simultaneously. More significantly, we propose a bi-directional interaction mechanism (BIM) of features across the parallel branch to provide complementary clues. The local spatial features and the global spectral features interact through the BIM which could emphasize the local spatial details and add spatial constraints to overcome spectral variability, and can further improve classification performance. With extensive experiments on three benchmark datasets, including Indian Pines, Pavia University, and WHU-Hi-HanChuan, ISSFormer can accomplish superior classification accuracy and visualization performance.

Visual object tracking in natural scenes is a popular but challenging task, owing to the difficulties of feature representation from various changes of the targets, such as size change, deformation, illumination change, rotations, motion blur, background clutter, etc. High-speed hyperspectral imaging systems capture hyperspectral videos (HSVs) in wide spectral ranges and provide abundant spectral and spatial information to tell targets apart from backgrounds, alleviating the model drift in appearance-based tracking methods. However, different hyperspectral imagers, such as near-infrared (NIR), red-to-near-infrared (RedNIR), and visible (VIS), obtain heterogeneous types of data that could not be handled by common object trackers. In this paper, a domain adaptive Transformer framework is proposed for hyperspectral object tracking. Considering the HSVs are from different types of sensors, their heterogeneous features are learned in an adversarial way by domain label reverse learning with a gradient reversed layer. To fully utilize the spectral information in HSV frames, a band-wise spatial attention module (BSAM) is designed to emphasize the salient area near the target of interest. We adopt a Siamese-like Transformer tracker as the main structure for tracking. Our tracker outperforms top-ranking methods on a hyperspectral object tracking benchmark dataset containing three types, 87 hyperspectral videos in total. The comparison experiments validate the effectiveness of the proposed method. The source code and trained models of this work will be publicly available soon at https://github.com/LianYi233/Trans-DAT.

Existing wavelet pooling methods discard the high-frequency sub-bands, which can improve the noise-robustness of convolutional neural networks (CNNs) but lose the essential detailed features. Besides, most of them depend on different wavelets, which is not adaptive. In this paper, a novel efficient lifting-based wavelet pooling (LWPooling) is proposed to alleviate the problems above. Firstly, wavelet pooling is rethought based on the equivalence of 2D discrete wavelet transform (DWT) and standard average pooling (SAP), which suggests the lack of detailed information on traditional wavelet pooling. Secondly, the efficient LWPooling module is proposed to adaptively capture and preserve the critical high-frequency features via lifting-based wavelets. It can constrain the features linear independence, which efficiently makes important features salient. Thirdly, the lifting-based wavelet collaborative network (LWCNet) is constructed for classification and segmentation tasks based on the efficient LWPooling module. Experiments are validated on Cifar10, Cifar100, and ADE20K datasets. It suggests that the efficient LWPooling can enhance CNN’s representation and achieve a particular performance advantage compared to average, maximum, and original wavelet pooling. Besides, the proposed LWCNet shows the potential for scene parsing. The code implementation will be available at https://github.com/yutinyang/LWCNet.

Solving the complex challenges of sophisticated terrain and multi-scale targets in remote sensing (RS) images requires a synergistic combination of Transformer and convolutional neural network (CNN). However, crafting effective CNN architectures remains a major challenge. To address these difficulties, this study introduces the knowledge guided evolutionary Transformer for RS scene classification (Evo RSFormer). It amalgamates adaptive evolutionary CNN (Evo CNN) with Transformers in a hybrid strategy synergistically, which combines fine-grained local feature extraction of CNNs with long-range contextual dependency modeling of Transformers. Furthermore, for the development of Evo CNN blocks, this paper presents a knowledge-guided adaptive efficient multi-objective evolutionary neural architecture search (MOE2-NAS) strategy. This approach markedly diminishes the labor-intensive characteristics associated with traditional CNN design, striking a balance for both accuracy and compactness. Additionally, by leveraging domain knowledge from natural scene analysis into the RS field, MOE2-NAS facilitates the efficiency of classical NAS. It utilizes a priori knowledge to generate promising initial solutions and constructs a surrogate model for efficient search. The effectiveness of the proposed Evo RSFormer has been rigorously tested on various benchmark RS datasets, including UC Merced, NWPU45, and AID. Empirical results strongly support the superiority of Evo RSFormer over existing methods. Furthermore, experiments on MOE2-NAS have been studied to confirm the important role of knowledge guidance in improving the efficiency of NAS.

Vision transformers (ViTs) are rapidly evolving and are widely used in computer vision. However, high-performance ViTs require many computations, which limit their further development in the vision field. In this article, a novel evolutionary dual-stream transformer (E-DST) model is proposed to alleviate the computational resource demand problem. A hybrid attention mechanism structure is proposed for a DST model. The DST model uses a dual-branch structure to fuse convolutional and transformer features. Combining the features learned by the transformer and convolution effectively saves model computational resources. In addition, an evolutionary optimizer is proposed to optimize the parameters of the model. The excellent search ability of the evolutionary algorithm is utilized to optimize the transformer model parameters. The convergence of the evolutionary optimizer is proved in this article. In addition, the proposed E-DST model is experimentally compared with a variety of classic models and their deformations based on three datasets. And, the evolutionary optimizer proves its generality in convolutional and recurrent neural networks. The experimental results show that the E-DST model can effectively reduce computational resources and that the evolutionary optimizer can solve large-scale optimization problems. In conclusion, our proposed method is feasible and effective.

The construction of machine learning models involves many bi-level multiobjective optimization problems (BL-MOPs), where upper-level (UL) candidate solutions must be evaluated via training weights of a model in the lower level (LL). Due to the Pareto optimality of subproblems and the complex dependency across UL solutions and LL weights, a UL solution is feasible if and only if the LL weight is Pareto optimal. It is computationally expensive to determine which LL Pareto weight in the LL Pareto weight set is the most appropriate for each UL solution. This article proposes a bi-level multiobjective learning framework (BLMOL), coupling the above decision-making process with the optimization process of the upper-level MOP (UL-MOP) by introducing LL preference$\boldsymbol {r}$. Specifically, the UL variable and$\boldsymbol {r}$are simultaneously searched to minimize multiple UL objectives by evolutionary multiobjective algorithms. The LL weight with respect to$\boldsymbol {r}$is trained to minimize multiple LL objectives via gradient-based preference multiobjective algorithms. In addition, the preference surrogate model is constructed to replace the expensive evaluation process of the UL-MOP. We consider a novel case study on multitask graph neural topology search. It aims to find a set of Pareto topologies and their Pareto weights, representing different tradeoffs across tasks at UL and LL, respectively. The found graph neural network is employed to solve multiple tasks simultaneously, including graph classification, node classification, and link prediction. Experimental results demonstrate that BLMOL can outperform some state-of-the-art algorithms and generate well-representative UL solutions and LL weights.

Video object tracking has been a popular area in the field of computer vision. As video data evolves, more special perspectives and challenging video data are constantly kept up to date. This poses challenges for object tracking tasks and places higher demands on the generalization capabilities of the models. In this article, we propose a novel quantum evolutionary learning tracker (QELT) for video. The model combines quantum evolution with deep networks for tracking video objects. The model uses a QELT to generate a reliable population of candidate regions and a deep network for classification. In particular, the quantum evolutionary predictor predicts the object motion state through rotation operator and trajectory inference, and provides motion state information for the tracker. The predictor can incorporate object history contextual information and can provide stable candidate estimation populations for the model in case of failure of appearance features. Both quantum evolution and deep networks are combined to form an end-to-end online video object tracker. In addition, we propose a new video object tracking evaluation algorithm, Balanced Intersection over Union. The evaluation algorithm uses aspect ratios to balance the share of overlap and distance. Finally, we test the model on the OTB 2015 dataset for natural video and on the SV248A10-SOT dataset for satellite video. The performance of the proposed model is also analyzed and validated by comparing it with more than 20 classical tracker models. The experimental results show that our model has high generalization ability and robustness.

Heterogeneous object-level change detection (CD) aims to detect the state of the objects and whether they have changed from multitemporal multimodal data. In this article, a new cross-modality interactive change detector (CICD) is proposed for object-level CD from multitemporal optical and synthetic aperture radar (SAR) images. The CICD consists of a backbone, cross-modal interactive module (CIM), neck, and head. CIM is designed to work with features extracted from modalities by the backbone network, enabling it to identify more changes in objects. Moreover, to address data imbalances in change categories caused by variations in satellite revisit cycles and aircraft flight plans, we introduce a heterogeneous class balanced module (HCBM). An eliminate adversarial network (EAN) is constructed as the main component of the HCBM. It is used to eliminate objects to augment images in which objects appear in only one temporal instant, thus reducing imbalances in the dataset. Extensive experiments are conducted on the multimodal object-level change dataset (MOCD), and the results show that CICD can achieve state-of-the-art performance.

Deep learning models (algorithms) have demonstrated their superior performance in interpreting Earth science and remote sensing data. However, adversarial examples generated with perturbations imperceptible to humans could render deep learning algorithms ineffective. This significant vulnerability of deep learning models, thus, inspires the exploration of defense methods resistible to adversarial examples. Although numerous countermeasures against adversarial examples have been proposed, the design of a universally applicable defense method across multiple scenarios still remains to be explored. In this study, we propose an effective denoising diffusion-based defense restore network (D3R-Net) based on the denoising diffusion model from the perspective of adversarial restoration, which transforms the adversarial examples into clean samples. Utilizing a highly effective denoising diffusion probabilistic model (DDPM), our D3R-Net transforms input adversarial examples into a state of noise, where diverse forms of adversarial noise transition into Gaussian noise. Subsequently, it captures semantic information through a series of iterative denoising steps. The pixel distribution of adversarial examples is restored in the proposed network to match the original distribution, enabling the classifier to identify adversarial examples correctly. Furthermore, we introduce a combined filtering module to preserve the semantic information of the original image, thereby further enhancing the defensive performance. Instead of modifying the model structure or excluding suspected samples, the proposed method restores the adversarial examples, making it simple yet effective and applicable to a broader range of scenarios. Extensive experiments are conducted on four benchmark datasets, and the results demonstrate that D3R-Net has significant defense capabilities against known and unknown attacks. Our source code is available athttps://github.com/SIM-xidian/D3R-Net.

Change detection involves analysis of discrepancies between two phases. However, when the unchanged elements are known, the changed features to be identified become straightforward. In addition, remote sensing image is constrained by limited spectral information, which leads to blurred boundaries between different semantics. Based on these two prior knowledge, in this artical, we introduce a novel change detection framework, named the mutually reinforcing similarity network (MutSimNet). This architecture aims to minimize false alarms along changing boundaries and reduce misjudgment rates among outliers. First, similarity learning is applied to change detection. The relationship between the two phases is considered when deriving the change feature maps. Second, we devise a mutually reinforcing loss function that integrates initial features with final features. Third, a self-attention module is connected in the feature pyramid network. This design mitigates information loss during the down-sampling process. Fourth, an attention feature fusion strategy is proposed for the integration of multi-layer features. This strategy takes into account the interaction between layer-by-layer features. Fifth, experimental results validate MutSimNet’s efficiency, particularly its ability to focus on edge contour learning. The MutSimNet also achieves superior performance on two benchmark datasets and predicts positive samples with higher probability. The codebase is accessible at https://github.com/ly-yu/MutSimNet.

Convolutional neural networks (CNNs) and self-attention (SA) are highly effective techniques used for the fusion of multisource remote sensing (RS) data, and they have found extensive application in Earth observation (EO) tasks. Nevertheless, CNNs are insufficient for the comprehensive extraction of contextual information and the representation of the sequential properties of spectral features. Furthermore, the loss of edge geometry information is often a consequence of information mining, which limits its application in RS. To address the abovementioned limitations, we propose a method called “multiscale geometric prior aware cross-modal network (MGPACNet)” for RS image fusion classification. First, a geometric prior feature enhanced residual module (GPFEResM) is created to extract shallow multiscale geometric edge prior features and detailed information from multimodal RS data to enhance feature boundary information. Second, a multiscale global-local spatial-spectral feature extraction module (MG-LS2FEM) uses multiscale spatial modeling and global-local spectral modeling to perceive rich semantic information in the spatial-spectral domain. Finally, a dual attention fusion module (DAFM) is designed to use pixel-level SA and cross-attention between heterogeneous data to achieve deep aggregation and cross-focusing of cross-modal information in two branches, and enhance the complementarity of heterogeneous data. A comprehensive examination of public RS data (hyperspectral-synthetic aperture radar (HS-SAR) Augsuburg/Berlin, hyperspectral-light detection and ranging (HS-LiDAR) Trento/MUUFL) from four distinct modalities (HS/SAR/LiDAR) has revealed that our method outperforms alternative models.

In recent years, change detection (CD) has emerged as an increasingly intricate research domain. However, in natural images, the orientation of objects is often aligned with the image boundaries, whereas in RS images, the imaging angles are random. As a result, existing CD methods encounter limitations when effectively representing vector features. In this article, we propose a rotation-invariant CD architecture named RFormer. It effectively utilizes direction-sensitive position embedding (DSPE) to represent features in RS images. To address the challenge of the quadratic growth in attention mechanism complexity with sequence length, we introduce low-cost cross attention (LC2A) to reduce its complexity to$1/{C^{2}}$. Furthermore, we employ the implicit timing extraction process (TEP) to represent interframe bitemporal features. TEP plays a crucial role in mitigating prediction biases caused by seasonal changes in land cover and prevents overconfident discrimination by the classifier in CD tasks. Experimental results demonstrate that RFormer achieves competitive performance on WHU, deeply supervised image fusion network (DSIFN)-CD, CDD, and LEVIR-CD datasets.

Remote sensing image captioning (RSIC) is an interesting but challenging cross-modal reasoning task for computer vision and natural language processing. Most of the recent popular approaches for RSIC utilize encoder-decoder architectures, which focus on visual features captured by convolutional neural network (CNN)-based encoder and semantic information by recurrent neural network (RNN)-based or long short-term memory (LSTM)-based decoder, but encounter difficulties with multiscale, multicategories, and direction ambiguity challenges. To make the most of semantic understanding ability of Transformers, in this article, we propose a new attention-based visual-linguistic reasoning framework with dual Transformer for RSIC. Specifically, Swin Transformer (SwinT) encoder with shifted window partitioning scheme is introduced for multiscale visual feature extraction to discover the intrinsic relationship in the objects, and then, a Transformer language model (TLM) with self-attention and cross attention is designed as the decoder to generate a well-formed sentence for the image. Extensive experiments are conducted on the public RSIC benchmark datasets, including UCM-Captions, Sydney-Captions, and RSICD. The impressive performance verifies the effectiveness and superiority of the proposed method. In addition, the source code and models of this work are publicly available athttps://github.com/LianYi233/TrTr-CMR.

Surrounding contexts are generally perceived as interfering with object tracking in satellite videos, leading to model drift. From another perspective, they can also be seen as reference objects of the tracked target, the dynamic interactions between them could provide essential information. In this article, a high-order relation learning transformer (HRLT) is proposed for satellite video object tracking, which not only models the high-order interactions of different target-context pairs but also reasons the associations between these high-order relations across multiple frames. First, a spatial high-order relation reasoning (SHR2) module is designed to model the high-order interactions between the target and scene contexts. Second, a temporal high-order relation reasoning (THR2) module is proposed to associate and reason these spatial high-order relations across multiple frames. Third, historical high-order relations are collected to provide more reasoning bases for the current frame prediction. Finally, qualitative and quantitative evaluations are performed on the SV248S, SkySat, and VISO datasets. The results show that HRLT outperforms 20 popular methods in different challenging scenarios.

In deep learning, convolutional neural networks (CNNs) and transformers have gained excellent achievements in remote sensing scene parsing. Strong feature representation ability is still a challenge for them. Besides, the complex scenes are still essential challenges for deep learning in remote sensing scene parsing. In this article, an efficient local–global lifting transformer (LGLFormer) framework is proposed to ease the challenges above. It effectively combines CNNs, transformer, and wavelet transform to build a strong local–global (LG) feature representation network. Besides, global feature learning driven by LG adaptive features is proposed based on the 2-D LG adaptive feature extractor (LGAFE) and refined global feature attention module. The 2-D LG lifting feature extractor is inspired by the lifting scheme, which introduces local and global dependency. Furthermore, two LG lifting schemes are proposed, including the series and parallel modes, which can effectively learn LG relations between pixels. Finally, experiments are validated on three remote sensing benchmark datasets. The proposed LGLFormer achieves the state-of-the-art with 99.02%, 99.2%, and 99.48% overall accuracy (OA) on AID, WHU-RS19, and UCM datasets, respectively. In addition, LGLFormer shows good convergence with competitive parameters. The experimental code will be available athttps://github.com/yutinyang/LGLFormer.

Tiny objects in satellite videos are usually not independent individuals, there exist rich semantic and temporal relations with each other. Thus, modeling and reasoning the variation of such intrinsic relationships can be beneficial for tiny object tracking. In this paper, a relation learning reasoning method is proposed for tiny object tracking in satellite videos. The core of the proposed is the relation reasoning network that consists of a key context module, a global semantic module, and a relation reasoning module sequentially. First, the key context module exploits global key contexts which explicitly or implicitly contribute to the target object, modeling the intrinsic relations with the target. Second, to reason the contribution, the global semantic module analyses the interaction between them in the same frame. Third, the relation reasoning module deduces the target based on the variation of the semantic relations among different frames. Such a relation learning reasoning approach which takes the target as the core is aligned with the satellite tiny object tracking task, significantly improves the identification performance in dense similarity scenes and the retrieval ability after completely occluded. Furthermore, the proposed method is shown to report improved qualitative and quantitative results on Jilin-1 and SkySat satellite video datasets.

Most existing polarimetric synthetic aperture radar (PolSAR) image classification methods are only valid under the assumption of identical imaging platforms and terrain categories for both training and test sets. To overcome this limitation, we propose a meta-graph representation learning (MGRL) method for PolSAR image classification with cross-platform and cross-category implementation. First, an integrated network is developed to learn the global-local representations of PolSAR images, which consists of a trumpet convolutional network (TCN) to learn the local scattering features of pixels and a graph convolutional network (GCN) for modeling the global structure of polarization information. Then, a comprehensive and transferable embedding of pixels is derived by collaborative optimization on multiple meta-learning tasks, which enables MGRL to recognize new classes not seen during training. Thus, the learned transferable representations can be quickly adapted to cross-platform and cross-category tasks with few labeled samples. Extensive experiments on several live airborne and spaceborne PolSAR datasets validate the effectiveness and advantages of MGRL over its counterparts.

Satellite video tracking tasks are often characterized by blurred foreground boundaries in vast scenes, a wide range of targets varying in scale, and irregular changes in appearance. These challenges significantly impact the optimization of robust tracker performance. Therefore, it is imperative to extract diverse features with dynamic adaptive learning capabilities for the target being tracked in each sequence. In this article, we explore a novel adaptive multi-scale Transformer (MT) tracker for satellite videos to explore the potential spatiotemporal information of the target effectively. Specifically, a multi-scale spatial Transformer (MSST) is designed to leverage stage-by-stage spatial reduction and channel doubling, thereby enhancing the representation capabilities for the tracked target. In dynamic feature learning, an adaptive temporal Transformer (ATT) is then introduced based on multiple cross attentions, which analyzes the adaptive learning capacity for the dynamic target. It analyzes the weight proportion of different attentions automatically in the specific sequence through the learnable parameters. Finally, a multi-scale feature (MSF) regression module is crafted to improve the positioning accuracy of targets with low pixel counts in satellite scenes. This module accomplishes precise annotation of target boxes by effectively fusing features from diverse stages. We evaluate the proposed tracker performance on several public satellite datasets, including SatSOT, SV248S, and VISO. Experimental results show that the performance of our model can be comparable to the state-of-the-art trackers.

Change detection (CD) based on deep learning has achieved excellent performance in recent years. However, these models exhibit limited capability in complete temporal modeling or face problems with fine-grained spatial features being overshadowed by the temporal context. Pure CNN-based CD pipelines also struggle to establish long-range connections. In this article, a globally aware continuous-time redistribution network (GCRNet) is proposed for RSCD. First, a boundary extraction branch is designed to preserve the semantic invariance of objects within the same boundary. This is achieved by providing boundary attention to adaptively guide the integration of temporal and spatial information. Then, a globally aware operator (GAO) is developed to obtain global interaction features. GAO utilizes the convolution theorem, which combines the Fourier transform and inverse Fourier transform, achieving it with low computational costs. Finally, an adaptive feature redistribution (AFR) module is designed to increase the distance between positive and negative samples in the latent space with change perception. It alleviates the effects of the severe class imbalance issue. Experimental results demonstrate that our proposed GCRNet surpasses 13 state-of-the-art CD methods. It achieves F1-score 0.33%, 0.62%, 0.84%, 0.17%, and 1.54% higher than the second-best model on the LEVIR-CD, LEVIR-CD+, WHU, CDD, and DSIFN datasets. The code of GCRNet is available athttps://github.com/XiaowenZhang-kuku/GCRNet.

Robust feature learning has always been a research hotspot in dynamic temporal tasks. It makes the model almost unaffected by some challenging properties. The sequential nature of the transformer means attractive for temporal learning tasks, making it perform well in the video field. It is a current research hotspot for learning effective features by utilizing the target motion trends in satellite videos with multiple attributes, such as similar objects (SOBs) interference and occlusion. In this article, a novel discriminative temporal learning transformer tracker (DTLTracker) is introduced to characterize the dynamic target information for satellite videos. A discriminative transformer (DT) is proposed to comprehensively explore the dynamic target features with multiple attention mechanisms. It focuses on the primary information of the search area, making the target more discriminative. A fast convergence (FC) filter is designed to accelerate the weights convergence in calculating the target correlation operation, thereby ensuring the efficiency of model learning. The effectiveness and convergence have been demonstrated for the proposed optimization method. Additionally, a motion prior correction (MPC) module is constructed to utilize temporal information for target tracklet prediction, assisting the tracker in predicting the correct target. Numerous experiments are performed on three satellite videos to verify the effectiveness and feasibility of the proposed DTLTracker. It shows robustness compared to the state-of-the-art trackers on some challenging properties.

Satellite imagery, due to its long-range imaging, brings with it a variety of scale-preferred tasks, such as the detection of tiny/small objects, making the precise localization and detection of small objects of interest a challenging task. In this article, we design a knowledge discovery network (KDN) to implement the renormalization group theory in terms of efficient feature extraction (FE). Renormalized connection (RC) on the KDN enables “synergistic focusing” of multiscale features. Based on our observations of KDN, we abstract a class of RCs with different connection strengths, called$n21$C, and generalize it to feature pyramid network (FPN)-based multibranch detectors. In a series of FPN experiments on the scale-preferred tasks, we found that the “divide-and-conquer” idea of FPN severely hampers the detector’s learning in the right direction due to the large number of large-scale negative samples and interference from background noise. Moreover, these negative samples cannot be eliminated by the focal loss function. The RCs extends the multilevel feature’s “divide-and-conquer” mechanism of the FPN-based detectors to a wide range of scale-preferred tasks, and enables synergistic effects of multilevel features on the specific learning goal. In addition, interference activations in two aspects are greatly reduced and the detector learns in a more correct direction. Extensive experiments of 17 well-designed detection architectures embedded with$n21$Cs on five different levels of scale-preferred tasks validate the effectiveness and efficiency of the RCs. Especially the simplest linear form of RC—E421C performs well in all tasks, and it satisfies the scaling property of renormalization group theory. All experiments can be trained and tested on a graphics card with 8 GB of video memory, which greatly enhances the applicability of our methodology. We hope that our approach will transfer a large number of well-designed detectors from the computer vision community to the remote sensing community. Datasets and codes will be available at:https://github.com/rabbitme/

Semi-supervised change detection (SSCD) has experienced rapid development, with numerous semi-supervised methods being proposed to reduce the reliance on labeled data in change detection. Existing approaches typically rely on manually set high-confidence thresholds to select robust pseudo-labels. However, the single-pixel threshold filtering method for pseudo-labels (STFP) lacks context correlation, cannot eliminate high-confidence false positive samples, and leads to erroneously filtering out low-confidence true positive samples. To address this issue, we propose robust instance-based semi-supervised learning change detection (RISL) for remote sensing images. RISL evaluates the reliability of each instance object by linking the semantic information of the context, thereby generating robust pseudo-labels. In RISL, firstly, a simple boundary trimming module (BT) as a preprocessing method for change prediction map is introduced. BT can effectively remove low-confidence false positive samples while avoiding confusion in the category of instance objects, thereby improving the quality of instance objects. Then, we propose a reliable instance evaluation module (RIEM) to evaluate the reliability of each instance object. RIEM combines the semantic information of the entire instance and establishes correlations between sample contexts to determine the reliability of the instance, effectively eliminating high false positive samples. In addition, the consistency regularization (CR) is integrated into RISL, and a new strategy suitable for RIEM is constructed. This strategy enhances the model’s generalization ability by mining and hiding semantic information from different views of unlabeled data. Experimental results on the challenging WHU-CD, LEVIR-CD, and CDD-CD datasets show that the proposed method achieves 89.80%, 90.01%, and 87.56% F1 scores on labeled data with 5% distribution. RISL achieves state-of-the-art performance compared to other methods.

With the maturity of big data and computing power, deep learning has provided an end-to-end efficient solution for fault diagnosis of rotating machinery. However, the diagnosis performance is commonly affected by complex working environment and limited labeled samples. While considering these undesirable effects and borrowing from multisource fusion techniques, we propose a novel fault diagnosis method based on cross-sensor correlative feature learning and fusion. First, global–local temporal encoder is utilized to learn the time-domain features of multiple sensor data. Meanwhile, time–frequency encoder is performed to obtain the corresponding time–frequency domain features. Then, features of the two modes are fused to get the initial results. Finally, they are put through cross-sensor correlative channel-aware fusion to achieve a final result. Furthermore, two datasets are selected to verify the effectiveness of the proposed method. The results demonstrate that our method is effective, robust, and suitable for diagnosis under limited data and complex conditions.

Modulation types clustering (MC) is crucial for adaptive high-frequency communication between devices in the Industrial Internet of Things. The strength of MC resides in its self-supervised framework, enabling it to extract modulation features efficiently without any manual labeling. However, the misalignment of proxy tasks and erroneous pseudolabeling constrain the performance of prevalent MC feature extraction techniques that utilize time series signals. In this article, we compare the saliency maps on time–frequency image (TFI) with that on time series signal, highlighting the consistency of TFI reconstruction with modulation feature extraction. Subsequently, in order to address the sensitivity of K-means to outliers, Sinkhorn–Knopp labeling (SKLb) is proposed to balance the scale of clusters and neighboring distances. Moreover, in consideration of the potential instability of the SKLb iteration result in backpropagation, the Sinkhorn–Knopp loss is proposed to ensure stable training of the model. Finally, two models, SK-IDC and SK-STDC, were tested on four datasets. Experimental results on these datasets present that our approach outperforms original signal representation and prevalent deep clustering methods, achieving State-of-the-Art performance.

Connections between visual components are ubiquitous. Graphs, as a highly flexible data structure, not only allow imposing relational induction bias on data, but can provide a completely distinct learning perspective for regular image data. In this paper, we propose a hierarchical dynamic graph clustering network (HDGCN) for visual feature learning. We construct hierarchical graph representations in graph domain in an adaptive, data-adaptive and task-adaptive manner. First, the initial graph is constructed in high-dimensional feature domain of images. To mine the hierarchical geometric features in latent graph space, adaptive clustering network (ClusterNet) is performed to learn discriminative clusters and generates cluster-based coarse graph. Then, graph convolutional networks (GCNs) are used to diffuse, transform and aggregate information among clusters. So, the intra-class and inter-class information is fully explored to increase the discriminativity of graph representations. Next, coarsened graph representations are mapped to grid based on its affinity with linear projection features. To further improve the task adaptation of clusters and hierarchical graph representations, ClusterNet and GCNs are fused in the same framework for end-to-end training and clusters is updated dynamically. We have conducted extensive experiments on classification and segmentation tasks. The experimental results fully validate the robustness of the proposed algorithm.

Multisource remote sensing image fusion classification aims to produce accurate pixel-level classification maps by combining complementary information from different sources of remote sensing data. Existing methods based on Convolutional Neural Networks (CNN-based) utilize a patch-based learning framework, which has a high computational cost, leading to poor real-time performance. In contrast, methods based on Fully Convolutional Networks (FCN-based) can process the entire image directly, achieving fast inference. However, FCN-based methods require high computational resources and exhibit shortcomings in feature fusion, hindering practical applications. In this paper, a lightweight FCN-based Progressive Hierarchical Fusion Network (PHFNet) is tailored for multisource remote sensing image classification. PHFNet comprises a pyramid dual-path encoder and a pyramid decoder. In the encoder, cross-source features are hierarchically fused via the adaptive modulation fusion module (AMF), which leverages style calibration for cross-source alignment and promotes the complementarity of the fusion feature. In the decoder, we introduced an improved convolutional gated recurrent unit (iConvGRU) to progressively integrate the semantic and detailed information of hierarchical features, producing a context-enhanced global representation. In addition, we consider the relation between the channel number, convolutional kernel size, and parameter count to make the model as lightweight as possible. Comprehensive evaluations on three multisource remote sensing datasets demonstrate that PHFNet improves overall accuracy by 1.5% to 2.8% with a low computational overhead compared to state-of-the-art methods. The source code is avaliable athttps://github.com/ShirlySmile/PHFNet.

Constructing effective proxy data is one of the core challenges in data-free knowledge distillation. The existing models ignore the influence of the category entanglement of the generated data on the distillation. To alleviate this issue, imitating the human learning process, a new category-aware curriculum learning mechanism is proposed in this paper to perform data-free knowledge distillation, called CCL-D. The main ideology of this category-aware curriculum learning mechanism is to provide a new learning mode for data generation and network training, which enables the model to realize the knowledge distillation process from easy to difficult through automated curriculum learning. In this novel learning mechanism, a category-aware monitoring module is proposed to constrain the category attribute of generated data. Based on this monitoring module, the curriculum learning process for data generation and network training is designed and applied. Initially, the generator is guided to obtain new data with clear category features. The utilization of data with apparent category features is easy for student network training, and it enables the student network to learn clear and significant category features at the early training stage. Subsequently, the generator is guided to generate data with category entanglement. Utilizing these new data with category entanglement problems can improve the recognition ability of the student network to interclass interference and enhance network robustness. The effectiveness of the CCL-D is verified on the six benchmark experimental datasets (MNIST, CIFAR-10, CIFAR-100, SVHN, Caltech-101, Tiny-Imagenet).

For accurate segmentation, effective feature extraction has always been a challenging problem, since the variability of appearance and the fuzziness of object boundaries. Convolutional neural networks have recently gained recognition in feature representation learning. However, it is only conducted in the spatial domain, and lacks effective representation of directionality, singularity and regularity in the spectral domain for anomaly detection of images. This is the key to feature learning representation of high-order singularity. To solve this problem, a multi-scale contourlet knowledge guide learning network is proposed in this paper. It is novel in this sense that, different from the CNNs in the spatial domain, the proposed method learns the multi-scale contourlet sparse representation to obtain more effective and sparse features in multi-scales and multi-directions. Furthermore, the contourlet knowledge guide learning can enhance the representation of spectral domain features. It is shown that the proposed network can learn the multi-level discriminative features and capture the more accurate object boundaries. The segmentation ability in theoretical analysis and experiments on five polyp segmentation datasets (CVC-ColonDB, CVC-ClinicDB, Kvasir-SEG, ETIS-LaribPolypDB, EndoSceneStill) and two building datasets (Massachusetts, WHU) are compared with developed methods. It must be emphasized that there is potential in effective feature learning representation and the generalization capability of the proposed method in deep learning, recognition and interpretation.

Inspired by the sparse and hierarchical features representation in the ventral stream of the human visual system, the biologically inspired multi-scale contourlet attention network (BMCAnet) is proposed to extract robust discriminative features. First, we constructed the multi-scale contourlet filter banks as a population of neurons in the primary visual cortex (V1), and extracted sparse features in a multi-scale and multi-direction way. It simulated a simple cell in V1 that responds to stimuli in a specific direction. Second, in order to refine contourlet features adaptively, the Shannon block attention module (SBAM) is introduced by integrating Shannon entropy as the third branch of the channel attention module (CAM), thus the weights of contourlet coefficients can be learned adaptively. Third, the responses of the spatial and spectral features are pooled by the proposed contourlet pooling layer to obtain the invariant structure features with the specified rules, which roughly stimulate the pooling process of complex cells in the V1 area. Last, the combination of global average pooling (GAP) and full connection (FC) is used for classification. The competitive results on eight databases demonstrate that the BMCAnet can effectively extract sparse and effective features for the classification tasks.

Real-world data typically follows a long-tailed distribution. When a small sample of tail classes does not cover the underlying distribution well, methods such as class re-balancing strategies and decoupled training are difficult to work, and additional knowledge needs to be introduced to recover the underlying distribution of the tail classes. In this work, we observe that the similarity between the variances of the feature distributions increases with the class similarity. Then, we also find that well-represented feature distributions typically contain multiple subcenters, which allows for denser samples at the edges of the distribution and promotes model learning to more robust decision bounds. Based on these observations, we propose to calibrate the feature distribution of the tail class by transferring the variance of the feature distribution of the head class, and then sample from the calibrated tail class distribution to generate augmented samples. To coordinate with the tail class calibration method, we also propose label-aware noise suppression (LANS) for reducing the generation of noisy samples and a three-stage training scheme for reshaping decision boundaries and compacting feature learning. Experimental results on iNaturalist2018, ImageNet-LT, CIFAR-10-LT, and CIFAR-100-LT show that our method achieves state-of-the-art performance in most metrics compared to similar approaches.

Modeling contextual relationships in images as graph inference is an interesting and promising research topic. However, existing approaches only perform graph modeling of entities, ignoring the intrinsic geometric features of images. To overcome this problem, a novel multiresolution interpretable contourlet graph network (MICGNet) is proposed in this article. MICGNet delicately balances graph representation learning with the multiscale and multidirectional features of images, where contourlet is used to capture the hyperplanar directional singularities of images and multilevel sparse contourlet coefficients are encoded into graph for further graph representation learning. This process provides interpretable theoretical support for optimizing the model structure. Specifically, first, the superpixel-based region graph is constructed. Then, the region graph is applied to code the nonsubsampled contourlet transform (NSCT) coefficients of the image, which are considered as node features. Considering the statistical properties of the NSCT coefficients, we calculate the node similarity, i.e., the adjacency matrix, using Mahalanobis distance. Next, graph convolutional networks (GCNs) are employed to further learn more abstract multilevel NSCT-enhanced graph representations. Finally, the learnable graph assignment matrix is designed to get the geometric association representations, which accomplish the assignment of graph representations to grid feature maps. We conduct comparative experiments on six publicly available datasets, and the experimental analysis shows that MICGNet is significantly more effective and efficient than other algorithms of recent years.

The feature representation learning process greatly determines the performance of networks in classification tasks. By combining multiscale geometric tools and networks, better representation and learning can be achieved. However, relatively fixed geometric features and multiscale structures are always used. In this article, we propose a more flexible framework called the multiscale dynamic curvelet scattering network (MSDCCN). This data-driven dynamic network is based on multiscale geometric prior knowledge. First, multiresolution scattering and multiscale curvelet features are efficiently aggregated in different levels. Then, these features can be reused in networks flexibly and dynamically, depending on the multiscale intervention flag. The initial value of this flag is based on the complexity assessment, and it is updated according to feature sparsity statistics on the pretrained model. With the multiscale dynamic reuse structure, the feature representation learning process can be improved in the following training process. Also, multistage fine-tuning can be performed to further improve the classification accuracy. Furthermore, a novel multiscale dynamic curvelet scattering module, which is more flexible, is developed to be further embedded into other networks. Extensive experimental results show that better classification accuracies can be achieved by MSDCCN. In addition, necessary evaluation experiments have been performed, including convergence analysis, insight analysis, and adaptability analysis.

Domain generalization (DG) is one of the critical issues for deep learning in unknown domains. How to effectively represent domain-invariant context (DIC) is a difficult problem that DG needs to solve. Transformers have shown the potential to learn generalized features, since the powerful ability to learn global context. In this article, a novel method named patch diversity Transformer (PDTrans) is proposed to improve the DG for scene segmentation by learning global multidomain semantic relations. Specifically, patch photometric perturbation (PPP) is proposed to improve the representation of multidomain in the global context information, which helps the Transformer learn the relationship between multiple domains. Besides, patch statistics perturbation (PSP) is proposed to model the feature statistics of patches under different domain shifts, which enables the model to encode domain-invariant semantic features and improve generalization. PPP and PSP can help to diversify the source domain at the patch level and feature level. PDTrans learns context across diverse patches and takes advantage of self-attention to improve DG. Extensive experiments demonstrate the tremendous performance advantages of the PDTrans over state-of-the-art DG methods.

Recently, the excellent performance of transformer has attracted the attention of the visual community. Visual transformer models usually reshape images into sequence format and encode them sequentially. However, it is difficult to explicitly represent the relative relationship in distance and direction of visual data with typical 2-D spatial structures. Also, the temporal motion properties of consecutive frames are hardly exploited when it comes to dynamic video tasks like tracking. Therefore, we propose a novel dynamic polar spatio-temporal encoding for video scenes. We use spiral functions in polar space to fully exploit the spatial dependences of distance and direction in real scenes. We then design a dynamic relative encoding mode for continuous frames to capture the continuous spatio-temporal motion characteristics among video frames. Finally, we construct a complex-former framework with the proposed encoding applied to video-tracking tasks, where the complex fusion mode (CFM) realizes the effective fusion of scenes and positions for consecutive frames. The theoretical analysis demonstrates the feasibility and effectiveness of our proposed method. The experimental results on multiple datasets validate that our method can improve tracker performance in various video scenarios.

The multispectral (MS) and the panchromatic (PAN) images belong to different modalities with specific advantageous properties. Therefore, there is a large representation gap between them. Moreover, the features extracted independently by the two branches belong to different feature spaces, which is not conducive to the subsequent collaborative classification. At the same time, different layers also have different representation capabilities for objects with large size differences. In order to dynamically and adaptively transfer the dominant attributes, reduce the gap between them, find the best shared layer representation, and fuse the features of different representation capabilities, this article proposes an adaptive migration collaborative network (AMC-Net) for multimodal remote-sensing (RS) images classification. First, for the input of the network, we combine principal component analysis (PCA) and nonsubsampled contourlet transformation (NSCT) to migrate the advantageous attributes of the PAN and the MS images to each other. This not only improves the quality of images themselves, but also increases the similarity between the two images, thereby reducing the representational gap between them and the pressure on the subsequent classification network. Second, for the interaction on the feature migrate branch, we design a feature progressive migration fusion unit (FPMF-Unit) based on the adaptive cross-stitch unit of correlation coefficient analysis (CCA), which can make the network automatically learn the features that need to be shared and migrated, aiming to find the best shared-layer representation for multifeature learning. And we design an adaptive layer fusion mechanism module (ALFM-Module), which can adaptively fuse features of different layers, aiming to clearly model the dependencies among multiple layers for different sized objects. Finally, for the output of the network, we add the calculation of the correlation coefficient to the loss function, which can make the network converge to the global optimum as much as possible. The experimental results indicate that AMC-Net can achieve competitive performance. And the code for the network framework is available at: https://github.com/ru-willow/A-AFM-ResNet.

For complex data, high dimension and high noise are challenging problems, and deep matrix factorization shows great potential in data dimensionality reduction. In this article, a novel robust and effective deep matrix factorization framework is proposed. This method constructs a dual-angle feature for single-modal gene data to improve the effectiveness and robustness, which can solve the problem of high-dimensional tumor classification. The proposed framework consists of three parts, deep matrix factorization, double-angle decomposition, and feature purification. First, a robust deep matrix factorization (RDMF) model is proposed in the feature learning, to enhance the classification stability and obtain better feature when faced with noisy data. Second, a double-angle feature (RDMF-DA) is designed by cascading the RDMF features with sparse features, which contains the more comprehensive information in gene data. Third, to avoid the influence of redundant genes on the representation ability, a gene selection method is proposed to purify the features by RDMF-DA, based on the principle of sparse representation (SR) and gene coexpression. Finally, the proposed algorithm is applied to the gene expression profiling datasets, and the performance of the algorithm is fully verified.

The emergence of neural architecture search (NAS) algorithms has removed the constraints on manually designed neural network architectures, so that neural network development no longer requires extensive professional knowledge, trial and error. However, the extremely high computational cost limits the development of NAS algorithms. In this article, in order to reduce computational costs and to improve the efficiency and effectiveness of evolutionary NAS (ENAS) is investigated. In this article, we present a fast ENAS framework for multiscale convolutional networks based on evolutionary knowledge transfer search (EKTS). This framework is novel, in that it combines global optimization methods with local optimization methods for search, and searches a multiscale network architecture. In this article, evolutionary computation is used as a global optimization algorithm with high robustness and wide applicability for searching neural architectures. At the same time, for fast search, we combine knowledge transfer and local fast learning to improve the search speed. In addition, we explore a multiscale gray-box structure. This gray box structure combines the Bandelet transform with convolution to improve network approximation, learning, and generalization. Finally, we compare the architectures with more than 40 different neural architectures, and the results confirmed its effectiveness.

Recently, contrastive learning (CL) has exhibited considerable advantages for automatic modulation classification (AMC) with a scarcity of labeled samples. Nevertheless, the majority of the available CL-based AMC methods use the simple signal augmentation strategy and suffer from interference from false negatives. To explore the more generalizable global temporal semantics within signals, a novel masked contrastive learning with hard negatives (MCLHN) method is proposed in this paper. MCLHN first strategically incorporates semantic-preserving data augmentation, ensuring the diversity and semantic invariance of signals. Second, MCLHN adopts an encoder with temporal masking to enable robust temporal modeling. Moreover, a debiased hardness-weighted contrastive (DHWC) loss is designed to balance the adverse impact of the debiased strategy and the advantage of hard negatives. Extensive experiments are conducted on several benchmark datasets, and the experimental results demonstrate the superior performance and generalization capability of MCLHN to other methods. Significantly, the performance of MCLHN with only one labeled sample per modulation under each signal-to-noise ratio (SNR) rivals that of other methods with five to twenty times the number of labeled samples.

To address the challenges of long-tailed classification, researchers have proposed several approaches to reduce model bias, most of which assume that classes with few samples are weak classes. However, recent studies have shown that tail classes are not always hard to learn, and model bias has been observed on sample-balanced datasets, suggesting the existence of other factors that affect model bias. In this work, we systematically propose a series of geometric measurements for perceptual manifolds in deep neural networks, and then explore the effect of the geometric characteristics of perceptual manifolds on classification difficulty and how learning shapes the geometric characteristics of perceptual manifolds. An unanticipated finding is that the correlation between the class accuracy and the separation degree of perceptual manifolds gradually decreases during training, while the negative correlation with the curvature gradually increases, implying that curvature imbalance leads to model bias. Therefore, we propose curvature regularization to facilitate the model to learn curvature-balanced and flatter perceptual manifolds. Evaluations on multiple long-tailed and non-long-tailed datasets show the excellent performance and exciting generality of our approach, especially in achieving significant performance improvements based on current state-of-the-art techniques. Our work opens up a geometric analysis perspective on model bias and reminds researchers to pay attention to model bias on non-long-tailed and even sample-balanced datasets. The code and model will be made public.

Automatic modulation classification (AMC) is a crucial stage in the spectrum management, signal monitoring, and control of wireless communication systems. The accurate classification of the modulation format plays a vital role in the subsequent decoding of the transmitted data. End-to-end deep learning methods have been recently applied to AMC, outperforming traditional feature engineering techniques. However, AMC still has limitations in low signal-to-noise ratio (SNR) environments. To address the drawback, we propose a novel AMC-Net that improves recognition by denoising the input signal in the frequency domain while performing multi-scale and effective feature extraction. Experiments on two representative datasets demonstrate that our model performs better in efficiency and effectiveness than the most current methods.

In scenarios with long-tailed distributions, the model's ability to identify tail classes is limited due to the under-representation of tail samples. Class rebalancing, information augmentation, and other techniques have been proposed to facilitate models to learn the potential distribution of tail classes. The disadvantage is that these methods generally pursue models with balanced class accuracy on the data manifold, while ignoring the ability of the model to resist interference. By constructing noisy data manifold, we found that the robustness of models trained on unbalanced data has a long-tail phenomenon. That is, even if the class accuracy is balanced on the data domain, it still has bias on the noisy data manifold. However, existing methods cannot effectively mitigate the above phenomenon, which makes the model vulnerable in long-tailed scenarios. In this work, we propose an Orthogonal Uncertainty Representation (hOUR) of feature embedding and an end-to-end training strategy to improve the long-tail phenomenon of model robustness. As a general enhancement tool, OUR has excellent compatibility with other methods and does not require additional data generation, ensuring fast and efficient training. Comprehensive evaluations on long-tailed datasets show that our method significantly improves the long-tail phenomenon of robustness, bringing consistent performance gains to other long-tailed learning methods.

Precise crop classification plays a significant role in the agriculture field. An appropriate data source for precise crop classification is high spatial resolutions hyperspectral imagery (H2 imagery) acquired by unmanned aerial vehicle (UAV). However, for imagery with many different classes of crops, crop classification of UAV H2 imagery is a huge challenge. The significant spectral diversity, spatial heterogeneity and nonlinear data structure of UAV H2 imagery results in poor spectral discriminability. To improve the discriminability, a kernel tensor slice sparse coding-based classifier (KTSSCC) is proposed for precise crop classification of UAV H2 imagery in this research. The kernel tensor representation mechanism in KTSSCC can reduce the nonlinear separation while well preserving the spectral characteristics and spatial constraints of land-covers, and thus the discriminability is greatly improved. Furthermore, this paper puts forward the kernel tensor slice sparse orthogonal matching pursuit (KTSSOMP) algorithm to optimize kernel tensor slice sparse coding in the spectral space, which greatly reduces the computation cost. Moreover, there are very few parameters to be tuned in our proposed model. We assess the performance of KTSSCC on two real UAV hyperspectral imagery datasets, and find that, based on visual and quantitative results, it provides satisfactory crop classification results and outperforms the state-of-the-art approaches.

Specific emitter identification (SEI) is crucial for attacking and defending Internet of Things (IoT) devices in untrusted scenarios or battlefield environments. However, existing SEI methods usually require annotation information, which is often unavailable in noncooperative communications and untrusted scenarios. In this article, we propose a signal contrastive self-supervised clustering (SCSC) method for unsupervised SEI applications. First, we propose SCSC with 1-D fingerprint pyramid feature extractor (1D-FPFE) for obtaining hierarchical subtle features of emitter signals. Then, we propose a bit-pulse selection (BPS) strategy and several signal data augmentation methods. By constructing signal positive and negative instance pairs through data augmentation, our approach generates cluster preference representations in a contrastive self-supervised learning manner. Extensive experimental results based on communication burst emitter dataset show that SCSC achieves an accuracy improvement of about 26% over the current best communication signal clustering algorithm. Moreover, SCSC also exhibits good performance and generalization for 30 emitter clustering and few-shot unlabeled signal clustering.

Internet of Things (IoT) networks are often subject to many malicious attacks in untrusted environments, and automatic modulation classification (AMC) is an effective way to combat IoT physical-layer threats. However, most existing AMC methods assume sufficient labeled signals and invariant signal distribution, which is often impossible in untrusted environments. In this article, a new meta-learning method is proposed for a few-shot AMC with distribution bias. First, a multi-frequency octave ResNet (MFOR) is constructed to learn coarse (low-frequency) and fine (high-frequency) features, which can efficiently identify the modulation type of the signal while saving computational resources. Second, a large number of classification-related meta-tasks are established for training MFOR to explore general knowledge in signal classification, and then transfer it to the AMC. Different with deep neural networks (DNNs) that learn a mapping by multiple instances, the MFOR with meta-learning (denoted as M-MFOR) can improve the generalization ability of new AMC tasks with very few instances and distribution bias. Furthermore, we find that the distribution bias between data can be reduced by adjusting the normalized distribution and propose a class-related mixup. Extensive experiments are taken on several datasets to investigate the effectiveness of M-MFOR. The results show its feasibility and superiority over existing methods.

Most existing classification methods design complicated and large deep neural network (DNN) model to deal with the ubiquitous spectral variability and nonlinearity of hyperspectral images (HSIs). However, their application is blocked by limited training samples and considerable computational costs in real scenes. To solve these problems, we propose a simple spectral hierarchical feature fusion and selection network (HFFSNet). Specifically, we apply 1-D grouped convolution for dimensionality reduction and multilevel feature extraction, then the multilevel features are fused to assist the adaptive feature selection of different layer features via the soft attention mechanism, and finally the selected features are fused to further enhance the feature representation. Extensive experimental results on three hyperspectral datasets demonstrate the effectiveness of the proposed network.

In recent years, the joint classification approach of hyperspectral image (HSI) and light detection and ranging (LiDAR) data based on deep learning (DL) has received increasing attention. However, existing methods either lack interaction between heterogeneous features during feature extraction or treat them equally during the interaction, inevitably resulting in redundant information stacking and reaching the performance bottleneck. To this end, we propose a novel discrepant bi-directional interaction fusion network (DBIFNet) for the collaborative classification of HSI and LiDAR data. First, a discrepant bi-directional interaction module (DBDIM) is designed to establish correlations between heterogeneous features to enhance the respective feature learning. Furthermore, a cross-modal attention fusion module (CAFM) is developed to dynamically fuse multi-modal features, which can further improve classification performance. Extensive experiments on the Houston and Trento datasets demonstrate that the proposed DBIFNet can achieve competitive classification performance.

In this letter, a kernel tensor sparse coding model (KTSCM) is proposed for precise crop classification of unmanned aerial vehicle (UAV) hyperspectral image (HSI). Benefited from the kernel tensor representation mechanism in KTSCM, which can not only improve the linear separation but also well preserving the spatial-spectral structures of land-covers, the discriminability of UAV HSI is greatly improved. The L1-norm based tensor sparsity makes the tensor operation in KTSCM can be equivalently converted to matrix operation, which greatly reduces the computation cost. Furthermore, the analytical solution to KTSCM allows it be well optimized with very few iterations. The performance of KTSCM is assessed on two real UAV HSIs. The experimental results indicate that KTSCM can provides rapid and accurate crop classification results with limited labeled pixels and outperforms the related counterparts.

Global average pooling (GAP) plays an important role in traditional channel attention. However, there is the disadvantage of insufficient information to use the result of GAP as the channel scalar. At the same time, the existing spatial attention models focus on the areas of interest using average pooling or convolutional networks, but there is a loss of feature information and neglect of the structural feature. In this paper, dual wavelet attention is proposed, which can effectively alleviate the aforementioned problems and enhance the representation ability of CNNs. Firstly, the equivalence between the sum of the low-frequency subband coefficients of 2D DWT (Haar) and GAP is proved. On this basis, the statistical characteristics of low-frequency and high-frequency subbands are effectively combined to obtain the channel scalars, which can better measure the importance of each channel. In addition, 2D DWT can effectively capture the approximate and detailed structural features. Thus, wavelet spatial attention is proposed, which can effectively focus on the key spatial structural features. Different from traditional spatial attention, it can better curve the structural and spatial attention for different channels. The experiments are verified on four natural image data sets and three remote sensing scene classification data sets, which shows the effectiveness and versatility of the proposed methods. The code of this paper will be available athttps://github.com/yutinyang/DWAN.

During target tracking process, the state of the target is usually unpredictable. In theory, it is often beneficial to automatically assign suitable features to describe the specific target in each frame. Inspired by this, in this paper, we propose a novel dynamic feature-adaptive tracking framework (DFAT) which automatically assigns appropriate features to the consecutive frames during tracking process to boost the tracking performance. To implement DFAT, a large pool consisting of trackers/experts based on correlation filtering (CF) is constructed which is called candidate pool (CandPool). The diversity of the experts lies in their feature configurations and we call them candidate experts (CandExp). In this way, different features can be assigned for continuously changed scenarios and the target. Then to assign suitable experts, for each frame, we design the dynamic tracking process as the following three steps: (1) Several experts which are called executive experts (ExeExp) are selected from the CandPool according to CandExps’ past performance. (2) The ExeExps generate the tracking results and the performance of them are evaluated via a novel evaluation mechanism. (3) The selection rate of each CandExp in the CandPool is updated according to the performance evaluation and the final tracking result is selected. To better evaluate the CandExp, we propose two novel criteria: (1) content similarity weighted intra-evaluation, and (2) response confidence based self-evaluation. Compared with traditional post-event ensemble trackers that use fixed experts, the proposed method learns to dynamically assign appropriate ExeExps selected from a large CandPool which leads to adaption to different cases. Moreover, overfitting caused by fixed experts can also be mitigated via dynamic tracking. Experiments on both public available general and satellite videos based data sets demonstrate the superiority of the proposed method.

With the increasing accessibility of remote sensing videos, remote sensing tracking is gradually becoming a hot issue. However, accurately detecting and tracking in complex remote sensing scenes is still a challenge. In this article, we propose a collaborative learning tracking network for remote sensing videos, including a consistent receptive field parallel fusion module (CRFPF), dual-branch spatial-channel co-attention (DSCA) module, and geometric constraint retrack strategy (GCRT). Considering the small-size objects of remote sensing scenes are difficult for general forward networks to extract effective features, we propose a CRFPF-module to establish parallel branches with consistent receptive fields to separately extract from shallow to deep features and then fuse hierarchical features adaptively. Since the objects and their background are difficult to distinguish, the proposed DSCA-module uses the spatial-channel co-attention mechanism to collaboratively learn the relevant information, which enhances the saliency of the objects and regresses to precise bounding boxes. Considering the interference of similar objects, we designed a GCRT-strategy to judge whether there is a false detection through the estimated motion trajectory and then recover the correct object by weakening the feature response of interference. The experimental results and theoretical analysis on multiple datasets demonstrate our proposed method's feasibility and effectiveness. Code and net are available at https://github.com/Dawn5786/CoCRF-TrackNet.

Recently, multi-modal remote sensing image (MRSI) classification has attracted increasing attention of researchers. However, classification of MRSI with limited labeled instances is still a challenging task. In this paper, a novel self-supervised cross-modal contrastive learning method is proposed for MRSI classification. Joint intra- and cross-modal contrastive learning are used to better mine multi-modal feature representations during pre-training, and the intra- and cross-modal contrastive learning objectives are jointly optimized, whereby it encourages the learned representation to be semantically consistent within and between modalities simultaneously. Moreover, a simple but effective hybrid cross-modal fusion module (HCFM) is designed in the fine-tuning stage, which could better compactly integrate complementary information across these modalities for more accurate classification. Extensive experiments are taken on four benchmark datasets (i.e., Houston 2013, Augsburg, Trento, and Berlin), and the results show that the proposed method outperforms state-of-the-art methods.

Multisource remote sensing images provide complementary multidimensional information for reliable and accurate classification. However, gaps in imaging mechanisms result in heterogeneity between multiple source images. During fusion, this heterogeneity causes the generated multisource representations may be redundant and ignore discriminative uni-source information, which significantly hampers the fusion classification performance. To address this challenge, we introduce a novel multisource joint representation learning method for remote sensing image fusion classification, termed Multisource Information Bottleneck Fusion Network (MIBF-Net). Based on the Information Bottleneck principle, MIBF-Net employs mutual information constraints to effectively integrate multisource information, generating a comprehensive and non-redundant multisource representation. Specifically, MIBF-Net first introduces an attribution-driven noise adaptation layer to dynamically balance the speed of feature learning across sources for extracting discriminative uni-source intrinsic information. Furthermore, a cross-source relationship encoding module is designed to fully explore cross-source complex dependencies for enhancing the richness of fused representations. Finally, we design an information bottleneck fusion module to fuse uni-source semantic information and cross-source information while reducing redundancy. In particular, we employ variational inference techniques to effectively address the mutual information optimization problem and provide theoretical derivations. Extensive experimental results on three heterogeneous multisource remote sensing data benchmarks show that the model significantly outperforms the state-of-the-art methods.

Supervised remote sensing (RS) image segmentation has achieved remarkable success with large amounts of manually labeled data, which may be difficult to acquire in some practical application scenarios. Semisupervised RS image segmentation can efficiently utilize the knowledge embedded in unlabeled data to improve recognition performance, which is of great significance for the generalization application of segmentation models. In this work, we propose an end-to-end semisupervised RS image segmentation method based on weak-to-strong consistency learning, denoted as WSCL. Specifically, a common strong data augmentation technique for image segmentation is introduced to provide powerful input perturbation to decouple self-biased cognition. By forcing weakly augmented, and strongly augmented perspectives from the same sample to be consistent, WSCL not only enables the model to steadily learn knowledge contained in unlabeled data but also alleviates overfitting. In addition, a novel sparse dual-view cross-sample image generation method is presented to generate new training samples, which helps provide a more comprehensive diversity of perturbations. Furthermore, an adaptive re-weighting strategy based on the entropy maps of the outputs of strongly perturbed samples is proposed to suppress noise, guiding the training process in a positive direction. Extensive experiments demonstrate the significant advantage of WSCL over other advanced methods, achieving new state-of-the-art under several evaluation metrics on DFC22, iSAID, MER, MSL, Vaihingen, and GID-15 datasets. The source code is open-sourced at https://github.com/xiaoqiang-lu/WSCL.

The complementary and heterogeneous properties fusion of multimodal data (such as hyperspectral, lidar, and synthetic aperture radar data) can significantly improve the accuracy of remote sensing (RS) images joint classification. Thus, we propose a spatial-spectral bilinear representation fusion network (S2BRFNet), which captures long-range dependencies cross-modality and within the same modality to achieve the final joint classification. Firstly, a cross-modal spatial-spectral representation module (S2RM) is designed, it utilizes spatial-spectral attention and self-attention between heterogeneous data to enhance the characterization capabilities of cross-modal complementary properties and spatial-spectral features of single-source data. Secondly, a semantic space-guided bilinear feature fusion module (S2BFM) is developed, which uses deep and shallow features to regain fine-grained features. It uses shallow location details to improve the semantic prediction of deep features. Furthermore, it uses the different representation capabilities of different layers for objects with obvious feature differences to enhance the feature advantages. Therefore, rich global context information is obtained. Finally, the semantic space re-weight strategy is used to guide the outer product fusion of heterogeneous features, which enhances the ability of the network to identify similar features. Classification experiments are carried out on four common datasets of different modality combinations (HS-SAR-DSM Augsburg, Berlin, Trento, and Muufl), and this can prove the superiority of the S2BRFNet.

Deep Learning-based (DL) methods have dominated the task of semantic segmentation of remote sensing images. However, the sizes of different objects vary widely, and there is a great deal of label-noise due to the inevitable shadows. Therefore, there is an urgent need for a method that can precisely handle complex ground data. In this paper, we propose an Inter-Class Enhanced Network (ICEN) for representing features of varying sizes. It comprises two branches: Sparse Representation Network (SPN) and Feature Extraction Network (FEN). Then, a Class-Perception Block is inserted between the two branches to instruct the SPN’s low-level semantic features to be merged into the deeper network. Such a block can reduce label-noise in remote sensing image segmentation. In addition, the proposed EIRI provides a more precise classification process for target edges containing many misclassified points without requiring excessive computational overhead. The experimental results of our proposed Class-Perception Network (C-PNet) achieve competitive performance on the Vaihingen, Potsdam, LoveDA, and UAVid datasets.

Most popular visual trackers for natural scenarios always adopt handcraft features or deep features to track the target in a video. However, they face with difficulties in discriminative feature representation and usually suffer from severe model drift for satellite videos, especially when encountering challenges of dim and small targets, low contrast or similar target interference. To overcome these difficulties, we propose a Contourlet-based Siamese Learning Tracker (CSLT), which mainly aims at tracking dim and small objects in satellite videos. In contrast to conventional methods, the contourlet transform enriches directional multi-resolution information which is crucial to discriminative feature representation for dim and small targets in satellite video frames that lack distinguishable appearance features. We jointly use multi-resolution features with deep features by spatial-attention fusion strategy and then track the targets by a Siamese structure network. To further improve the accuracy and robustness, a model drift alarm and calibration module, including translation drifting penalty and rotation drifting penalty, is employed during tracking. We conduct extensive comparisons with 16 popular state-of-the-art trackers on three satellite video datasets. The experimental results validate the effectiveness of the proposed tracker.

Contrastive self-supervised learning (CSSL) is a promising method in extracting effective features from unlabeled data. It performs well in image-level tasks, such as image classification and retrieval. However, the existing CSSL methods are not suitable for pixel-level tasks, e.g., change detection (CD), since they ignore the correlation between local patches or pixels. In this paper, we firstly propose a multi-cue contrastive self-supervised learning (MC-CSSL) method to derive dense features for change detection. Besides data augmentation, the MC-CSSL takes advantage of more cues based on the semantic meaning and temporal correlation of local patches. Specially, the positive pair is built from local patches with the similar semantic meaning or temporal ones with the same geographic location. The assumption is that local patches belonging to the same kind of land-covering tend to share similar features. Secondly, the affinity matrix is truncated and introduced to extract change information between two temporal patches obtained from different types of sensors. As a result, some initial unchanged pixels are selected to serve as the supervision for mapping the dense features into a consistent space. Based on the distance between all bi-temporal pixels in the consistent space, a difference image (DI) is generated and more unchanged pixels can be available. The dense feature mapping and unchanged pixel updating proceed alternately. The proposed CD method is evaluated in both homogeneous and heterogeneous cases and the experimental results demonstrate its effectiveness and priority after comparison with some existing state-of-the-art methods. The source code will be available at https://github.com/Yang202308/ChangeDetection_CSSL.

Deep convolutional neural networks (CNNs) are significant in remote sensing. Due to the strong local representation learning ability, CNNs have excellent performance in remote sensing scene classification. However, CNNs focus on location-sensitive representations in the spatial domain and lack contextual information mining capabilities. Meanwhile, remote sensing scene classification still faces challenges, such as complex scenes and significant differences in target sizes. To address the problems and challenges above, more robust feature representation learning networks are necessary. In this paper, a novel and explainable spatial-frequency multi-scale Transformer framework, SF-MSFormer, is proposed for remote sensing scene classification. It mainly comprises spatial-domain and frequency-domain multi-scale Transformer branches, which consider the spatial-frequency global multi-scale representation features. Besides, the texture-enhanced encoder is designed in the frequency-domain multi-scale Transformer branch, which is adaptive to capture the global texture features. In addition, an adaptive feature aggregation module is designed to integrate the spatial-frequency multi-scale feature for final recognition. The experimental results verify the effectiveness of SF-MSFormer and show better convergence. It achieves state-of-the-art results (98.72%, 98.6%, 99.72%, and 94.83% overall accuracies, respectively) on the AID, UCM, WHU-RS19, and NWPU-RESISC45 datasets. Besides, the feature visualizations evaluate the explainability of the texture-enhanced encoder. The code implementation of this article will be available at https://github.com/yutinyang/SF-MSFormer.

For remote sensing image segmentation, the boundaries of objects are difficult to distinguish, which is ignored by most methods. Therefore, it is challenging how to excavate and recover the boundaries of objects accurately. In this article, we propose a boundary-aware multi-scale network (BMNet) to solve this problem. The key components of BMNet include the scale attention module (SA-module) and boundary guidance module (BG-module). Specifically, SA-module is proposed to guide the refinement of multi-scale features in a context-aware way. It enhances the discriminability of multi-scale features by establishing contextual dependencies, which enables the refinement of the prediction of objects. Then, BG-module is proposed to enable networks to distinguish the boundary of objects. It utilizes manifold information of features to generate boundary guidance maps and forces the network to focus more on the boundary of objects. The effectiveness of the proposed BMNet is demonstrated on two public remote sensing datasets: ISPRS 2-D semantic labeling Potsdam dataset and Vaihingen dataset, where BMNet achieves better segmentation than prevalent methods. Finally, the experimental results indicate that BMNet can produce sharper boundaries of objects to reconstruct more detailed segmentation results.

Convolutional neural networks (CNNs) have superior feature learning capabilities with large numbers of labeled samples. The reality is that labeling these samples is costly in terms of human labor. Existing data augmentation methods alleviate the scarcity of labeled samples. However, these methods are not suitable for synthetic aperture radar (SAR) images, owing to special imaging mechanisms and observational objects. The generative SAR images by existing augmented methods show structure distortion. To address this issue, we introduce a curvelet adversarial augmented neural network (CA2NN) for SAR image classification. Specifically, an$\text{A}^{2}$NN is established, which consists of two generative streams and one discriminative stream. In the generative stream, through the mutual transformation between the whole and partial images, more new samples with structural consistency are generated to augment the limited labeled data. In the discriminative stream, these generated samples show certain appearance variations after adversarial training based on the novel joint discriminant criterion. Simultaneously, given the multiscale and multidirectional nature of SAR images, we construct discretized curvelet in 2-D space, aiming to extract the singularity features and avoid overfitting. By integrating curvelet kernels into$\text{A}^{2}$NN, CA2NN can automatically generate more representative features adapting to complex terrain, while greatly reducing the complexity of the network. Experiments are conducted on the SAR images with large-scale and complex scenes, suggesting that the proposed approach significantly improves the classification performance with few labeled samples.

Recently, Hyperspectral Image Classification (HIC) with noisy labels is attracting increasing interest. However, existing methods usually neglect to explore feature-dependent knowledge to reduce label noise, and thus perform poorly when the noise ratio is high or the clean samples are limited. In this paper, a novel Triple Contrastive Representation Learning (TCRL) framework is proposed from a deep clustering perspective for robust HIC with noisy labels. The TCRL explores the cluster-level, instance-level, and structure-level representation of HIC by defining triple learning loss. First, the strong and weak transformation are defined for hyperspectral data augmentation. Then, a simple yet effective lightweight Spectral Prior Attention-based Network (SPAN) is presented for spatial-spectral feature extraction of all augmented samples. Additionally, cluster-level and instance-level contrastive learning are performed on two projection subspaces for clustering and distinguishing samples respectively. Meanwhile, structure-level representation learning is employed to maximize the consistency of data after different projections. Taking the feature-dependent information learned by triple representation learning, our proposed end-to-end TCRL can effectively alleviate the overfitting of classifier to noisy labels. Extensive experiments have been taken on three public datasets with various noise ratios and two types of noise. The results show that the proposed TCRL could provide more robust classification results when training on noisy datasets compared with state-of-the-art methods, especially when clean samples are limited. The code will be available at https://github.com/Zhangxy1999.

Deep representation learning has improved automatic remote change detection (RSCD) in recent years. Existing methods emphasize primarily convolutional neural networks (CNNs) or Transformer-based networks. However, most of them neither effectively combine CNNs and Transformer nor use prior geometric information to refine regions. In this paper, a novel geometric representation Transformer (GeoFormer) is proposed for high-resolution RSCD. GeoFormer utilizes convolutional information to guide the Transformer by employing geometric prior knowledge. Specifically, the proposed GeoFormer consists of three carefully designed components: the geometric-based Swin Transformer (Geo-Swin Transformer) encoder, the Laplace attention fusion (LAFusion) module, and the UNet++CD decoder. Firstly, Geo-Swin Transformer is a novel designed non-local Siamese encoder that combines geometric convolution with Transformer to provide local geometric representation information for remote contextual features. Then, a LAFusion module is proposed to achieve robust bi-temporal feature fusion, which is founded on attention mechanism and edge information. Finally, UNet++CD decodes fine-grained information from the fused features by dense multiscale upsampling process. Experimental results demonstrate that the proposed GeoFormer performs better than benchmark methods on four change detection datasets (LEVIR-CD, WHU-CD, DSIFN-CD, and CDD) and is able to detect the edges of change regions more precisely. Our code is available at https://github.com/Jiaxzhao/GeoFormer.

In view of the good correlation measurement ability of correntropy, in this article, we propose a temporal local correntropy representation (TLCE) method based on the local correntropy matrix for fault diagnosis of machines. In TLCE, a sample is divided into several segments, and then, the correlation between these segments is expressed by correntropy. Finally, the correntropy matrix composed of the correntropy is regarded as the feature of each sample. The proposed TLCE model is validated by experiments of three bearing datasets and one gear dataset. And results demonstrate that compared with other methods, TLCE has obvious advantages, such as effectiveness and robustness.

In this paper, an Adaptive Fusion Transformer (AFT) is proposed for unsupervised pixel-level fusion of visible and infrared images. Different from the existing convolutional networks, transformer is adopted to model the relationship of multi-modality images and explore cross-modal interactions in AFT. The encoder of AFT uses a Multi-Head Self-attention (MSA) module and Feed Forward (FF) network for feature extraction. Then, a Multi-head Self-Fusion (MSF) module is designed for the adaptive perceptual fusion of the features. By sequentially stacking the MSF, MSA, and FF, a fusion decoder is constructed to gradually locate complementary features for recovering informative images. In addition, a structure-preserving loss is defined to enhance the visual quality of fused images. Extensive experiments are conducted on several datasets to compare our proposed AFT method with 21 popular approaches. The results show that AFT has state-of-the-art performance in both quantitative metrics and visual perception.

Multimodal Image fusion is becoming urgent in multi-sensor information utilization. However, existing end-to-end image fusion frameworks ignore a priori knowledge integration and long-distance dependencies across domains, which brings challenges to the network convergence and global image perception in complex scenes. In this paper, a conditional generative adversarial network with transformer (TCGAN) is proposed for multimodal image fusion. The generator is to generate a fused image with the source images content. The discriminators are adopted to distinguish the differences between the fused image and the source images. Adversarial training makes the final fused image to maintain the structural and textural details in the cross-modal images simultaneously. In particular, a wavelet fusion module makes the inputs contain image content from different domains as much as possible. The extracted convolutional features interact in the multiscale cross-modal transformer fusion module to fully complement the associated information. It makes the generator to focus on both local and global context. TCGAN fully considers the training efficiency of the adversarial process and the integrated retention of redundant information. Various experimental results of TCGAN have highlighted targets, rich details, and fast convergence properties on public datasets.

Feature representation has received more and more attention in image classification. Existing methods always directly extract features via convolutional neural networks (CNNs). Recent studies have shown the potential of CNNs when dealing with images' edges and textures, and some methods have been explored to further improve the representation process of CNNs. In this article, we propose a novel classification framework called the multiscale curvelet scattering network (MSCCN). Using the multiscale curvelet-scattering module (CCM), image features can be effectively represented. There are two parts in MSCCN, which are the multiresolution scattering process and the multiscale curvelet module. According to multiscale geometric analysis, curvelet features are utilized to improve the scattering process with more effective multiscale directional information. Specifically, the scattering process and curvelet features are effectively formulated into a unified optimization structure, with features from different scale levels being efficiently aggregated and learned. Furthermore, a one-level CCM, which can essentially improve the quality of feature representation, is constructed to be embedded into other existing networks. Extensive experimental results illustrate that MSCCN achieves better classification accuracy when compared with state-of-the-art techniques. Eventually, the convergence, insight, and adaptability are evaluated by calculating the trend of loss function's values, visualizing some feature maps, and performing generalization analysis.

The fusion of hyperspectral image (HSI) and multispectral image (MSI) refers to enhance the spatial resolution of HSI with the help of a corresponding MSI that has a high spatial resolution to finally obtain an HSI with high resolution in both spatial and spectral domains. In this letter, we propose a variational tensor subspace decomposition-based fusion method to fully explore the differences and correlations among three modes of the HSI tensor. Experimental results on two HSI datasets show that the proposed method can achieve superior performance compared with existing state-of-the-art fusion methods with high computational efficiency.

With the development of the imaging technology of various sensors, multisource image classification has become a key challenge in the field of image interpretation. In this article, a novel classification method, called the deep multiview union learning network (DMULN), is proposed to classify multisensor data. First, an associated feature extractor is designed to process the multisource data by canonical correlation analysis (CCA) in the head of the network. Second, an improved deep learning architecture with two branches is presented to extract high-level view features from the associated features. Third, a novel pooling, called view union pooling, is proposed to fuse the multiview feature from the deep model. Finally, the fused feature is fed into the classifier. The proposed framework is easy to optimize since it is an end-to-end network. Extensive experiments and analysis on the datasets IEEE_grss_dfc_2017 and IEEE_grss_dfc_2018 show that the proposed method achieves comparable results. Our results demonstrate that abundant multisource information can improve the classification performance.

Panchromatic (PAN) and multispectral (MS) images have coordinated and paired spatial spectral information, which can complement each other and make up for their shortcomings for image interpretation. In this article, a novel classification method called the deep group spatial-spectral attention fusion network is proposed for PAN and MS images. First, the MS image is processed by unpooling to obtain the same resolution as that of the PAN image. Second, the group spatial attention and group spectral attention modules are proposed to extract image features. The PAN and the processed MS images are regarded as the input of the two modules, respectively. Third, the features from the previous step are fused by the attention fusion module, which aims to fully fuse multilevel features, take into account both the low-level features and the high-level features, and maintain the global abstract and local detailed information of the pixels. Finally, the fusion feature is fed into the classifier and the resulting map is obtained by pixel level. Extensive experiments and analysis on four datasets show that the proposed method achieves comparable results.

The excellent performance of graph convolutional networks (GCNs) on non-Euclidean data has drawn widespread attention from the hyperspectral image classification (HSIC) community, where the predefined graph (including node modeling and adjacency matrix calculation) plays a key role. However, existing GCN-based methods rely on manual efforts in constructing and updating graphs, and the superpixel-based node features lack high-level semantics. In this article, we propose an automatic graph learning convolutional network (Auto-GCN), which unifies the graph learning and HSIC in a “network-in-network” manner. Specifically, the graph is employed to model the interaction of the high-order tensors. Considering the powerful learning and representation capabilities of convolutional neural networks (CNNs), the semisupervised Siamese network (SiamNet) is embedded into GCNs and HSIC networks to accomplish the automatic learning and dynamic updating of the graph. GCNs further encode and infer the dynamic graph, and then, the learnable graph reprojection matrix is designed to assign graph representations to pixels. The dynamic graph serves the HSIC task during forward propagation, while the HSIC task continuously corrects the graph during backward propagation. Therefore, the “automatic” of the proposed Auto-GCN is not only reflected in the fact that the graph representation is designed and updated by an end-to-end network but is also HSIC task-oriented. The experimental results show that the proposed Auto-GCN outperforms other state-of-the-art methods on four publicly available hyperspectral datasets.

Scatter targets of complex land covers in polarimetric synthetic aperture radar (PolSAR) images are often randomly oriented and cause randomly fluctuating echoes, which brings a challenge to PolSAR image classification. Therefore, many existing methods have alleviated this problem through orientation compensation. However, there are still two obstacles that limit the improvement of classification accuracy. On the one hand, generally, these methods process PolSAR images with fixed polarization rotation angles, which is experience-dependent and inflexible. On the other hand, for the different land covers of a PolSAR image, the existing methods do not consider these rotation angles separately. For the first obstacle, we design a group of convolution kernels called polarization rotation kernels (PRKs) and utilize them to build the polarimetric convolutional neural network (CNN) (PolCNN). The PolCNN is the base network of our final model, and it can learn polarization rotation angles adaptively. For the second obstacle, we extend the PolCNN into a multipath structure, the final model polarimetric multipath CNN (PolMPCNN). The polarization rotation angles of different land covers are directly related to the networks of different paths within the PolMPCNN. Furthermore, we also put forward the two-scale sampling and the stagewise training algorithm in order that our PolMPCNN can fit different scales of PolSAR targets and pays more attention to difficult training samples. Experiments on real PolSAR images show that the proposed model achieves the best classification results with an extremely low sampling rate of 0.1%.

Convolutional neural networks (CNN) can extract shift-invariant features, and have been widely applied in change detection task. However, common CNN lacks noise robustness and needs supervised data, to alleviate these problems, in this paper, we propose a novel deep shearlet network (ShearNet) for change detection in SAR images. In the network, a shearlet denoising layer (SDL) is designed to enhance the representation ability of common CNN. In SDL, feature maps are decomposed into subband coefficients by shearlet transform (ST). Due to optimal sparse representation property and highly direction sensitivity of ST, the network can capture important geometric information. Then, hard-threshold shrinkage is applied to high frequency subbands to drop small coefficients that are most likely to be noise, so that reduce the effect of noise. Finally, ShearNet is trained by introducing a noise-robust loss with noisy labels. The noisy labels are obtained by deep clustering that shows more robustness than existing preclassification methods. This fine-tuning process novelly follows the paradigm of learning from noisy labels to aside the difficulty of precisely labeling samples. Our experimental results on multiple real SAR datasets show that ShearNet can boost accuracy, and have better applicability for change detection in SAR images. The source code is available at https://github.com/yizhilanmaodhh/ShearNet.

Autofocus plays a key role in synthetic aperture radar (SAR) imaging, especially for high-resolution imaging. In the literature, the minimum-entropy-based algorithms (MEA) have been proved to be robust and have been widely applied in SAR. However, this kind of method needs hundreds of iterations and is computationally expensive. In this paper, we proposed a non-iterative autofocus scheme based on deep learning and minimum-entropy criterion. It’s an unsupervised framework, which utilizes entropy as the loss function. In this scheme, deep neural networks are utilized for feature extraction and parameter estimation. Based on this scheme, two autofocus models (autofocus network and progressive autofocus network) are proposed. After training, the network learned the rules of autofocus from a large number of examples. Experimental results on real SAR data show that the proposed methods have focusing quality close to the state-of-the-art but with real-time focusing speed.

Semantic segmentation based on deep learning has achieved impressive results in recent years, but these results are supported by a large amount of labeled data which requires intensive annotation at the pixel level, particularly for high-resolution remote sensing (RS) images. In this work, we propose a simple yet efficient semisupervised learning framework based on linear sampling self-training, named LSST, to improve the performance of RS image semantic segmentation. Specifically, the classical pseudo-labeling-based self-training paradigm is enhanced by injecting strong data augmentations (SDA) applicable to RS images, based on which a powerful baseline is constructed. Nevertheless, the problem of insufficient data training to generate pseudo-labels with a high level of noise persists, and the noisy pseudo-labels will continue to accumulate and impede model improvement during the re-training phase. Previous works commonly employ a pre-defined threshold to remove noise, but it will lead to overfitting the model to easily identified classes. To address it, a method using linear sampling (LS) is presented for assigning thresholds to different classes in an adaptive manner, which provides noiseless regions for re-training. Experiments prove that the proposed pixel-wise selection is more available for segmentation than image-level selection in RS images. Finally, LSST achieves state-of-the-art on several datasets and different evaluation metrics. The source code of the this paper is available at https://github.com/xiaoqiang-lu/LSST.

To maximize the complementary advantages of synergistic multimodal, a transfer representation learning fusion network (TRLF-Net) is proposed for multisource remote sensing images collaborative classification in this article. First, with respect to the feature encoding, we design a dual-branch attention sparse transfer module (DAST-Module), which combines the spatial and channel attention (CA) masks to migrate the advantage attributes of the panchromatic (PAN) and the MS images mutually. This not only enhances their respective image advantages but also facilitates the sparse fusion of low-level features. Second, for the separation of multiscale information, a deep dual-scale decomposition module (DDSD-Module) is designed, which allows the decompose of high-frequency and low-frequency components. Then it uses the decomposed information to make the essential difference as small as possible, and the surrounding contour difference is as large as possible of the complementary multimodal image through the design of the loss function. Finally, to address the problem of large intraclass and small interclass differences, we develop a representation fusion of the global and local features’ module (RFGAL-Module). It mainly adopts global features to sort local features within classes, and then outputs them in a cascade. Thus, the characterization ability of features is improved, and the global and local features are used in a coordinated manner to accomplish the sample classification tasks. In particular, the experimental results demonstrate that TRLF-Net can obtain much improved accuracy and efficiency. The code is accessible in:https://github.com/ru-willow/SRLF-Net.

This paper proposes a practical end-to-end neural network framework to detect tiny moving vehicles in satellite videos with low imaging quality. Some instability factors such as illumination changes, motion blurs, and low contrast to the cluttered background make it difficult to distinguish true objects from noise and other point-shaped distractors. Moving vehicle detection in satellite videos can be carried out based on background subtraction or frame differencing. However, these methods are prone to produce lots of false alarms and miss many positive targets. Appearance-based detection can be an alternative but is not well-suited since classifier models are of weak discriminative power for the vehicles in top view at such low resolution. This article addresses these issues by integrating motion information from adjacent frames to facilitate the extraction of semantic features and incorporating the Transformer to refine the features for key points estimation and scale prediction. Our proposed model can well identify the actual moving targets and suppress interference from stationary targets or background. The experiments and evaluations using satellite videos show that the proposed approach can accurately locate the targets under weak feature attributes and improve the detection performance in complex scenarios.

Remote sensing object tracking is a novel and challenging problem due to the negative effects of weak features and background noise. In this paper, from the perspective of attention-focus deep learning, we propose a Joint Siamese Attention-Aware Network (JSANet) for efficient remote sensing tracking which contains both self-attention and cross-attention modules. First, the self-attention modules we propose emphasize the interdependent channel-wise coefficient via channel attention and conduct corresponding space transformation of spatial domain information with spatial attention. Second, the cross-attention is designed to aggregate rich contextual interdependencies between the siamese branches via channel attention and excavate association produces reliable correspondence with spatial attention. In addition, a composite feature combine strategy is designed to fuse multiple attention features. Experimental results on the Jilin-1 satellite video datasets demonstrate that the proposed JSANet achieves state-of-the-art performance in terms of precision and success rate, demonstrate the effectiveness of the proposed methods.

The aim of multispectral (MS) and panchromatic (PAN) image fusion is to obtain an MS image that has high resolution in both spectral and spatial domains. During the fusion process, there are two important issues, i.e., spectral information preservation and spatial information enhancement. In this article, we propose a dual-collaborative fusion model that considers not only the spectral correlation collaboration but also the spatial-spectral collaboration. First, the features of PAN and MS images are extracted by a shared feature embedding network. Then, in order to enhance the spatial details, the PAN features are decomposed into four subbands, and the collaborative relationships among subbands are fully explored to refine the features. After the refinement of the subbands, the high-frequency components are directly taken as the inputs of the reconstruction network, while the low-frequency components are transformed by the guidance generation network to accomplish the spatial-spectral collaboration and also make preparations for the spectral adjustment. To explore the spectral correlation collaboration, a novel graph convolutional network is designed for the modulation of intraspectral relationships. Finally, the adjusted MS features are combined with the high-frequency components of PAN features to reconstruct the high-resolution MS image. Experimental results show that the proposed method outperforms traditional state-of-the-art pan-sharpening methods as well as the available deep learning-based ones.

In this article, we propose a sparse feature clustering network (SFCNet) for change detection in synthetic aperture radar (SAR) images. One of the principal problems in dealing with SAR images is to reduce the impact of speckle noise. Therefore, based on a neural network framework for change detection, we introduce the multiobjective sparse feature learning (MO-SFL) model where the sparsity of representation is adaptively learned in order to increase the robustness to different levels of noise. For learning the semantic information of changed and unchanged pixels, the network is fine-tuned by the correctly labeled samples selected from coarse results. The selection criterion influences the change detection result a lot. Therefore, we construct a novel cross-entropy clustering loss (CEC) by introducing a clustering regularization term to learn the discriminative representations. Experiments on simulate and real SAR images demonstrate the superiority of the proposed method over compared methods.

Recently, deep learning based multispectral (MS) and panchromatic (PAN) image fusion methods have been proposed, which extracted features automatically and hierarchically by a series of non-linear transformations to model the complicated imaging discrepancy. But they always pay more attention to the extraction and compensation of spatial details and use the mean squared error or mean absolute error as a loss function, regardless of the preservation of spectral information contained in multispectral images. For the sake of the improvements in both spatial and spectral resolution, this paper presents a novel fusion model that takes the spectral preservation into consideration, and learns the spectral cues from the process of generating a spectrally refined multispectral image, which is constrained by a spectral loss between the generated image and the reference image. Then these spectral cues are used to modulate the PAN features to obtain final fusion result. Experimental results on reduced-resolution and full-resolution datasets demonstrate that the proposed method can obtain a better fusion result in terms of visual inspection and evaluation indices when compared with current state-of-the-art methods.

Contrastive self-supervised learning (CSSL) has achieved promising results in extracting visual features from unlabeled data. Most of the current CSSL methods are used to learn global image features with low-resolution that are not suitable or efficient for pixel-level tasks. In this paper, we propose a coarse-to-fine CSSL framework based on a novel contrasting strategy to address this problem. It consists of two stages, one for encoder pre-training to learn global features and the other for decoder pre-training to derive local features. Firstly, the novel contrasting strategy takes advantage of the spatial structure and semantic meaning of different regions and provides more cues to learn than that relying only on data augmentation. Specifically, a positive pair is built from two nearby patches sampled along the direction of the texture if they fall into the same cluster. A negative pair is generated from different clusters. When the novel contrasting strategy is applied to the coarse-to-fine CSSL framework, global and local features are learned successively by forcing the positive pair close to each other and the negative pair apart in an embedding space. Secondly, a discriminant constraint is incorporated into the per-pixel classification model to maximize the inter-class distance. It makes the classification model more competent at distinguishing between different categories that have similar appearance. Finally, the proposed method is validated on four SAR images for land-cover classification with limited labeled data and substantially improves the experimental results. The effectiveness of the proposed method is demonstrated in pixel-level tasks after comparison with the state-of-the-art methods.

In this paper, a novel unsupervised change detection method called adaptive Contourlet fusion clustering based on adaptive Contourlet fusion and fast non-local clustering is proposed for multi-temporal synthetic aperture radar (SAR) images. A binary image indicating changed regions is generated by a novel fuzzy clustering algorithm from a Contourlet fused difference image. Contourlet fusion uses complementary information from different types of difference images. For unchanged regions, the details should be restrained while highlighted for changed regions. Different fusion rules are designed for low frequency band and high frequency directional bands of Contourlet coefficients. Then a fast non-local clustering algorithm (FNLC) is proposed to classify the fused image to generate changed and unchanged regions. In order to reduce the impact of noise while preserve details of changed regions, not only local but also non-local information are incorporated into the FNLC in a fuzzy way. Experiments on both small and large scale datasets demonstrate the state-of-the-art performance of the proposed method in real applications.

Video object detection, a basic task in the computer vision field, is rapidly evolving and widely used. In recent years, deep learning methods have rapidly become widespread in the field of video object detection, achieving excellent results compared with those of traditional methods. However, the presence of duplicate information and abundant spatiotemporal information in video data poses a serious challenge to video object detection. Therefore, in recent years, many scholars have investigated deep learning detection algorithms in the context of video data and have achieved remarkable results. Considering the wide range of applications, a comprehensive review of the research related to video object detection is both a necessary and challenging task. This survey attempts to link and systematize the latest cutting-edge research on video object detection with the goal of classifying and analyzing video detection algorithms based on specific representative models. The differences and connections between video object detection and similar tasks are systematically demonstrated, and the evaluation metrics and video detection performance of nearly 40 models on two data sets are presented. Finally, the various applications and challenges facing video object detection are discussed.

Global average pooling (GAP) allows convolutional neural networks (CNNs) to localize discriminative information for recognition using only image-level labels. While GAP helps CNNs to attend to the most discriminative features of an object, e.g., head of a bird or one man's bag, it may suffer if that information is missing due to camera viewpoint changes and intraclass variations in some tasks. To circumvent this issue, we propose one new module to help CNNs to see more, namely, Spatial Rescaling (SpaRs) layer. It introduces spatial relations among the feature map activations back to the model, guiding the model to focus on a broad area in the feature map. With simple implementation, it can be inserted into CNNs of various architectures directly. SpaRs layer consistently improves the performance over the reidentification (re-ID) models. Besides, the new module based on different normalization methods also demonstrates the superiority of fine-grained and general image classification benchmarks. The visualization method shows the changes in activated regions when equipped with the SpaRs layer for better understanding. Our code is publicly available at https://github.com/HRanWang/Spatial-Re-Scaling.

Due to the complementary properties of different types of sensors, change detection between heterogeneous images receives increasing attention from researchers. However, change detection cannot be handled by directly comparing two heterogeneous images since they demonstrate different image appearances and statistics. In this article, we propose a deep pyramid feature learning network (DPFL-Net) for change detection, especially between heterogeneous images. DPFL-Net can learn a series of hierarchical features in an unsupervised fashion, containing both spatial details and multiscale contextual information. The learned pyramid features from two input images make unchanged pixels matched exactly and changed ones dissimilar and after transformed into the same space for each scale successively. We further propose fusion blocks to aggregate multiscale difference images (DIs), generating an enhanced DI with strong separability. Based on the enhanced DI, unchanged areas are predicted and used to train DPFL-Net in the next iteration. In this article, pyramid features and unchanged areas are updated alternately, leading to an unsupervised change detection method. In the feature transformation process, local consistency is introduced to constrain the learned pyramid features, modeling the correlations between the neighboring pixels and reducing the false alarms. Experimental results demonstrate that the proposed approach achieves superior or at least comparable results to the existing state-of-the-art change detection methods in both homogeneous and heterogeneous cases.

Incremental Signal Classification (ISC) aims to continuously identify and classify unknown signal categories, which is essentially an open-set classification task. In this paper, a new Deep Ensemble Siamese Network (DESN) is constructed for unknown category detection and incremental accumulation of signals from the detected category. Then the accumulated samples are used to update a One-dimensional Convolution Network (OCN) for incremental learning of new signal categories. Experimental results show that the proposed method can achieve accurate detection and accumulation of unknown signals, and is feasible for practical ISC.

Existing learned-based image compression methods have shown impressive performance. However, most of them rely on the consistency of distribution between training images and test images, which limits the robustness of the trained model. In this paper, we propose a novel compression method called sparse flow adversarial model (SFAM). SFAM employs a deep generative framework to learn a reversible and stable mapping between image distributions, thus it can work in varied scenes for robust compression. Moreover, a sparse adversarial map is introduced into SFAM, to constrain the SFAM to generate more sparser features for efficient compression. Extensive experiments are conducted on different datasets, in which the effectiveness and robustness of the proposed method is verified. Meanwhile, SFAM is trained only once and it can work well on three different datasets, which also proves the robustness of the proposed SFAM.

This paper proposes a Simple Semi-supervised Object Detection framework for Remote Sensing images, which is named SSOD-RS. SSOD-RS contains two parts, improved self-training and consistency regularization based on strong data augmentations with improved mixup. Firstly, as an augmentation algorithm, Object First mixup (OF-mixup) is proposed to adjust the weight of objects and the background, which expands the distribution of training samples while reducing the interference of the remote sensing complex background to the features of objects. Secondly, the strategy of training with assembling loss and fine-tuning is introduced into self-training to make the model fit the feature distribution of the true-labels after learning the features from the pseudo-labels. Experimental results demonstrate that SSOD-RS making use of unlabeled images can significantly improve the accuracy of the model.

Abstract Remote sensing image fusion plays important roles in numerous applications, including monitoring, metrology, and agriculture. Image fusion gathers essential information from several image sources and consolidates them into a single image called a fused image. The fused image involves relevant data, and it is more informative than any other images extracted from one source. This study proposed a pansharpening technique based on image filtering utilising a bilateral filter to generate high‐frequency details from panchromatic image. The various types of side window guided filters are employed to enhance the multispectral band from panchromatic image and then used these filters to adjust spatial data misfortune that happens when images are combined. Experimental results demonstrated that the proposed method provides consistent results concise with reported by the previous research in terms of subjective and objective assessments on remote sensing data.

Most of the available graph-based semisupervised hyperspectral image classification methods adopt the cluster assumption to construct a Laplacian regularizer. However, they sometimes fail due to the existence of mixed pixels whose recorded spectra are a combination of several materials. In this paper, we propose a geometric low-rank Laplacian regularized semisupervised classifier, by exploring both the global spectral geometric structure and local spatial geometric structure of hyperspectral data. A new geometric regularized Laplacian low-rank representation (GLapLRR)-based graph is developed to evaluate spectral-spatial affinity of mixed pixels. By revealing the global low-rank and local spatial structure of images via GLapLRR, the constructed graph has the characteristics of spatial-spectral geometry description, robustness, and low sparsity, from which a more accurate classification of mixed pixels can be achieved. The proposed method is experimentally evaluated on three real hyperspectral datasets, and the results show that the proposed method outperforms its counterparts, when only a small number of labeled instances are available.

Saliency detection is a hot topic in recent years and much efforts have been made to address it from different perspectives. However, current saliency models cannot meet the needs for diversified scenes due to their limited generalization capability. To tackle this problem, in this paper, we propose a hybrid saliency model, which can fuse heterogeneous visual cues for robust salient object detection. A new contour cue is first introduced to provide discriminative saliency information for scene description. Its realization is based on a discrete optimization objective and can be solved efficiently with an iterative algorithm. Followed by this, the contour cue is taken as a part of a hybrid sparse learning model, in which cues from different domains can interact and complement with each other for joint saliency fusion. This saliency fusion model is parameter-free and its numerical solution can be obtained using gradient descent methods. Finally, we advance an object proposal-based collaborative filtering strategy to generate high quality saliency maps from the above fusion results. Compared with traditional methods, the proposed saliency model can fuse heterogeneous cues in a unified optimization framework rather than combine them separately. Therefore, it has favorable modeling capability under diversified scenes where the saliency patterns appear quite differently. To verify the effectiveness of the proposed method, we take experiments on four large saliency benchmark datasets and compare it with other 26 state-of-the-art saliency models. Both qualitative and quantitative evaluation results indicate the superiority of our method, especially in challenging situations. Besides, we apply our saliency model to ship detection of radar platforms and promising results are obtained over traditional detectors.

Deep learning (DL) has become a hot topic in the research field of hyperspectral image (HSI) classification. However, with increasing depth and size of deep learning methods, its application in mobile and embedded vision applications has brought great challenges. In this article, we address a network architecture search (NAS)-guided lightweight spectral–spatial attention feature fusion network (LMAFN) for HSI classification. The overall architecture of the proposed network is guided by several conclusions of NAS, which achieves fewer parameters and lower computation cost with deeper network structure by exploiting multiscale Ghost grouped with efficient channel attention (ECA) module for adaptively adjusting the weights of different channels. It helps fully extract spectral–spatial discriminant features to avoid information loss of the dimension reduction operation. Specifically, a multilayer feature fusion method is proposed to extract the fusion information of the spectral–spatial features of each layer by considering complementary information of different hierarchical structures. Therefore, high-lever spectral–spatial attributes are gradually exploited along with the increase in layers and the fusion of layers. The experimental verification on three real HSI data sets demonstrates that the proposed framework presents more satisfying classification performance and efficiency with deeper network structure and lower parameter size.

Supervised change detection methods always face a big challenge that the current scene (target domain) is fully unlabeled. In remote sensing, it is common that we have sufficient labels in another scene (source domain) with a different but related data distribution. In this article, we try to detect changes in the target domain with the help of the prior knowledge learned from multiple source domains. To achieve this goal, we propose a change detection framework based on selective adversarial adaptation. The adaptation between multisource and target domains is fulfilled by two domain discriminators. First, the first domain discriminator regards each scene as an individual domain and is designed for identifying the domain to which each input sample belongs. According to the output of the first domain discriminator, a subset of important samples is selected from multisource domains to train a deep neural network (DNN)-based change detection model. As a result, not only the positive transfer is enhanced but also the negative transfer is alleviated. Second, as for the second domain discriminator, all the selected samples are thought from one domain. Adversarial learning is introduced to align the distributions of the selected source samples and the target ones. Consequently, it further adapts the knowledge of change from the source domain to the target one. At the fine-tuning stage, target samples with reliable labels and the selected source ones are used to jointly fine-tune the change detection model. As the target domain is fully unlabeled, homogeneity- and boundary-based strategies are exploited to make the pseudolabels from a preclassification map reliable. The proposed method is evaluated on three SAR and two optical data sets, and the experimental results have demonstrated its effectiveness and superiority.

In the last five years, deep learning has been introduced to tackle the hyperspectral image (HSI) classification and demonstrated good performance. In particular, the convolutional neural network (CNN)-based methods for HSI classification have made great progress. However, due to the high dimensionality of HSI and equal treatment of all bands, the performance of these methods is hampered by learning features from useless bands for classification. Moreover, for patchwise-based CNN models, equal treatment of spatial information from the pixel-centered neighborhood also hinders the performance of these methods. In this article, we propose an end-to-end residual spectral-spatial attention network (RSSAN) for HSI classification. The RSSAN takes raw 3-D cubes as input data without additional feature engineering. First, a spectral attention module is designed for spectral band selection from raw input data by emphasizing useful bands for classification and suppressing useless bands. Then, a spatial attention module is designed for the adaptive selection of spatial information by emphasizing pixels from the same class as the center pixel or those are useful for classification in the pixel-centered neighborhood and suppressing those from a different class or useless. Second, two attention modules are also used in the following CNN for adaptive feature refinement in spectral-spatial feature learning. Third, a sequential spectral-spatial attention module is embedded into a residual block to avoid overfitting and accelerate the training of the proposed model. Experimental studies demonstrate that the RSSAN achieved superior classification accuracy compared with the state of the art on three HSI data sets: Indian Pines (IN), University of Pavia (UP), and Kennedy Space Center (KSC).

Extracting effective features is always a challenging problem for texture classification because of the uncertainty of scales and the clutter of textural patterns. For texture classification, spectral analysis is traditionally employed in the frequency domain. Recent studies have shown the potential of convolutional neural networks (CNNs) when dealing with the texture classification task in the spatial domain. In this article, we try combining both approaches in different domains for more abundant information and proposed a novel network architecture named contourlet CNN (C-CNN). The network aims to learn sparse and effective feature representations for images. First, the contourlet transform is applied to get the spectral features from an image. Second, the spatial-spectral feature fusion strategy is designed to incorporate the spectral features into CNN architecture. Third, the statistical features are integrated into the network by the statistical feature fusion. Finally, the results are obtained by classifying the fusion features. We also investigated the behavior of the parameters in contourlet decomposition. Experiments on the widely used three texture data sets (kth-tips2-b, DTD, and CUReT) and five remote sensing data sets (UCM, WHU-RS, AID, RSSCN7, and NWPU-RESISC45) demonstrate that the proposed approach outperforms several well-known classification methods in terms of classification accuracy with fewer trainable parameters.

The extreme learning machine (ELM) has drawn increasing attention in the field of machine learning due to its high accuracy and efficient learning. However, classical ELM works in batch and passive learning paradigms, which cannot deal with sequential data effectively. ELM has been extended to online sequential learning form (OS-ELM) and active learning form (AL-ELM), in which the former is used to improve training efficiency and the latter is mainly adopted to improve accuracy. In order to solve the problem of labeling samples difficulty and costly, poor sample validity, and continuous iterative learning in polarimetric synthetic aperture radar (PolSAR) image classification, we propose an online active extreme learning machine (OA-ELM) algorithm to combine the strengths and make up the weaknesses of OS-ELM and AL-ELM, which improves both efficiency and generalization ability. OA-ELM can learn from sequential data dynamically with low computational complexity and good generalization ability. Specifically, OA-ELM reduces time and memory cost for training via extended recursive least squares for optimization. It also improves accuracy using informative training samples selected by proposed discrepancy sampling (DS), which modifies an active query method called margin sampling (MS). Before applying MS to ELM, real-valued outputs of ELM need to be converted into probabilistic outputs first. Instead, the proposed DS can be applied to ELM directly by calculating the difference between the two largest actual nonprobabilistic outputs of ELM. Experimental results of PolSAR classification demonstrate that OA-ELM is effective and efficient compared with other algorithms in terms of accuracy and running time.

In this article, a new deep neural network based on sparse filtering and manifold regularization (DSMR) is proposed for feature extraction and classification of polarimetric synthetic aperture radar (PolSAR) data. DSMR uses a novel deep neural network (DNN) to automatically learn features from raw SAR data. During preprocessing, the spatial information between pixels on PolSAR images is exploited to weight each data sample. Then, in the pretraining and fine-tuning, DSMR uses the population sparsity and the lifetime sparsity (dual sparsity) to learn the global features and preserves the local structure of data by neighborhood-based manifold regularization. The dual sparsity only needs to tune a few parameters, and the manifold regularization cuts down the number of training samples. Experimental results on synthesized and real PolSAR data sets from different SAR systems show that DSMR can improve classification accuracy compared with conventional DNNs, even for data sets with a large angle of incidence.

Semi-supervised non-negative matrix factorization (NMF) exploits the strengths of NMF in effectively learning local information contained in data and is also able to achieve effective learning when only a small fraction of data is labeled. NMF is particularly useful for dimensionality reduction of high-dimensional data. However, the mapping between the low-dimensional representation, learned by semi-supervised NMF, and the original high-dimensional data contains complex hierarchical and structural information, which is hard to extract by using only single-layer clustering methods. Therefore, in this article, we propose a new deep learning method, called semi-supervised graph regularized deep NMF with bi-orthogonal constraints (SGDNMF). SGDNMF learns a representation from the hidden layers of a deep network for clustering, which contains varied and unknown attributes. Bi-orthogonal constraints on two factor matrices are introduced into our SGDNMF model, which can make the solution unique and improve clustering performance. This improves the effect of dimensionality reduction because it only requires a small fraction of data to be labeled. In addition, SGDNMF incorporates dual-hypergraph Laplacian regularization, which can reinforce high-order relationships in both data and feature spaces and fully retain the intrinsic geometric structure of the original data. This article presents the details of the SGDNMF algorithm, including the objective function and the iterative updating rules. Empirical experiments on four different data sets demonstrate state-of-the-art performance of SGDNMF in comparison with six other prominent algorithms.

In order to obtain high classification accuracy and reduce time consumption for large-scale polarimetric synthetic-aperture radar (PolSAR) data. In this letter, we propose a fast semisupervised classification algorithm using histogram-based density estimation (called FSHDE). First, a noniterative collaborative training using our proposed Wishart-clustering selection strategy is designed to expand the labeled sample set from unlabeled samples. Second, a fast feature mapping based on histogram density estimation is employed to reliably capture the interaction of nonlinear features. Third, submodular optimization is used to select optimal subspace features to reduce feature correlation. Experimental results on synthetic and real PolSAR data indicate that FSHDE greatly reduces the time consumption and improves the accuracy for terrain classification compared with the state-of-the-art methods.

High-dimensionality is a common characteristic of real-world data, which often results in high time and space complexity or poor performance of ensuing methods. Subspace learning, as one kind of dimension reduction method, provides a way to overcome the aforementioned problem. In this paper, we introduce multiobjective evolutionary optimization into subspace learning, and propose a Pareto-based sparse subspace learning algorithm for classification tasks. The proposed algorithm aims at minimizing two conflicting objective functions, the reconstruction error and the sparsity. A kernel trick derived from Gaussian kernel is implemented to the sparse subspace learning for the nonlinear phenomena of nature. In order to speed up the convergence, an entropy-driven initialization scheme and a gradient-descent mutation scheme are designed specifically. At last, a knee point is selected from the Pareto front to guarantee that we can obtain a solution with good classification performance, and yet as sparse as possible. The experiments and detailed analysis on real-life datasets and the hyperspectral images demonstrated that the proposed model achieves comparable results with the existing conventional subspace learning and evolutionary feature selection algorithms. Hence, this paper provides a more flexible and efficient approach for sparse subspace learning.

Recently, sparse coding-based image fusion methods have been developed extensively. Although most of them can produce competitive fusion results, three issues need to be addressed: 1) these methods divide the image into overlapped patches and process them independently, which ignore the consistency of pixels in overlapped patches; 2) the partition strategy results in the loss of spatial structures for the entire image; and 3) the correlation in the bands of multispectral (MS) image is ignored. In this paper, we propose a novel image fusion method based on convolution structure sparse coding (CSSC) to deal with these issues. First, the proposed method combines convolution sparse coding with the degradation relationship of MS and panchromatic (PAN) images to establish a restoration model. Then, CSSC is elaborated to depict the correlation in the MS bands by introducing structural sparsity. Finally, feature maps over the constructed high-spatial-resolution (HR) and low-spatial-resolution (LR) filters are computed by alternative optimization to reconstruct the fused images. Besides, a joint HR/LR filter learning framework is also described in detail to ensure consistency and compatibility of HR/LR filters. Owing to the direct convolution on the entire image, the proposed CSSC fusion method avoids the partition of the image, which can efficiently exploit the global correlation and preserve the spatial structures in the image. The experimental results on QuickBird and Geoeye-1 satellite images show that the proposed method can produce better results by visual and numerical evaluation when compared with several well-known fusion methods.

Although several works have been done on high-resolution inverse synthetic aperture radar (ISAR) imaging via compressive sampling technology, they only explore sparse priors of targets in the scene and the image is recovered cell by cell separately. In order to potentially enhance targets and suppress clutters for fast and higher quality imaging, in this paper, we advance a collaborative compressive ISAR (CC-ISAR) imaging approach, by exploring both sparse priors and saliency priors of targets. First, a geometric saliency map is derived by performing pulse contourlet transform on a preliminary image. Then, a graph Laplacian is constructed to regularize a multiple measurement vector problem for collaborative compressive radar imaging. Third, targets are approximately separated from the background in the saliency map, and salient weights are defined for the target and background, respectively, to derive a saliency weighted l1-norm optimization algorithm. Some experiments are taken on real ISAR data to evaluate the performance of the proposed method, and both visual results and numerical guidelines prove that CC-ISAR imaging method can obtain more accurate targets and outperform its counterparts.

Supervised deep neural networks (DNNs) have been extensively used in diverse tasks. Generally, training such DNNs with superior performance requires a large amount of labeled data. However, it is time-consuming and expensive to manually label the data, especially for tasks in remote sensing, e.g., change detection. The situation motivates us to resort to the existing related images with labels, from which the concept of change can be adapted to new images. However, the distributions of the related labeled images (source domain) and unlabeled new images (target domain) are similar but not identical. It impedes a change detection model learned from source domains being well applied to the target domain. In this paper, we propose a transferred deep learning-based change detection framework to solve this problem. It consists of pretraining and fine-tuning stages. In the pretraining process, we propose two tasks to be learned simultaneously, namely, change detection for the source domain with labels and reconstruction of the unlabeled target data. The auxiliary task aims to reconstruct the difference image (DI) for the target domain. DI is an effective feature, such that the auxiliary task is of much relevance to change detection. The lower layers are shared between these two tasks in the training process. It mitigates the distribution discrepancy between the source and target domains and makes the concept of change from the source domain adapt to the target domain. In addition, we evaluate three modes of the U-net architecture to merge the information for a pair of patches. To fine-tune the change detection network (CDN) for the target domain, two strategies are exploited to select the pixels that have a high possibility of being correctly classified by an unsupervised approach. The proposed method demonstrates an excellent capacity for adapting the concept of change from the source domain to the target domain. It outperforms the state-of-the-art change detection methods via experimental results on real remote sensing data sets.

A novel color image segmentation-based depth map upsampling method is proposed in this paper. In this method, the color image is segmented into a certain number of connected regions first. Based on the segmentation result, the target pixels will be interpolated by the seed pixels11The seed pixels are directly from the low resolution depth maps, i.e., the ones that have depth values. The targets are those without depth and to be interpolated. regionally. In the segmentation part, simple linear iterative clustering is introduced to generate superpixels in the first place. Then, the obtained superpixels will be judged whether they are correct-clustered or not, and the incorrect-clustered ones will be subdivided with an adaptive region-growing strategy. Third, the regions that have no seed will be constantly merged into their nearest neighbors, until seed pixel can be found in each independent region. Finally, adjacent regions that have quite small depth gaps will be united as one. The proposed color image segmentation strictly follows the guidance of the depth; therefore, the segmented regions adhere to the depth boundary well. In the interpolation part, the targets will be interpolated with their surrounding seeds weighted by a joint trilateral filter (JTF). The JTF is constructed by three terms: the color term, the distance term, and the region term, which are driven by the previous segmentation result. Experimental results indicate that our method greatly reduces depth bleeding and depth confusion artifacts, and leads to clear depth boundary in the up-sampled image. Comparisons with the state of art verify the advantages of the proposed method in both visual experience and quantitative evaluations.

To detect changed areas in multitemporal polarimetric synthetic aperture radar (SAR) images, this paper presents a novel version of convolutional neural network (CNN), which is named local restricted CNN (LRCNN). CNN with only convolutional layers is employed for change detection first, and then LRCNN is formed by imposing a spatial constraint called local restriction on the output layer of CNN. In the training of CNN/LRCNN, the polarimetric property of SAR image is fully used instead of manual labeled pixels. As a preparation, a similarity measure for polarimetric SAR data is proposed, and several layered difference images (LDIs) of polarimetric SAR images are produced. Next, the LDIs are transformed into discriminative enhanced LDIs (DELDIs). CNN/LRCNN is trained to model these DELDIs by a regression pretraining, and then a classification fine-tuning is conducted with some pseudolabeled pixels obtained from DELDIs. Finally, the change detection result showing changed areas is directly generated from the output of the trained CNN/LRCNN. The relation of LRCNN to the traditional way for change detection is also discussed to illustrate our method from an overall point of view. Tested on one simulated data set and two real data sets, the effectiveness of LRCNN is certified and it outperforms various traditional algorithms. In fact, the experimental results demonstrate that the proposed LRCNN for change detection not only recognizes different types of changed/unchanged data, but also ensures noise insensitivity without losing details in changed areas.

In this paper a self-paced learning-based probability subspace projection (SL-PSP) method is proposed for hyperspectral image classification. First, a probability label is assigned for each pixel, and a risk is assigned for each labeled pixel. Then, two regularizers are developed from a self-paced maximum margin and a probability label graph, respectively. The first regularizer can increase the discriminant ability of features by gradually involving the most confident pixels into the projection to simultaneously push away heterogeneous neighbors and pull inhomogeneous neighbors. The second regularizer adopts a relaxed clustering assumption to make avail of unlabeled samples, thus accurately revealing the affinity between mixed pixels and achieving accurate classification with very few labeled samples. Several hyperspectral data sets are used to verify the effectiveness of SL-PSP, and the experimental results show that it can achieve the state-of-the-art results in terms of accuracy and stability.

In past decades, saliency detection has received increasing attention from computer vision communities, for its potential usage in many vision-related tasks. However, finding representative and discriminative features to accurately locate salient regions from complex scenes remains a challenging problem. Recent research on primary visual cortex (V1) shows that vision neurons are sparsely connected to form a compact representation of natural scenes and different visual stimuli are processed separately according to their semantic importance. Inspired by the above characteristics of visual perception, in this paper we advance a novel saliency detection method via representative and discriminative dictionary learning. An assumption that salient and nonsalient information are sparsely coded under two separate dictionaries is cast on the problem and we propose to learn a compact background dictionary from the image itself for saliency estimation. Different from previous methods, our saliency cues are obtained via active learning strategies rather than artificially designed rules, and thus is more adaptive. Followed by this, a probabilistic inference model is deduced to fully excavate multisource information about the scenes for high-quality saliency map generation. This joint inference scheme takes both spatial and color space information into consideration and is proved to be quite effective in practice. Finally, to investigate the performance of the proposed model, some experiments are conducted on two benchmark data sets along with other 20 state-of-the-art saliency detection approaches. The experimental results show that our method outperforms its counterparts and can correctly detect salient regions, even when other methods fail. Besides, the usability of the proposed method in real application-based cases is verified by applying it to content-based image resizing and promising results are obtained.

In this paper, it is proposed the directional estimation model on the overcomplete dictionary, which bridges the compressed measurements of the image blocks and the directional structures of the dictionary. In the model, it is established the analytical method to estimate the structure type of a block as either smooth, single-oriented, or multioriented. Furthermore, the structures of each type of blocks are described by the structured subdictionaries. Then based on the obtained estimations and the constrains on the sparse dictionaries, the original image will be estimated. To verify the model, the nonconvex methods are designed for compressed sensing. Specifically, the greedy pursuit-based methods are established to search the subdictionaries obtained by the model, which achieve better local structural estimation than the methods without the directional estimation. More importantly, it is proposed the nonconvex image reconstruction method with direction-guided dictionaries and evolutionary searching strategies (NR_DG), where the evolutionary searching strategies are delicately designed for each type of the blocks based on the directional estimation. By the experimental results, it is shown that the NR_DG method performs better than the available two-stage evolutionary reconstruction method.

This paper introduces the popular sparse representation method into the classical fuzzy c-means clustering algorithm, and presents a novel fuzzy clustering algorithm, called fuzzy double c-means based on sparse self-representation (FDCM_SSR). The major characteristic of FDCM_SSR is that it can simultaneously address two datasets with different dimensions, and has two kinds of corresponding cluster centers. The first one is the basic feature set that represents the basic physical property of each sample itself. The second one is learned from the basic feature set by solving a spare self-representation model, referred to as discriminant feature set, which reflects the global structure of the sample set. The spare self-representation model employs dataset itself as dictionary of sparse representation. It has good category distinguishing ability, noise robustness, and data-adaptiveness, which enhance the clustering and generalization performance of FDCM_SSR. Experiments on different datasets and images show that FDCM_SSR is more competitive than other state-of-the-art fuzzy clustering algorithms.

Panchromatic (PAN) and multispectral (MS) imagery classification is one of the hottest topics in the field of remote sensing. In recent years, deep learning techniques have been widely applied in many areas of image processing. In this paper, an end-to-end learning framework based on deep multiple instance learning (DMIL) is proposed for MS and PAN images’ classification using the joint spectral and spatial information based on feature fusion. There are two instances in the proposed framework: one instance is used to capture the spatial information of PAN and the other is used to describe the spectral information of MS. The features obtained by the two instances are concatenated directly, which can be treated as simple fusion features. To fully fuse the spatial–spectral information for further classification, the simple fusion features are fed into a fusion network with three fully connected layers to learn the high-level fusion features. Classification experiments carried out on four different airborne MS and PAN images indicate that the classifier provides feasible and efficient solution. It demonstrates that DMIL performs better than using a convolutional neural network and a stacked autoencoder network separately. In addition, this paper shows that the DMIL model can learn and fuse spectral and spatial information effectively, and has huge potential for MS and PAN imagery classification.

In this paper, a new pansharpening method is proposed by constructing a set of multiscale geometric support tensor filters (MGSTFs). First, a least-square ridgelet support tensor machine is developed to derive a series of MGSTFs. Then the source images are formulated as tensors and filtered by MGSTFs to capture geometric and salient features of images. These features are then fused at each scale and direction to obtain the fused products. The distortions can be reduced by exploring the tensor formulation of multispectral data and endowing the filters’ directionality to capture the geometric details of images. Some experiments are carried out on several groups of QuickBird and GeoEye-1 images, and the results show that our proposed method can simultaneously reduce spectral distortions and preserve spatial details in the fused image.

Under the umbrella of tensor algebra, this paper proposes a new sparse-coding-based classifier (SCC) for hyperspectral imagery classification (HIC). By utilizing the tensor forms of hyperspectral pixels, we advance a tensor sparse-coding model which preserves as many original spatial constraints of a pixel and its spatial neighbors as possible. Furthermore, to alleviate the classification uncertainty resulted from widely existing mixed pixels, this paper constructs a regularization term for maximizing the likelihood of sparse-coding tensor defined on the posterior class probability. By combining the tensor sparse coding with maximizing likelihood estimation, a hybrid probabilistic SCC with spatial neighbor tensor (HPSCC-SNT) is proposed, which makes the pixels be well represented by the training pixels belonging to the same class. The performance of HPSCC-SNT is evaluated on three real hyperspectral imagery data sets, and the results show that it can achieve accurate and robust HIC results, and outperforms the state-of-the-art methods.

This paper proposes a unified mutual learning framework based on image hierarchies, which integrates structured sparsity with tree-graph matching to conquer the problem of weakly supervised image cosegmentation. We focus on the interaction between two common-object properties: saliency and similarity. Most existing cosegmentation methods only pay emphasis on either of them. The proposed method realizes the learning of the prior knowledge for structured sparsity with the help of treegraph matching, which is capable of generating object-oriented salient regions. Meanwhile, it also reduces the searching space and computational complexity of tree-graph matching with the attendance of structured sparsity. We intend to thoughtfully exploit the hierarchically geometrical relationships of coherent objects. Experimental results compared with the state-of-thearts on benchmark datasets confirm that the mutual learning framework are capable of effectively delineating co-existing object patterns in multiple images.

Recognizing scenes from synthetic aperture radar (SAR) images has been a challenging task due to the increasing resolution of SAR data. Extracting discriminative features from SAR images is extremely difficult for their sensitivity to target aspect. Considering the intractability of the available deep neural networks in practical implementations, in this brief, we propose a simple and efficient deep sparse tensor filtering network (DSTFN) for SAR image classification. An SAR image is first organized into a data tensor by an overlapped partition. Then, a set of dimension-inseparable geometric filters is developed from a least squares support vector machine, followed by a learned sparse filtering of tensors. Finally, the constructed sparse tensor filters are cascaded to a deep network to automatically extract the discriminative features of the image for accurate classification. Simulations are carried out to verify the effectiveness of the proposed DSTFN.

Finding accurate injection components is the key issue in pan-sharpening methods. In this paper, a low-rank pan-sharpening (LRP) model is developed from a new perspective of offset learning. Two offsets are defined to represent the spatial and spectral differences between low-resolution multispectral and high-resolution multispectral (HRMS) images, respectively. In order to reduce spatial and spectral distortions, spatial equalization and spectral proportion constraints are designed and cast on the offsets, to develop a spatial and spectral constrained stable low-rank decomposition algorithm via augmented Lagrange multiplier. By fine modeling and heuristic learning, our method can simultaneously reduce spatial and spectral distortions in the fused HRMS images. Moreover, our method can efficiently deal with noises and outliers in source images, for exploring low-rank and sparse characteristics of data. Extensive experiments are taken on several image data sets, and the results demonstrate the efficiency of the proposed LRP.

The graph model-based semisupervised machine learning is well established. However, its computational complexity is still high in terms of the time consumption especially for large data. In this letter, we propose a fast semisupervised classification algorithm using the recently presented spatial-anchor graph for a large polarimetric synthetic aperture radar (Pol-SAR) data, named as Fast Spatial-Anchor Graph (FSAG) based algorithm. Based on an initial superpixel segmentation on the PolSAR image, the homogenous regions are obtained. The border pixels are reassigned to the most similar superpixel according to majority voting and distance measurement. Then, feature vectors are weighted within local homogenous regions. The refined spatial-anchor graph is constructed with these regions, and the semisupervised classification is conducted. Experimental results on synthesized and real PolSAR data indicate that the proposed FSAG greatly reduces time consumption and maintains the accuracy for terrain classifications compared with state-of-the-art graph-based approaches.

We propose a novel image representation framework based on Gaussian model and evolutionary optimization (EO). In this framework, image patches are categorized into smooth and nonsmooth ones, and the two categories are treated distinctively. For a smooth patch, we formulate it as the summation of a direct component and a variation component (VC). We observe that the values of all VCs in an image can be well fitted by a Gaussian distribution, according to which we present an efficient reconstruction approach based on maximizing the logarithm a posteriori probability. For a nonsmooth patch, we introduce the mechanism of EO to solve a combinatorial optimization over a principal component analysis dictionary. In addition, we develop two approaches for estimating the coefficients of the atoms. Experiment results demonstrate that the proposed framework obtains the state-of-the-art results in several image inverse problems.

Most of the existing spatial-spectral-based hyperspectral image classification (HSIC) methods mainly extract the spatial-spectral information by combining the pixels in a small neighborhood or aggregating the statistical and morphological characteristics. However, those strategies can only generate shallow appearance features with limited representative ability for classes with high interclass similarity and spatial diversity and therefore reduce the classification accuracy. To this end, we present a novel HSIC framework, named deep multiscale spatial-spectral feature extraction algorithm, which focuses on learning effective discriminant features for HSIC. First, the well pretrained deep fully convolutional network based on VGG-verydeep-16 is introduced to excavate the potential deep multiscale spatial structural information in the proposed hyperspectral imaging framework. Then, the spectral feature and the deep multiscale spatial feature are fused by adopting the weighted fusion method. Finally, the fusion feature is put into a generic classifier to obtain the pixelwise classification. Compared with the existing spectral-spatial-based classification techniques, the proposed method provides the state-of-the-art performance and is much more effective, especially for images with high nonlinear distribution and spatial diversity.

The increasing resolution of synthetic aperture radar (SAR) images makes ship targets appear to be more structured and shaped and nevertheless contain many weak echoes in their resolution cells, which brings great challenges for accurate scene understanding. In this paper, inspired by the multilayer selective cognition property of the human visual system, we advance a new hierarchical saliency filtering method for fast and accurate ship detection in high-resolution SAR images. The saliency of targets is first explored to develop a random-forest-based hierarchical sparse model (HSM) for the selection of candidate target regions. Then, a dynamic constant-false-alarm-rate-based contour saliency model (CSM) is proposed to gradually filter out the false alarms from candidate regions and extract the target outlines for accurate detection. Because of a rapid capture of regions of interest in the HSM and dynamic false alarm removal in the CSM, our method can make efficient ship detection in high-resolution SAR images possible by working in a coarse-to-fine manner. Finally, the proposed ship detector is tested on high-resolution SAR data collected from TerraSAR and RADARSAT satellites, showing significant agreement with the ground truth. It is also compared with other classical ship detectors, in terms of both speed and accuracy, and shows superior performance, particularly in complex scenes.

Fusing low spatial resolution hyperspectral (LRHS) images and high spatial resolution multispectral (HRMS) images to obtain high spatial resolution hyperspectral images (HRHS) has received increasing interests in recent years. In this paper, a new group spectral embedding (GSE)-based LRHS and HRMS image fusion method is proposed by exploring the multiple manifold structures of spectral bands and the low-rank structure of HRHS data. First, a low-rank factorization fusion (LRFF)-based robust recovery model is developed for HRHS images, by regarding HRMS images as the spectral degradation of HRHS images and exploring the group sparse prior of difference images. Then, an assumption that grouped spectral bands share the similar local geometry is cast on LRHS and HRHS images, to formulate a GSE regularizer in the LRFF model. Finally, an iterative optimization algorithm based on augmented Lagrangian multiplier is advanced to recover HRHS images. Experimental results on several data sets show the effectiveness of the proposed method on visual and numerical comparison.

Recently, very high resolution (VHR) panchromatic and multispectral (MS) remote-sensing images can be acquired easily. However, it is still a challenging task to fuse and classify these VHR images. Generally, there are two ways for the fusion and classification of panchromatic and MS images. One way is to use a panchromatic image to sharpen an MS image, and then classify a pan-sharpened MS image. Another way is to extract features from panchromatic and MS images, respectively, and then combine these features for classification. In this paper, we propose a superpixel-based multiple local convolution neural network (SML-CNN) model for panchromatic and MS images classification. In order to reduce the amount of input data for the CNN, we extend simple linear iterative clustering algorithm for segmenting MS images and generating superpixels. Superpixels are taken as the basic analysis unit instead of pixels. To make full advantage of the spatial-spectral and environment information of superpixels, a superpixel-based multiple local regions joint representation method is proposed. Then, an SML-CNN model is established to extract an efficient joint feature representation. A softmax layer is used to classify these features learned by multiple local CNN into different categories. Finally, in order to eliminate the adverse effects on the classification results within and between superpixels, we propose a multi-information modification strategy that combines the detailed information and semantic information to improve the classification performance. Experiments on the classification of Vancouver and Xi’an panchromatic and MS image data sets have demonstrated the effectiveness of the proposed approach.

A new method for Polarimetric Synthetic Aperture Radar (PolSAR) terrain classification based on Deep Sparse Filtering Network (DSFN) is proposed in this paper. It uses a novel deep learning network to learn features from the input raw data automatically. And the spatial information between pixels on PolSAR image is combined into the input data. Moreover, unlike the conventional deep networks, the DSFN only needs to tune very few parameters during pre-training and fine-tuning. A real PolSAR data is used to verify the proposed method. Experimental results show that the proposed DSFN is efficient with less parameters and effectively improves the classification accuracy compared with conventional deep networks.

Although the graph-based machine learning has received considerable attention in the remote sensing area and it has been widely used for terrain classification, the construction of graph in most existing algorithms still takes large memory and plenty of computational time especially for large Polarimetric Synthetic Aperture Radar (PolSAR) data. Addressing these issues, we propose a fast semi-supervised classification method based on parallel auction graph in this paper. The spatial relation between pixels is firstly preprocessed using the superpixel segmentation. Then we divide the PolSAR data into multiple groups, and each of them is used to construct a sparse auction graph. The semi-supervised classification is performed parallel on those graphs. Experimental results on simulated and real PolSAR data demonstrate its efficiency and effectiveness compared with existing methods.

A synthetic aperture radar (SAR) imaging system usually produces pairs of bright area and dark area when depicting the ground objects, such as a building or tree and its shadow. Many buildings (trees) are aggregated together to form urban areas (forests). It means that the pairs of bright and dark areas often exist in the aggregated scenes. Conventional unsupervised segmentation approaches usually segment the scenes (e.g., urban areas and forests) into different regions simply according to the gray values of the image. However, a more convincing way is to regard them as the consistent regions. In this paper, we aim at addressing this issue and propose a new SAR image segmentation approach via a hierarchical visual semantic and adaptive neighborhood multinomial latent model. In this approach, the hierarchical visual semantic of SAR images is proposed, which divides SAR images into aggregated, structural, and homogeneous regions. Based on the division, different segmentation methods are chosen for these regions with different characteristics. For the aggregated region, locality-constrained linear coding-based hierarchical clustering is used for segmentation. For the structural region, visual semantic rules are designed for line object location, and a geometric structure window-based multinomial latent model is proposed for segmentation. For the homogeneous region, a multinomial latent model with adaptive window selection is proposed for segmentation. Finally, these results are integrated together to obtain the final segmentation. Experiments on both synthetic and real SAR images indicate that the proposed method achieves promising performances in terms of the consistencies of the regions and the preservations of the edges and line objects.

Inspired by a popular deep neural network, i.e., deep belief network (DBN), a novel method for polarimetric synthetic aperture radar (POL-SAR) image classification is proposed in this paper. For the particularity of POL-SAR data, a new type of restricted Boltzmann machine (RBM) is specially defined, which we name the Wishart-Bernoulli RBM (WBRBM), and is used to form a deep network named as Wishart DBN (W-DBN). Numerous unlabeled POL-SAR pixels are made full use of in the modeling of POL-SAR pixels by W-DBN. In addition, the coherency matrix is used directly to represent a POL-SAR pixel without any manual feature extraction, which is simple and time saving. Local spatial information, together with the confusion matrix, is used in this paper to clean the preliminary classification result obtained by the method based on W-DBN. Making full use of the prior knowledge of POL-SAR data and local spatial information, the proposed method overcomes shortcomings of traditional methods, in which they are sensitive to extracted features and slow to execute. The experiments, tested on three POL-SAR data sets, show that the proposed method produces better results and is much faster than traditional methods.

Recently a few works of semi-supervised learning methods based on graph have been proposed for remote sensing. The common idea of these methods are that they build a graph using the samples of the image. Most of their time complexity is relatively large, and they ignore the spatial information of the image, which leads to unsatisfactory classification results. this paper proposes a novel semi-supervised classification method based on anchor-spatial graph for large PolSAR data. Firstly the unsupervised Wishart clustering is performed to select representative samples, which served as anchors according to the least distance between samples. Then an anchor graph is built using the selected anchors according to the multiple features of the samples. And it is further combined with the spatial information of the samples to construct an anchor-spatial graph. Finally the class information from small quantities of labeled samples propagates to the unlabeled ones. Experimental results show that the proposed method has a low time complexity compared with existing works and it could effectively cut down the processing time for large PolSAR data meanwhile keeps the classification accuracy.

The past few years have witnessed prosperity of spectral-spatial processing of hyperspectral images. In this letter, in order to determine the optimal projection subspace of spectrums, we define discriminate spectral-spatial margins (DSSMs) to reveal the local information of hyperspectral pixels and explore the global structures of both labeled and unlabeled data via low-rank representation (LRR). Heterogeneous and homogeneous spectral-spatial neighbors of hyperspectral pixels are used to define DSSMs. By maximizing the DSSM of hyperspectral data and casting an LRR manifold regularizer on finding better projection, both the local and global information of hyperspectral data can be well explored to determine more discriminative features. Some experiments are taken on several real hyperspectral data sets, and the results exhibit its efficiency and superiority to the counterparts, when only a small number of labeled samples are available.

Component substitution (CS) technique is a famous framework for merging multispectral (MS) and panchromatic (Pan) images. The synthetic intensity component is important in the CS fusion framework. In this letter, we propose an optimization model to obtain the adaptive weights. The adaptive weights are computed by maximizing an objective function, which measures the radiometric similarity between the low-scale intensity image and the spatially degraded Pan image. Correlation coefficient, mean-structural-similarity index, and mutual information are used as the similarity criteria, respectively. A particle-swarm-optimization algorithm is adopted to solve the single objection optimization problem. The proposed CS framework is compared with popular CS-based fusion methods. Visual analysis and quality results demonstrate that the proposed adaptive CS fusion framework has superior performance.

The l 0 regularized problem in compressed sensing reconstruction is nonconvex with NP-hard computational complexity. Methods available for such problems fall into one of two types: greedy pursuit methods and thresholding methods, which are characterized by suboptimal fast search strategies. Nature-inspired algorithms for combinatorial optimization are famous for their efficient global search strategies and superior performance for nonconvex and nonlinear problems. In this paper, we study and propose nonconvex compressed sensing for natural images by nature-inspired optimization algorithms. We get measurements by the block-based compressed sampling and introduce an overcomplete dictionary of Ridgelet for image blocks. An atom of this dictionary is identified by the parameters of direction, scale and shift. Of them, direction parameter is important for adapting to directional regularity. So we propose a two-stage reconstruction scheme (TS_RS) of nature-inspired optimization algorithms. In the first reconstruction stage, we design a genetic algorithm for a class of image blocks to acquire the estimation of atomic combinations in all directions; and in the second reconstruction stage, we adopt clonal selection algorithm to search better atomic combinations in the sub-dictionary resulted by the first stage for each image block further on scale and shift parameters. In TS_RS, to reduce the uncertainty and instability of the reconstruction problems, we adopt novel and flexible heuristic searching strategies, which include delicately designing the initialization, operators, evaluating methods, and so on. The experimental results show the efficiency and stability of the proposed TS_RS of nature-inspired algorithms, which outperforms classic greedy and thresholding methods.

Recently, compressive hyperspectral imaging (CHI) has received increasing interests, which can recover a large range of scenes with a small number of sensors via compressed sensing (CS) theory. However, most of the available CHI methods separate and vectorize hyperspectral cubes into spatial and spectral vectors, which will result in heavy computational and storage burden in the recovery. Moreover, the complexity of real scene makes the sparsifying difficult and thus requires more measurements to achieve accurate recovery. In this paper, these two issues are addressed, and a new CHI approach via sparse tensors and nonlinear CS (NCS) is advanced for accurate maintenance of image structure with limited number of sensors. Based on a multidimensional multiplexing (MDMP) CS scheme, the observed measurements are denoted as tensors and a nonlinear sparse tensor coding is adopted, to develop a new tensor-NCS (T-NCS) algorithm for noniterative recovery of hyperspectral images. Moreover, two recovery schemes are advanced for T-NCS, including example-aided and self-learning CHI approaches. Finally, some experiments are performed on three real hyperspectral data sets to investigate the performance of T-NCS, and the results demonstrate its efficiency and superiority to the counterparts.

Parametric probability models are common references for channel characterization. However, the limited number of samples and uncertainty of the propagation scenario affect the characterization accuracy of parametric models for body area networks. In this paper, we propose a sparse nonparametric probability model for body area wireless channel characterization. The path loss and root-mean-square delay, which are significant wireless channel parameters, can be learned from this nonparametric model. A comparison with available parametric models shows that the proposed model is very feasible for the body area propagation environment and can be seen as a significant supplement to parametric approaches.

In conventional terrain classification for the polarimetric SAR (POLSAR) images, color features are rarely involved unless in one recent supervised work. Unlike that work, the color features are exploited for the unsupervised classification in this paper. Firstly, based on the polarimetric decomposition of the POLSAR data, the common color spaces, such as RGB, HSI, and CIELab are calculated. The color feature is quantitatively selected from these color spaces by introducing the color entropy. Then together with the spatial information, extended scattering power entropy and the copolarized ratio, the adaptive Mean-shift algorithm is used to segment the POLSAR image. Finally, the segments are merged according to the Wishart distance measurement. The experiments using AIRSAR L-band POLSAR data indicate that the proposed method has better discriminative ability for urban areas and for boundary preservation compared with existing works.

This paper proposes a novel ship detection scheme in coastal regions for high-resolution synthetic aperture radar (SAR) imagery based on prior knowledge of the different properties presented by target and clutter. To begin with, image segmentation and land masking are applied to eliminate the areas that are unlikely to contain targets and get the index image which indicates the likely target positions. Ship detection is conducted only on these likely target positions using power ring algorithm (PR), which can avoid unnecessary and exhaustive searches. In the discrimination stage, two new features named number of 8 connected regions and average power of target areas are proposed and used to form a discriminative feature group. Unlike most discriminators, which are based on supervised learning, we use an unsupervised method based on K-means clustering to deal with the situations where there are few or no labeled samples. Experimental results show that the proposed scheme is fast in speed and can detect most of the targets while few false alarms occur.

The recently developed Compressed Sensing (CS) theory has made the super-resolution of spectrum estimation possible. In this paper, we exploit the joint sparsity of received signals to develop a new Compressive Direction-of-Arrival Estimation approach via a new Regularized Multiple Measurment FOCal Underdetermined System Solver (RMM-FOCUSS) Algorithm. It can overcome the resolution limitation of traditional spatial energy spectrum estimation algorithm, such as MUSIC algorithm, and present more accurate estimation of direction of multiple sources when there are a few numbers of antenna units. Some experiments are taken to validate the performance of our proposed method.

Advances in machine learning technology have made efficient Super-Resolution Image Reconstruction (SRIR) possible. In this paper, we advance a hierarchical support vector machine (HSVM) to learn representative features of both training and test Low-Resolution (LR) image patches. Then a sparse manifold assumption is cast on training patch features to find local HR neighbors for each test LR input. The reconstructed High-Resolution (HR) patches can then be derived via Neighbors Embedding (NE) technology with the help of the HR neighbors from training HR patches, and compensated for the LR images. Some experiments are taken on realizing a 3X amplification of natural images, the recovered results prove its efficiency and superiority to its counterparts visually and qualitatively.

Exploring the sparsity in classifying hyperspectral vectors proves to lead to state-of-the-art performance. To learn a compact and discriminative dictionary for accurate and fast classification of hyperspectral images, a data-driven Compressive Sampling (CS) and learning sparse coding scheme are use to reduce the dimensionality and size of the dictionary respectively. First, a sparse radial basis function (RBF) kernel learning network (S-RBFKLN) is constructed to learn a compact dictionary for sparsely representing hyperspectral vectors. Then a data-driven compressive sampling scheme is designed to reduce the dimensionality of the dictionary, and labels of new samples are derived from coding coefficients. Some experiments are taken on NASA EO-1 Hyperion data and AVIRIS Indian Pines data to investigate the performance of the proposed method, and the results show its superiority to its counterparts.

In this letter we explore the sparse sensing and learning mechanism of the human visual system, to propose a dual-sparse single-hidden-layer feedforward neural network (SLFNN) for the hyperspectral imagery classification. Firstly a large SLFNN is randomly initialized and trained by an extreme learning algorithm, and then the input and hidden neurons are simultaneously reduced by imposing a sparse constraint on the weights of the network. Then a saliency map is derived via the recent developed compressive sensing theory, and a joint sparse optimization approach is proposed to realize a one-step rapid selection of significant neurons. The reduction of input neurons can realize an automatic band-subset-selection of hyperspectral bands to remove the redundancy of hyperspectral vectors, and the reduction of hidden neurons can avoid the high computational cost at runtime and potential overfitting. Some experiments are taken on AVIRIS imagery data to investigate the performance of the proposed method, and the results show that it can achieve accurate and rapid classification.

In this letter, a relaxed clustering assumption and spatial Laplace-regularizer-based semisupervised hyperspectral image classifier is proposed. Considering the mixed pixels and noise intrinsic in hyperspectral image, we relax the clustering assumption employed in most of the available classifiers so that the similar hyperspectral vectors tend to share the “similar” labels instead of the “same” label, to formulate a modified spectral similarity regularizer. Moreover, the spatial homogeneity assumption is cast on hyperspectral pixels to construct a spatial regularizer, to overcome the salt-and-pepper misclassification of images. The effectiveness of our proposed method is evaluated via experiments on AVIRIS data, and the results show that it exhibits state-of-the-art performance, particularly when there are a small number of training samples.

In this letter, we propose a new spatio-spectral Laplacian support vector machine (SS-LapSVM) for semi-supervised hyperspectral image classification. The clustering assumption on spectral vectors is used to formulate a manifold regularizer, and neighborhood spatial constraints of hyperspectral images are designed to construct a spatial regularizer. Moreover, a non-iterative optimization procedure is presented to solve this dual-regularized SVM, which makes rapid classification possible. By combining spatial and spectral information together, SS-LapSVM can avoid the speckle-like misclassification of hyperspectral images in the original Lap-SVM. The performance of SS-LapSVM is evaluated on AVIRIS image data taken over Indiana's Indian Pine, and the results show that it can achieve accurate and rapid classification with a small number of labeled data, and outperform state-of-the-art semi-supervised approaches.

In order to improve the classification performance for hyperspectral image (HSI), a sparse representation classifier based on spectral information divergence (SID) is proposed. SID measures the discrepancy of probabilistic behaviors between the spectral signatures of two pixels from the aspect of information theory, which can be more effective in preserving spectral properties. Thus, the new method measures the similarity between the reconstructed pixel and the true pixel by SID instead of by the L2 norm used in traditional sparse model. Moreover, the spatial coherency across neighboring pixels sharing a common sparsity pattern is taken into account during the construction of SID-based joint sparse representation model. We propose a new version of the orthogonal matching pursuit method to solve SID-based recovery problems. The proposed SID-based algorithms are applied to real HSI for classification. Experimental results show that our algorithms outperform the classical sparse representation based classification algorithms in most cases.

Exploring the geometric prior in the dimensionality reduction (DR) of hyperspectral image data (HID) is an important issue because it can overcome the possible overclassification of spectrally homogeneous areas in the HID classification. In this paper, the local geometric similarity of hyperspectral vectors is explored in both the manifold domain and image domain, and a semisupervised dual-geometric subspace projection (DGSP) approach is proposed for the DR of HID, by utilizing both labeled and unlabeled samples. First, the geometric information in the manifold domain is captured by a sparse coding-based geometric graph, and then, a local-consistency-constrained geometric matrix is defined to reveal the geometric structure in the image domain. Second, unlabeled samples are used to refine the geometric structure by defining a pairwise similarity matrix. Third, three scatter matrices are then derived from these similarity matrices to find the optimal subspace projection that captures the most important properties of the subspaces with respect to classification. Some experiments are taken on the airborne visible infrared imaging spectrometer (AVIRIS) HID to prove the efficiency of the proposed method.

Neighbors embedding (NE) technology has proved its efficiency in single image super resolution (SISR). However, image patches do not strictly follow the similar structure in the low-resolution and high-resolution spaces, consequently leading to a bias to the image restoration. In this paper, considering that patches are a set of data with multiview characteristics and spatial organization, we advance a dual-geometric neighbor embedding (DGNE) approach for SISR. In DGNE, multiview features and local spatial neighbors of patches are explored to find a feature-spatial manifold embedding for images. We adopt a geometrically motivated assumption that for each patch there exists a small neighborhood in which only the patches that come from the same feature-spatial manifold, will lie approximately in a low-dimensional affine subspace formulated by sparse neighbors. In order to find the sparse neighbors, a tensor-simultaneous orthogonal matching pursuit algorithm is advanced to realize a joint sparse coding of feature-spatial image tensors. Some experiments are performed on realizing a 3X amplification of natural images, and the recovered results prove its efficiency and superiority to its counterparts.

In this study, the authors propose a new semi‐supervised low‐rank representation graph for pattern recognition. A collection of samples is jointly coded by the recently developed low‐rank representation (LRR), which better captures the global structure of data and implements more robust subspace segmentation from corrupted samples. By using the calculated LRR coefficients of both labelled and unlabelled samples as the graph weights, a low‐rank representation graph is established in a parameter‐free manner under the framework of semi‐supervised learning. Some experiments are taken on the benchmark database to investigate the performance of the proposed method and the results show that it is superior to other related semi‐supervised graphs.

Advances in machine learning technology have made efficient image denoising possible. In this paper, we propose a new ridgelet support vector machine (RSVM) for image noise reduction. Multiscale ridgelet support vector filter (MRSVF) is first deduced from RSVM, to produce a multiscale, multidirection, undecimated, dyadic, aliasing, and shift-invariant geometric multiscale ridgelet support vector transform (GMRSVT). Then, multiscale dictionaries are learned from examples to reduce noises existed in GMRSVT coefficients. Compared with the available approaches, the proposed method has the following characteristics. The proposed MRSVF can extract the salient features associated with the linear singularities of images. Consequently, GMRSVT can well approximate edges, contours and textures in images, and avoid ringing effects suffered from sampling in the multiscale decomposition of images. Sparse coding is explored for noise reduction via the learned multiscale and overcomplete dictionaries. Some experiments are taken on natural images, and the results show the efficiency of the proposed method.

In this paper, we propose a sparse kernel representation classification algorithm (SKRC) for images classification and recognition. The training dictionary is composed by labeled samples directly, and both training dictionary and testing sample are mapped into feature space from original sample space by the sparse kernel which employs the “center” samples matrix constructed by a method similar to k-means clustering. Then in the feature space, the basic sparse representation based classification method is employed. We test our proposed algorithm on some different public database, and the results show that our proposed method can achieve higher classification accuracy without much time consumed.

Recently, single image super-resolution reconstruction (SISR) via sparse coding has attracted increasing interest. In this paper, we proposed a multiple-geometric-dictionaries-based clustered sparse coding scheme for SISR. Firstly, a large number of high-resolution (HR) image patches are randomly extracted from a set of example training images and clustered into several groups of "geometric patches," from which the corresponding "geometric dictionaries" are learned to further sparsely code each local patch in a low-resolution image. A clustering aggregation is performed on the HR patches recovered by different dictionaries, followed by a subsequent patch aggregation to estimate the HR image. Considering that there are often many repetitive image structures in an image, we add a self-similarity constraint on the recovered image in patch aggregation to reveal new features and details. Finally, the HR residual image is estimated by the proposed recovery method and compensated to better preserve the subtle details of the images. Some experiments test the proposed method on natural images, and the results show that the proposed method outperforms its counterparts in both visual fidelity and numerical measures.

In this paper, a directional multi-resolution ridgelet network (DMRN) is proposed based on ridgelet theory. By using ridgelet as the activation function, DMRN has great capabilities in catching essential features of "direction-rich" data for its multi-resolution property in direction besides scale and position. It proves to be able to approximate any multivariate function in a more stable and efficient way, and is optimal in approximating functions with spatial inhomogeneities. Using binary ridgelet frame for its design, DMRN is characteristic of more flexible structure. Possibilities of applications to regression and recognition are included to demonstrate its superiority.

In this paper, a ridgelet kernel approach is proposed for approximation of multivariate functions, especially those with certain kinds of spatial inhomogeneities. It is based on ridgelet theory, kernel and regularization technology from which we can deduce a regularized kernel regression form. Taking the objective function solved by quadratic programming to define a fitness function, we use particle swarm optimization algorithm to optimize the directions of ridgelets. Experiments in the tasks of regression prove its efficiency.

The particle swarm optimization algorithm is a new methodology in evolutionary computation. It has been found to be extremely effective is solving a wide range of engineering problems, however, it is of low efficiency in dealing with the discrete problems. In this paper, a new discrete particle swarm optimization algorithm based on quantum individual is proposed. It is simpler and more powerful than the algorithms available. The simulation experiments and its application in the CDMA also prove its high efficiency.

In this paper, an algorithm - the quantum evolutionary algorithm (QEA) is introduced. It is characterized by a representation of quantum chromosomes, quantum mutation and quantum crossover. Its advantages lies on better diversity of individuals, effective guidance of mutation and the avoidance of prematurity by crossover. Some simulations are given to illustrate its efficiency and better performance than its counterpart. Finally, we applied it to the multi-user detection in DS-CDMA, and good results are attained.