The booming remote sensing (RS) technology is giving rise to a novel multimodality generalization task, which requires the model to overcome data heterogeneity while possessing powerful cross-scene generalization ability. Moreover, most vision-language models usually describe surface materials using universal texts, lacking proprietary linguistic prior knowledge specific to different RS modalities. In this work, we formalize RS multimodality generalization (RSMG) as a learning paradigm, and propose a frequency-aware vision-language multimodality generalization network (FVMGN) for RS image classification. Specifically, a diffusion-based training-test-time augmentation (DTAug) strategy is designed to reconstruct multimodal land-cover distributions, enriching input information for FVMGN. Following that, to overcome multimodal heterogeneity, a multimodal wavelet disentanglement (MWDis) module is developed to learn cross-domain invariant features by resampling low and high frequency components in the frequency domain. Considering the characteristics of RS vision modalities, shared and proprietary class texts is designed as linguistic inputs for the transformer-based text encoder to extract diverse text features. For multimodal vision inputs, a spatial-frequency-aware image encoder (SFIE) is constructed to realize local-global feature reconstruction and representation. Finally, a multiscale spatial-frequency feature alignment (MSFFA) module is suggested to construct a unified semantic space, ensuring refined multiscale alignment of different text and vision features in spatial and frequency domains. Extensive experiments show that FVMGN has the excellent multimodality generalization ability compared with state-of-the-art methods.

Multimodal remote sensing (RS) images exhibit distinct structure and distribution characteristics, making it challenging to design an effective multimodal RS image classification algorithm. Moreover, although existing deep learning-based methods have become the darling in the multimodal RS image classification, they usually lack effective exploration and explicit integration for generative information from different modalities. Aiming at the above challenges, a generative information-guided heterogeneous cross-fusion network with contrastive learning (GIHCN) is proposed for multimodal RS image classification. Firstly, to simulate the land-cover distributions from different modal data, a multimodal generative information learning architecture (MGILA) is constructed to capture the unsupervised heterogeneous distribution features. Secondly, to achieve bidirectional modeling between heterogeneous data and the reconstructed land-cover distributions, a heterogeneous data & generative information cross-attention module (HGCM) is designed to explore the complementarity between multimodal data and the reconstructed land-cover distributions. HGCM can provide the heterogeneous generative information for current modal data or provide the heterogeneous data support for current modal generative information, thereby obtaining cross-fusion sources with different attributes. Furthermore, we achieve the effective feature extraction for different cross-fusion sources by a designed multimodal contrastive learning framework (MCLF). Notably, to capture local information and long-range dependencies, a hybrid classification network with convolutional neural network and Mamba (CMNet) is proposed as the feature extraction backbone of each cross-fusion source to further improve the classification performance. Finally, we construct a joint multimodality loss function for MCLF, which can reduce the distribution difference between modalities while focusing on the information flow within and across the modality. Experimental results on four multimodal RS datasets confirm the effectiveness of GIHCN compared with other state-of-the-art methods. The source code will be released at https://github.com/ZJier/GIHCN.

Transformers have been widely adopted in the field of hyperspectral image (HSI) classification. However, a significant drawback of transformers lies in their excessive number of parameters and the high computational overhead. To address this challenge, we propose a Channel-Reduced Transformer (CRFormer) for HSI classification. In an effort to enhance computational efficiency, we first introduce a cross-region tokenization (CRT) approach. This method effectively shortens the sequence length input to the transformer, thereby alleviating the computational burden. Additionally, we propose a channel-reduced multi-head self-attention (CR-MHSA) module. This module operates on only half of the input channels while still attaining comparable or even superior results. Experimental results conducted on three benchmark datasets demonstrate that our proposed method not only achieves superior classification accuracy but also significantly reduces computational complexity compared to other transformer-based approaches.

Micro-actions convey the emotions of characters in daily communication and offer richer semantic information compared to conventional actions. Accurate detection of these micro-actions is essential for video understanding. Due to their short duration, low intensity, and high overlap, micro-actions require more detailed video features, presenting a significant challenge for accurate detection. To address these challenges, we propose the 3D-SENet Adapter, which aggregates spatio-temporal information and enables end-to-end online video feature learning. We also find that incorporating background information significantly enhances the detection of small-scale micro-actions. Thus we develop the Cross-Attention Aggregation Detection Head, which integrates multi-scale features within the feature pyramid, thereby improving the detection accuracy of micro-actions occupying small regions in video frames. Our approach achieves first place in the Multi-label Micro-Action Detection (MMAD) and second place in the Micro-Action Recognition (MAR) of Micro-Action Analysis Grand Challenge.

The global modeling capability of vision transformer (ViT) has been well proven in the field of hyperspectral image (HSI) classification. However, ViT does not have the excellent local feature extraction capability compared with the convolutional neural network (CNN). Therefore, early-stage convolutions are often used to enhance ViT’s local representation ability. However, directly applying convolutions on high-dimensional HSI data increases computational overhead. Moreover, recent researches have observed that ViT may suffer from high redundancy in capturing multihead self-attention (MHSA). To address the above issues, we propose a multiscale super token transformer (MSSTT) model for HSI classification. We use a divide-and-conquer strategy to extract local features and global dependencies of HSI data at multiple granularities. Specifically, our proposed model incorporates two branches: a multiscale convolution (MSConv) branch that uses various convolutional kernels to extract diverse local features and a multiscale super token attention (MSSTA) branch for capturing global features with low redundancy. Finally, comparative experimental results with advanced methods show that the proposed MSSTT possesses better classification performance. On the Salinas (SA), Pavia University (PU), and Kennedy Space Center (KSC) datasets, the overall accuracies (OAs) of our MSSTT are 98.47%, 98.47%, and 99.38%, respectively. Code will be released athttps://github.com/zhangtaizheng/MSSTT.

Deep learning algorithms have shown significant advantages in hyperspectral image (HSI) classification. However, these algorithms usually require a large number of labeled samples and the annotation of these samples consumes massive time and resource costs. To achieve effective classification results in situations with small samples, a semi-supervised co-training model using convolution and transformer (SCM-CT) is proposed in this letter. Firstly, two different networks, namely multi-scale parallel CNN (MPCNN) and global and local transformer fusion network (GLTFN), are designed as co-training learners to extract multi-scale spectral-spatial features and global-local combined features in HSIs, respectively. Secondly, to ensure two learners generate reliable predictions and utilize more unlabeled samples with low confidence pseudo-labels, a self-adaptive threshold and conflict pseudo-labeling (SATCP) strategy is proposed to facilitate the model to learn more valuable spectral-spatial information from conflict predictions and improve the convergence speed and model performance. Finally, to prevent the learners from stepping into the collapse, the discrepancy loss is computed to reduce the similarity between the features extracted by the two learners, forcing them to learn different information from the same input. Experimental results on University of Pavia, Salinas Valley, and Houston 2013 datasets show that SCM-CT achieves overall accuracies of 97.42%, 95.60%, and 95.20%, respectively, outperforming the state-of-the-art methods.

The advancement of deep learning technology has significantly improved the performance of high-resolution remote sensing (HRRS) image change detection (CD) task. HRRS images on CD are usually captured under varying conditions, which may generate pseudo-changes caused by seasonal variations, changes in lighting angles, and object motion. These factors can interfere with the effective modeling of difference information. However, conventional methods of modeling difference information may not adequately address these challenges, leaving them susceptible to such factors. Therefore, this article proposes a multiscale alignment and progressive feature fusion network (MAPNet) based on convolutional neural network (CNN) and transformer, which can effectively model difference information. First, a flow- and attention-guided bitemporal alignment module (FA-BAFM) is designed to align bitemporal features and capture the differences between them. Second, a progressive difference feature fusion module (PDFFM) is developed to comprehensively fuse the difference information across various scales and levels. Finally, a local feature enhancement module (LFEM) is constructed to improve the ability of the backbone to extract both global and local features. Experimental results on the learning, vision, and remote sensing CD (LEVIR-CD), Wuhan University (WHU), and Sun Yat-Sen University CD (SYSU-CD) datasets show that MAPNet achieves F1-scores of 91.90%, 94.19%, and 83.69%, respectively, outperforming the state-of-the-art (SOTA) methods. The demo code will be released athttps://github.com/zlin9/mapnet.

Multiobjective clustering algorithms (MOCAs) are becoming increasingly popular with the merit of segmenting images from multiple perspectives. The performances of MOCAs highly depend on their fitness functions. However, most existing MOCAs adopt one pair of complementary fitness functions, which always measure intra-class compactness and inter-class separation, respectively. This may result in insufficient ability to recognize and mine feature structures from complex images. Moreover, information within color images, such as region information and uncertain information, can barely receive enough attention in MOCAs. To resolve these problems, we propose a reliable region information driven Kriging-assisted multiobjective rough fuzzy clustering algorithm (RRI-KMRFC). Firstly, a reliability-based region information extraction strategy (RRIES) is designed to obtain reliable image information with satisfactory regional homogeneity and abundant image details. Secondly, the derived region information is used to construct three complementary fitness functions, focusing on rough intra-class compactness, rough inter-class separation, and regional consistency, respectively. Such fitness functions can effectively identify the clustering structure, maintain contour details, and characterize uncertain information from color images. To efficiently optimize the proposed functions, an incremental Kriging-assisted evolutionary framework is presented to decrease the expensive function evaluations in which an improved infill sampling strategy is devised to assist in finding unexplored areas in the decision space. Finally, a rough fuzzy clustering validity index with reliable region information is proposed to select the optimal trade-off solution. Experiments performed on Berkeley and Weizmann images confirm the effectiveness and robustness of RRI-KMRFC.

Convolutional neural networks (CNNs) have attained remarkable performance in hyperspectral image (HSI) classification owing to excellent locally modeling ability. However, the existing CNNs cannot capture global context information from HSI. Recently, vision transformer (ViT) has been proven to be effective in the image field. However, its retrieval of local space information in HSI classification is not satisfactory, and the input mode always leads to the loss of spatial location information and local information. In this letter, we propose a novel convolution transformer fusion splicing network (CTFSN) for HSI classification. From the perspective of local information and global information, this method adopts two feature fusion ways of addition and channel stacking to capture hyperspectral features. First, to effectively utilize shallow features and preserve spatial location information, we propose a residual splicing convolution block to serialize HSI. In addition, the convolutional transformer fusion block (CTFB) is designed to achieve additional local modeling on the basis of capturing global features. Finally, the dual branch fusion splicing module is adopted to fuse and splice the local features from the depthwise residual block and the global features from CTFB. Experimental results on three widely used datasets show that our method is superior to several other state-of-the-art classification methods.

Recently, convolutional neural networks (CNNs) show excellent performance on the hyperspectral image (HSI) classification tasks. However, traditional CNNs usually have insufficient feature discrimination and a large number of network parameters. In response to the above problems, a residual dense asymmetric convolutional network (RDACN) for HSI classification is proposed in this paper. Firstly, we de-sign a novel residual dense asymmetric convolutional block to effectively leverage the information of the previous layers. Moreover, the block adopts two feature fusion methods of addition and channel stacking to capture discriminative hyperspectral feature. Secondly, the ordinary square convolutional kernel is replaced with the asymmetric convolutional kernels, which can reduce CNN parameters. Finally, experimental results on three well-known hyperspectral datasets show that RDACN achieves competitive classification performance compared with the state-of-the-art CNNs.

When we use deep learning to classify PolSAR images, the lack of labeled samples will affect the classification performance. This paper uses the ACGAN model to expand the training samples of PolSAR data, utilizing the generated samples and original samples together to train CNN for PolSAR image classification. In addition to random noise, the ACGAN model also inputs additional relevant guidance information to ensure that the new generated samples are more similar to the original data. Compared with GAN, the discriminator of ACGAN can distinguish not only whether the data is real or not, but also the class label of the data. Subsequently, the validity of our proposed method is verified on the San Francisco data set. Compared with other classical PolSAR classification methods, the accuracy of our proposed method is improved.

The concept of intuitionistic fuzzy set has been found to be highly useful to handle vagueness in data. Based on intuitionistic fuzzy set theory, intuitionistic fuzzy clustering algorithms are proposed and play an important role in image segmentation. However, due to the influence of initialization and the presence of noise in the image, intuitionistic fuzzy clustering algorithm cannot acquire the satisfying performance when applied to segment images corrupted by noise. In order to solve above problems, a robust intuitionistic fuzzy clustering with bias field estimation (RIFCB) is proposed for noisy image segmentation in this paper. Firstly, a noise robust intuitionistic fuzzy set is constructed to represent the image by using the neighboring information of pixels. Then, initial cluster centers in RIFCB are adaptively determined by utilizing the frequency statistics of gray level in the image. In addition, in order to offset the information loss of the image when constructing the intuitionistic fuzzy set of the image, a new objective function incorporating a bias field is designed in RIFCB. Based on the new initialization strategy, the intuitionistic fuzzy set representation, and the incorporation of bias field, the proposed method preserves the image details and is insensitive to noise. Experimental results on some Berkeley images show that the proposed method achieves satisfactory segmentation results on images corrupted by different kinds of noise in contrast to conventional fuzzy clustering algorithms.

Convolutional neural networks (CNNs) showed impressive performance for hyperspectral image (HSI) classification. Nevertheless, convolutional layers contain massive parameters, which restrict the deployment of CNNs on satellite and airborne platforms with limited storage and computing resources. In this letter, we propose a lightweight spectral-spatial convolution module (LS2CM) as an alternative to the convolutional layer. The proposed LS2CM can greatly reduce network parameters and computational complexity in terms of multiply-accumulate operations (MACs) while maintaining or even improving the classification performance. Furthermore, it is a plug-and-play component and can be used to upgrade existing CNN-based models for HSI classification. Experimental results on two benchmark HSI data sets demonstrate that the proposed LS2CM achieves competitive results in comparison with other state-of-the-art methods.

Hyperspectral image (HSI) can provide rich spectral information which can be helpful for accurate classification in many applications. Yet, incorporating spatial information in the classification process can improve the classification accuracy even further. Existing convolutional neural network (CNN) usually only focuses on local features in hyperspectral cubes, whereas the burgeoning vision transformer (ViT) is interested in global features in HSIs. In this letter, we propose a deep aggregated framework for HSI classification called convolution transformer mixer (CTMixer) to combine the advantages of the above two paradigms effectively. A group parallel residual block is firstly applied to capture local spectral-spatial features in the HSI patches. Secondly, a double-branch structure, consisting of the CNN and transformer branches, is developed to capture local-global hyperspectral features. Finally, to achieve an elegant combination of CNN and ViT, a novel local-global multi-head self-attention mechanism is proposed by introducing convolution operations in the multi-head self-attention mechanism to further improve the classification accuracy. Extensive experiments demonstrate that the CTMixer achieves competitive classification results on several common HSI datasets compared with other state-of-the-art networks. The source code for this work will be available at https://github.com/ZJier/CTMixer.

Rough clustering has attracted increasing attention due to well dealing with the fuzziness and uncertainty of data. It is well known that it needs to manually set the threshold to determine the upper and lower approximations of rough clusters, which may bring a great effect on the clustering performance. When applied to image segmentation, rough clustering is always sensitive to the initialized cluster centers and image noise. Furthermore, only one clustering criterion is considered in rough clustering, which cannot satisfy diverse practical requirements. To handle these issues, a particle competitive mechanism based multiobjective rough clustering algorithm (PCM-MORCA) for image segmentation is proposed. First, a rough intraclass compactness function considering the nonlocal spatial information derived from an image is constructed to overcome the sensitivity to image noise. Next, the constructed rough intraclass compactness function and an interclass separation function are optimized simultaneously to make cluster centers meet diverse segmentation requirements. Then, an adaptive threshold determination mechanism by which the threshold adaptively varies with the clustered data is presented to well determine the upper and lower approximations of rough clusters. After that, to effectively search appropriate cluster centers, a novel pair competition-based particle weight updating strategy is designed for multiobjective particle swarm optimization by improving the elite particle selection and particle update. Finally, a rough clustering index with the nonlocal spatial information is constructed for selecting the optimal solution for PCM-MORCA. Segmentation experiments on Berkeley and magnetic resonance images reveal that PCM-MORCA behaves well on the segmentation accuracy and noise robustness.

Deep learning-based PolSAR image classification models have obtained great performance. However, they require large-scale labeled samples for training. Therefore, the deficient labeled samples is a significant challenge. In this paper, we propose a deep co-training network for PolSAR image classification, which introduces the co-training into the deep networks and then both labeled and unlabeled pixels can be used in a semi-supervised way. Firstly, the deep co-training network is established by applying the convolutional neural network and complex-valued 3D convolution neural network as two base classifiers according to the characteristics of PolSAR data. Then a high-confidence sample selection strategy is proposed by applying a super-pixel restrained strategy in the co-training process and the reliability of the selected unlabeled samples are further enhanced. Experimental results show that the proposed method can obtain high classification accuracy with much less labeled samples.

The deep learning has been applied to PolSAR image classification tasks in many researches. But they are pixel-based or spatial-based. None of them take the polarimetric semantic information into consideration. In this paper, a hierarchical classification method based on the polarimetric semantic information and deep learning is proposed. The method combines deep learning and semantic model to benefit from both the discriminative deep features and the polarimetric semantic information. The semantic information can provide the priori knowledge to improve the classification result of deep learning. Experimental results show that the proposed method can well preserve image details meanwhile suppress the speckle noises.

Multiobjective evolutionary algorithms (MOEAs) are effective optimization methods. To improve the segmentation performance and time efficiency of MOEAs-based fuzzy clustering algorithms for color images, a semisupervised surrogate-assisted multiobjective kernel intuitionistic fuzzy clustering (S3MKIFC) algorithm is proposed in this article. The main contributions of S3MKIFC can be summarized as follows: 1) semisupervised kernel intuitionistic fuzzy objective functions are constructed for optimization to search satisfactory segmentation results; 2) to reduce the computational cost, the Kriging model is used to predict the values of objective functions instead of directly calculating the expensive objective functions; 3) a semisupervised selection strategy and a semisupervised model management mechanism are proposed to balance the convergence and diversity and improve the predicted accuracy of the Kriging model, respectively; and 4) a novel semisupervised kernel intuitionistic fuzzy cluster validity index is defined to select the optimal solution from the final nondominated solution set. Experimental results on two color image libraries demonstrate that S3MKIFC outperforms state-of-the-art methods in segmentation performance and meanwhile possesses a low time cost.

This paper presents a novel semi-supervised terrain classification method of polarimetric synthetic aperture radar (PolSAR) image based on complex-valued convolution neural network (CV-CNN). Our proposed method only needs a small number of labeled samples to achieve good classification results. First, a Wishart classifier is used to find highly reliable samples in PolSAR data. Then, a new semi-supervised deep recurrent CV-CNN (RCVCNN) classification model has been proposed to improve PolSAR image classification accuracy and effectively solve network overfitting. Finally, a real PolSAR dataset is used to verify the effectiveness of our algorithm. Compared with the other three state-of-the-art methods, the proposed one show improvements in accuracy and better consistency.

With the great successful development of deep learning, stacked auto-encoder (SAE) has been widely used in POLSAR image terrain classification. In this paper, we propose a complex-valued Wishart stacked auto-encoder (CV-WSAE) classification model for POLSAR data interpretation. The proposed method stacks a complex-valued Wishart autoencoder (CV-WAE) and a complex-valued auto-encoder (CVAE) for feature extraction and connects a linear classifier for image classification. It not only expends real-valued neural network to complex-valued, but also utilizes the statistical distribution of POLSAR image. What is more, all elements of CV-WSAE including input, hidden, output, encoder and decoder layers are complex-valued, and a complex back propagation algorithm is used for training processing. The experiments of a real POLSAR data illustrate that this method can obtain good classification accuracy.

Images are always contaminated by noise, increasing uncertainty. Fuzzy set (FS) theory is a useful tool for dealing with uncertainty in images. When comparing with the FS, an intuitionistic fuzzy set (IFS) can better describe the blurred characteristic in images due to the membership, nonmembership, and hesitation degrees. However, when applied to an image segmentation, the IFS cannot completely overcome the influence of noise. With the aim of performing noisy image segmentation under several criteria, this paper defines a noise robust IFS (NR-IFS) for an image and then presents a novel noise robust multiobjective evolutionary intuitionistic fuzzy clustering algorithm (NR-MOEIFC). A majority dominated suppressed similarity measure using the neighborhood statistics and the competitive learning is proposed to obtain the NR-IFS representation for the image corrupted by noise. Then, the NR-IFS is fully used to motivate the whole process of multiobjective evolutionary clustering: first, computing a three-parameter intuitionistic fuzzy distance measure; second, constructing intuitionistic fuzzy fitness functions; third, designing a nonuniform intuitionistic fuzzy mutation operator; and forth, defining an intuitionistic fuzzy cluster validity index to select the optimal solution from the final nondominated solution set. The histogram statistics of NR-IFS are adopted in the NR-MOEIFC to greatly reduce the computational complexity. Experimental results on Berkeley and real magnetic resonance images reveal that the NR-MOEIFC behaves well in noise robustness and segmentation performance while requiring a low time cost.