The shared anatomy among different human bodies can serve as a strong prior for effectively leveraging unlabeled data in semi-supervised medical image segmentation. Inspired by the success of masked image modeling, we notice that this prior can be explicitly realized by incorporating an auxiliary unsupervised gross anatomy reconstruction task into a teacher-student semi-supervised segmentation framework. In this auxiliary task, consistency is maintained between the student's predictions on masked images and the teacher's predictions on the original images. Despite its potential, we observe that the reconstruction difficulties of different organs/tissues can vary significantly and therefore reconstructing them requires tailored learning strategies. To address this issue, we introduce a difficulty-adapted mask mechanism based on the teacher-student framework, wherein the reconstruction difficulty is adapted to facilitate training. Specifically, we control the reconstruction difficulty by modulating two important factors: masked region ratio and masked class ratio. Accordingly, we design two corresponding mask strategies. 1) Region-based masking: randomly masks a fraction of each class according to an automatically computed mask ratio. 2) Class-based masking: masks the entire regions of the specific classes according to the class confidence predicted by the teacher model. During training, a conflict-aware gradient computation strategy is introduced to mitigate potential optimization conflicts arising from modulating the two reconstruction factors simultaneously. By building on vision transformers, we develop an Difficulty-adapted Masked Transformer (DMformer) for semi-supervised medical image segmentation. Extensive experiments demonstrate the superiority of DMformer, which outperforms the previous SOTA by 9.53% and 4.63% in terms of DSC on ACDC dataset with 5% labeled images and Synapse dataset with 30% labeled images, respectively.

The contemporary research paradigm in remote sensing hyperspectral monitoring increasingly relies on multisatellite and multiplatform Earth observation. While the traditional hyperspectral research framework based on single-image processing (SIP) has facilitated the application of idealized scenarios and the development of standardized evaluation benchmarks, it inherently constrains the model's ability to generalize feature representations across varying spatial and temporal domains. As hyperspectral data applications grow in complexity and data requirements, the limitations of SIP in addressing the demands of modern remote sensing tasks become increasingly apparent. To overcome these research limitations, we utilize the decentralized nature and data security features of federated learning to propose a cross-image hyperspectral image (HSI) federated learning approach for classification tasks. We first develop a client-oriented self-guided knowledge-enhanced personalized learning method that enhances the personalization of the local learning process by leveraging relevant features from other clients, thereby improving the learning efficiency of each client. To address the issue of "bias" in global knowledge caused by uneven data distribution across the federated learning process, we introduce a multiscale semantic aligned dynamic aggregation method to ensure fairness in integrating global knowledge. To our knowledge, this article is the first to explore the joint learning of HSI classification using federated learning. Accordingly, we have constructed open-set and closed-set datasets tailored to this task and have demonstrated the effectiveness of our method on these datasets. The code is available at: https://github.com/Gallipaxi/FedHIC.

Deep neural networks have achieved promising progress in signal modulation classification (SMC), playing an essential role in a variety of applications such as cognitive radio networks, cyber defense, and electronic surveillance. However, most existing SMC methods still follow the traditional machine learning paradigm that trains on static closed datasets, lacking the ability to cope with the challenge of continuous data distribution shifts in real communication scenarios. Directly applying the model to a new environment may lead to severe degradation of classification performance on previous scenarios, i.e., catastrophic forgetting. To address this, this article proposes the first domain-incremental learning (DIL) paradigm for SMC and designs a parameter-efficient isolation DIL (PID) method, which enables SMC models to rapidly adjust to new scenarios by extending only a few parameters, while significantly retaining classification capabilities on previous scenarios. Specifically, we first propose a parameter space decomposition-based classifier (PSD), separating the model parameters into a set of bases and corresponding coefficients. By freezing the bases and fine-tuning the low-dimensional coefficients, the catastrophic forgetting problem can be efficiently eliminated. Furthermore, we design a scene-aware domain controller (SDC) to select the most suitable domain-specific coefficients for each sample, thereby maintaining the SMC model's classification capabilities across all domains. The extensive experimental results show the superiority of the proposed PID, which achieves state-of-the-art (SOTA) overall performance. The code will be available at: https://github.com/SMC-IL/PID.

Unsupervised anomaly detection in hyperspectral images (HSI), aiming to detect unknown targets from backgrounds, is challenging for earth surface monitoring. However, current studies are hindered by steep computational costs due to the high-dimensional property of HSI and dense sampling-based training paradigm, constraining their rapid deployment. Our key observation is that, during training, not all samples within the same homogeneous area are indispensable, whereas ingenious sampling can provide a powerful substitute for reducing costs. Motivated by this, we propose an Asymmetrical Consensus State Space Model (ACMamba) to significantly reduce computational costs without compromising accuracy. Specifically, we design an asymmetrical anomaly detection paradigm that utilizes region-level instances as an efficient alternative to dense pixel-level samples. In this paradigm, a low-cost Mamba-based module is introduced to discover global contextual attributes of regions that are essential for HSI reconstruction. Additionally, we develop a consensus learning strategy from the optimization perspective to simultaneously facilitate background reconstruction and anomaly compression, further alleviating the negative impact of anomaly reconstruction. Theoretical analysis and extensive experiments across eight benchmarks verify the superiority of ACMamba, demonstrating a faster speed and stronger performance over the state-of-the-art. Code is released at https://github.com/PURE-melo/ACMamba.

Unsupervised hyperspectral change detection (UHCD), detecting subtle changes between bi-temporal images without manual annotations, is an essential but challenging task in the earth observation community. The current modus operandi often performs it in a feature comparison manner, which is limited by variations in imaging conditions. We observe that fully supervised paradigms using limited annotations are capable of overcoming this challenge. Based on this, we introduce a novel Observational Learning Paradigm (OraL) for UHCD by mimicking fully supervised paradigms. OraL comprises two sequential stages: Observation, which designs a spatial-temporal observation strategy (STO) that records the learning consistency of pixels under different training steps and views, to obtain reliable pseudo-labels. Reproduction, which retrains the model with these pseudo-labels and introduces a distribution-aware spectral learning strategy (DSL) to adaptively increase their learning difficulty according to spectral distributions, enhancing the robustness and generalization of the model. Extensive experiments on several public hyperspectral image datasets demonstrate its state-of-the-art performance and pluggability for previous unsupervised methods. Code will be made available.

The land cover analysis using hyperspectral images (HSIs) remains an open problem due to their low spatial resolution and complex spectral information. Recent studies are primarily dedicated to designing Transformer-based architectures for spatial-spectral long-range dependencies modeling, which is computationally expensive with quadratic complexity. Selective structured state space model (SSM; Mamba), which is efficient for modeling long-range dependencies with linear complexity, has recently shown promising progress. However, its potential in HSI processing that requires handling numerous spectral bands has not yet been explored. In this article, we innovatively propose S2Mamba, a spatial-spectral SSM for HSI classification, to excavate spatial-spectral contextual features, resulting in more efficient and accurate land cover analysis. In S2Mamba, two selective structured SSMs through different dimensions are designed for feature extraction, one for spatial, and the other for spectral, along with a spatial-spectral mixture gate (SMG) for optimal fusion. More specifically, S2Mamba first captures spatial contextual relations by interacting each pixel with its adjacent through a patch cross scanning (PCS) module and then explores semantic information from continuous spectral bands through a bidirectional spectral scanning (BSS) module. Considering the distinct expertise of the two attributes in homogenous and complicated texture scenes, we realize the SMG by a group of learnable matrices, allowing for the adaptive incorporation of representations learned across different dimensions. Extensive experiments conducted on HSI classification benchmarks demonstrate the superiority and prospect of S2Mamba. The code will be made available at:https://github.com/PURE-melo/S2Mamba.

Weakly supervised object detection (WSOD) and semantic segmentation with image-level annotations have attracted extensive attention due to their high label efficiency. Multiple instance learning (MIL) offers a feasible solution for the two tasks by treating each image as a bag with a series of instances (object regions or pixels) and identifying foreground instances that contribute to bag classification. However, conventional MIL paradigms often suffer from issues, e.g., discriminative instance domination and missing instances. In this article, we observe that negative instances usually contain valuable deterministic information, which is the key to solving the two issues. Motivated by this, we propose a novel MIL paradigm based on negative deterministic information (NDI), termed NDI-MIL, which is based on two core designs with a progressive relation: NDI collection and negative contrastive learning (NCL). In NDI collection, we identify and distill NDI from negative instances online by a dynamic feature bank. The collected NDI is then utilized in a NCL mechanism to locate and punish those discriminative regions, by which the discriminative instance domination and missing instances issues are effectively addressed, leading to improved object- and pixel-level localization accuracy and completeness. In addition, we design an NDI-guided instance selection (NGIS) strategy to further enhance the systematic performance. Experimental results on several public benchmarks, including PASCAL VOC 2007, PASCAL VOC 2012, and MS COCO, show that our method achieves satisfactory performance. The code is available at: https://github.com/GC-WSL/NDI.

With the rapid advances in deep learning techniques, remote sensing object detection has achieved remarkable achievements in recent years. However, tiny object detection remains unsatisfactory and suffers from two main drawbacks, including (1) the high sensitivity of IoU for location deviation in tiny objects and (2) the poor-quality feature representations of tiny objects. To address the aforementioned problems, we propose a Multi-stage Enhancement Network (MENet) that achieves the instance-level and feature-level enhancement of tiny objects from different stages of the detector. Since the IoU-based label assignment drastically deteriorates the positive samples for tiny objects, we first propose a Central Region-based (CR) label assignment to substitute it in the Region Proposal Network (RPN). The CR label assignment regards the anchors that fall into the central region of ground-truth boxes as positive samples, which provides more positive samples for tiny objects. Then, we design a Gated Context Aggregation (GCA) module that selectively aggregates valuable context information to enhance the feature representation of tiny objects. Additionally, we devise a positive RoI feature (pRoI) generator in the Region Convolutional Neural Network (R-CNN) to generate a rich diversity of high-quality positive RoI features for tiny objects. We conduct extensive experiments on AI-TOD and SODA-A datasets, and the results demonstrate the effectiveness of our proposed method.

Seed area generation is usually the starting point of weakly supervised semantic segmentation (WSSS). Computing the Class Activation Map (CAM) from a multi-label classification network is the de facto paradigm for seed area generation, but CAMs generated from Convolutional Neural Networks (CNNs) and Transformers are prone to be under- and over-activated, respectively, which makes the strategies to refine CAMs for CNNs usually inappropriate for Transformers, and vice versa. In this paper, we propose a Unified optimization paradigm for Seed Area GEneration (USAGE) for both types of networks, in which the objective function to be optimized consists of two terms: One is a generation loss, which controls the shape of seed areas by a temperature parameter following a deterministic principle for different types of networks; The other is a regularization loss, which ensures the consistency between the seed areas that are generated by self-adaptive network adjustment from different views, to overturn false activation in seed areas. Experimental results show that USAGE consistently improves seed area generation for both CNNs and Transformers by large margins, e.g., outperforming state-of-the-art methods by a mIoU of 4.1% on PASCAL VOC. Moreover, based on the USAGE-generated seed areas on Transformers, we achieve state-of-the-art WSSS results on both PASCAL VOC and MS COCO.

Hyperspectral image change detection (HSI-CD) can accurately identify changing regions by capturing subtle spectral differences and has become a research hotspot in the field of remote sensing (RS). Convolutional neural networks (CNNs) have excellent local context modeling capabilities and have been proven to be powerful feature extractors in HSI-CD. However, due to its inherent network structure limitation, CNN cannot well mine and represent the sequential properties of spectral features, especially the medium and long-term dependencies. In contrast, transformer-based network architecture shows a strong ability to model long-distance dependencies, which can fully mine and extract global features, but exhibits weak performance in extracting local information. To this end, we propose HSI-CD network based on adaptive contrastive learning (CTACL). Specifically, we first propose a parallel network of CNNs and transformers to mine local and global temporal-spatial-spectral features of HSI, respectively. Second, we propose adaptive contrastive learning to pre-train the network to learn the latent features of a large amount of unlabeled data and better mine and utilize local and global information. Experimental results on the farmland dataset show that the proposed method performs well.

Oriented object detection is a challenging task in remote sensing, where the detected objects can be represented by oriented bounding boxes (OBBs). Angle prediction in oriented object detection has been widely studied, due to its crucial role in object detection. However, the precision of angle prediction is severely limited by misalignments in most of the existing methods, including representation-, evaluation-, and optimization-based misalignments. To alleviate these misalignments, this paper presents a novel angle prediction method, called Angle Quality Estimation (AQE). Specifically, our proposed AQE transforms the angle prediction task into a distribution estimation task to address the representation misalignment problem and implicitly measure the quality of the predicted angles. Based on the estimated angle quality, we then propose a new metric to comprehensively evaluate the quality of OBBs. Then we propose an object aspect ratio based loss function to optimize angle prediction for addressing the optimization misalignment. Our proposed AQE is a plug-and-play method, which can be embedded on any existing oriented object detector. Experimental results on three public benchmarks, including DOTA, HRSC2016, and ICDAR2015 datasets, show that our method achieves better performance than the other state-of-the-art.

Few-shot remote sensing scene classification aims to identify unseen classes using only a small number of labeled samples. Considering the large intra-class variances and inter-class similarity of remote sensing scenes, most existing methods focus on feature extraction, ignoring the overfitting problem caused by insufficient samples. To this end, we propose a novel few-shot learning framework, called multibranch feature generation networks (MFGNets), which solves the few-shot scene classification from the source by online sample generation at the representation space. Specifically, we first build a feature generation net to transform the few-shot classification into a regular classification problem, in which the generated samples are achieved by combining the class-specific features with the sampled intra-class features. Then, to ensure the quality of the generated samples, we introduce two novel regularization terms: the intra-class diversity loss (ID-Loss) and the inter-class consistency loss (IC-Loss), which aid the model in generating more diverse samples. Furthermore, we introduce a scale-angle aware self-supervised pretext to learn scale-invariant and rotation-invariant features, improving the model’s feature representation capability in remote sensing scenes. We evaluate the proposed method on three publicly available datasets, namely UC_Merced, NWPU-RESISC45, and AID. Our approach has achieved state-of-the-art performance, with an improvement of more than 3.31%, 2.64%, and 6.86% on the most challenging 1-shot tasks, respectively.

Hyperspectral image change detection (HSI-CD) aims to detect subtle changes on the Earth’s surface through approximately continuous spectral information, which has gradually become a very important research hotspot in the field of remote sensing (RS). In recent years, convolutional neural networks (CNNs) based HSI-CD methods have shown strong feature extraction capabilities. However, due to the simple fusion of spectral information in the channel dimension by CNN, the medium and long-term sequence properties of spectral features cannot be well mined and represented. Most previous studies mainly extract semantic features from images at different times, ignoring the temporal correlation between features, which cannot fully extract and effectively utilize temporal-spatial-spectral features. To this end, this paper proposes a cascade spectral aware transformer (CAST) for HSI-CD. First, we propose a temporal-spatial transformer (TS-Former) to enhance the temporal correlation and spatial global relationship of extracted features, thereby addressing the insufficient consideration of temporal correlation. Second, a spectral awareness transformer (SA-Former) is designed to better mine and represent the sequence properties of spectral features, especially the medium and long-term dependencies. Finally, we observe a spectral distortion in the process of extracting temporal-spatial features and based on this present a spectral constraint module (SCM) to preserve the sequence properties of spectral features and reduce the distortion of the spectrum. Extensive experiments on three challenging hyperspectral datasets demonstrate that our method achieves state-of-the-art results. The code is available at: https://github.com/tianshunli/CAST.

Weakly supervised object detection (WSOD) is a challenging task, in which image-level labels (e.g., categories of the instances in the whole image) are used to train an object detector. Many existing methods follow the standard multiple instance learning (MIL) paradigm and have achieved promising performance. However, the lack of deterministic information leads to part domination and missing instances. To address these issues, this paper focuses on identifying and fully exploiting the deterministic information in WSOD. We discover that negative instances (i.e. absolutely wrong instances), ignored in most of the previous studies, normally contain valuable deterministic information. Based on this observation, we here propose a negative deterministic information (NDI) based method for improving WSOD, namely NDI-WSOD. Specifically, our method consists of two stages: NDI collecting and exploiting. In the collecting stage, we design several processes to identify and distill the NDI from negative instances online. In the exploiting stage, we utilize the extracted NDI to construct a novel negative contrastive learning mechanism and a negative guided instance selection strategy for dealing with the issues of part domination and missing instances, respectively. Experimental results on several public benchmarks including VOC 2007, VOC 2012 and MS COCO show that our method achieves satisfactory performance.

With the outbreak of COVID-19, it is urgent and necessary to design a system that can access to information from COVID-19 related documents. Current methods fail to do so since the knowledge about COVID-19, an emerging disease, keeps changing and growing. In this study, we design a dynamic document-based question answering system, namely Web Understanding and Learning with AI (WULAI-QA). WULAI-QA employs feature engineering and online learning to adapt to the non-stationary environment and maintains good and steady performance. We evaluate WULAI-QA's performance on a public question answering (https://www.datafountain.cn/competitions/424) and rank first. We demonstrate that WULAI-QA can learn from user feedback and is easy to use. We believe that WULAI-QA will definitely help people understand COVID-19 and play an important role to fight against the pandemic.

Ship detection is a significant and challenging task in remote sensing. Due to the arbitrary-oriented property and large aspect ratio of ships, most of the existing detectors adopt rotation boxes to represent ships. However, manual-designed rotation anchors are needed in these detectors, which causes multiplied computational cost and inaccurate box regression. To address the abovementioned problems, an anchor-free rotation ship detector, named GRS-Det, is proposed, which mainly consists of a feature extraction network with selective concatenation module (SCM), a rotation Gaussian-Mask model, and a fully convolutional network-based detection module. First, a U-shape network with SCM is used to extract multiscale feature maps. With the help of SCM, the channel unbalance problem between different-level features in feature fusion is solved. Then, a rotation Gaussian-Mask is designed to model the ship based on its geometry characteristics, which aims at solving the mislabeling problem of rotation bounding boxes. Meanwhile, the Gaussian-Mask leverages context information to strengthen the perception of ships. Finally, multiscale feature maps are fed to the detection module for classification and regression of each pixel. Our proposed method, evaluated on ship detection benchmarks, including HRSC2016 and DOTA Ship data sets, achieves state-of-the-art results.

We present a novel approach to improve the performance of distant supervision relation extraction with Positive and Unlabeled (PU) Learning. This approach first applies reinforcement learning to decide whether a sentence is positive to a given relation, and then positive and unlabeled bags are constructed. In contrast to most previous studies, which mainly use selected positive instances only, we make full use of unlabeled instances and propose two new representations for positive and unlabeled bags. These two representations are then combined in an appropriate way to make bag-level prediction. Experimental results on a widely used real-world dataset demonstrate that this new approach indeed achieves significant and consistent improvements as compared to several competitive baselines.

With the explosive increase of remote sensing images, content-based remote sensing image retrieval (CBRSIR) has aroused widespread attention. Convolutional Neural Network (CNN) based methods are widely used in CBRSIR due to the development of deep learning. However, common used CNN models have difficulties in holding shift-invariant property due to the widely used down-sampling method, which means a little shift of input may cause a mutation of feature representation. To mitigate the absence of shift-invariant in down-sampling, we propose the learnable blur kernel (LBK), that can enhance the feature extraction capability by leveraging more context information. We build on this concept without extra cost, which can be simply integrated with modern CNNs architecture. Our method is validated on the public remote sensing dataset and compared with other retrieval methods. The overall experimental results show that the proposed method achieves outstanding performance.

This paper proposes a Markov Decision Process and reinforcement learning based approach for domain selection in a multidomain Spoken Dialogue System built on a distributed architecture.In the proposed framework, the domain selection problem is treated as sequential planning instead of classification, such that confirmation and clarification interaction mechanisms are supported.In addition, it is shown that by using a model parameter tying trick, the extensibility of the system can be preserved, where dialogue components in new domains can be easily plugged in, without re-training the domain selection policy.The experimental results based on human subjects suggest that the proposed model marginally outperforms a non-trivial baseline.