Recently, Transformer has demonstrated its comparable performance in several vision tasks and multimedia domains. However, the key self-attention in Transformer computes global attention to the image primarily in the spatial dimension, which is a non-local operation. Such spatial attention lacks the ability to model the relationship among local regions of an image and the learned representations are biased with redundant channel information perturbation. To address this problem, we propose a new Multi-scale Weight-residual Transformer (MWT) for uni-and multi-modal. Specifically, we generate local and regional tokens by different convolutions and use them as query and key-value, respectively. In this way, the computed self-attention ensures both global information interaction and focuses attention on regional information, which is more relevant to the local information. It not only retains the global attention of transformers but also obtains the ability of local attention as CNNs. Moreover, we introduce a weight-residual network for channel dimension to alleviate the feature weakening in deeper layers of the network, which can improve the sensitivity of the model for key channel representations. These can build a more abstract high-level feature representation. Extensive experiments demonstrate the effectiveness of MWT on several uni- and multimodal benchmark tasks.

Recently, the DEtection TRansformer (DETR) and its variants have achieved success in object detection, but they still require manual intervention in complex scenes. Object detection methods cannot entirely replace human capabilities. Leveraging human perception to assist remote sensing object detection offers a new technical approach. In the process of manual verification, the most prominent signals reflecting human identification of specific targets are electroencephalogram (EEG) and eye movement. To our best knowledge, there is currently no publicly available dataset for this task. Therefore, we collect two types of human perception information while observing remote sensing images, forming a trimodal dataset including brain signals, eye movements, and remote sensing images. Upon this, we propose a multimodal object detection method, termed as BEI-DETR (Brain-Eye-Image DETR). Specifically, we decouple the classification and localization tasks in the object detection, using EEG for classification and eye movement information for localization. Subsequently, we propose a feature matching fusion module (FMFM) to fuse multimodal features and then put them into the decoder and prediction heads to obtain the final detection results. Comparative and ablation experiments demonstrate the satisfactory performance of BEI-DETR, indicating that our model can effectively learn human perceptual information for the remote sensing object detection task. The trimodal dataset and source code are publicly available at https://github.com/Hickey-Curry/BEI-DETR and https://github.com/Hickey-Curry/BEIDataset.

Few-shot remote sensing image scene classification (FS-RSISC) aims at classifying remote sensing images with only a few labeled samples. The main challenges lie in small inter-class variances and large intra-class variances, which are the inherent property of remote sensing images. To address these challenges, we propose a transfer-based Dual Contrastive Network (DCN), which incorporates two auxiliary supervised contrastive learning branches during the training process. Specifically, one is a Context-guided Contrastive Learning (CCL) branch and the other is a Detail-guided Contrastive Learning (DCL) branch, which focus on inter-class discriminability and intra-class invariance, respectively. In the CCL branch, we first devise a Condenser Network to capture context features, and then leverage a supervised contrastive learning on top of the obtained context features to facilitate the model to learn more discriminative features. In the DCL branch, a Smelter Network is designed to highlight the significant local detail information. And then we construct a supervised contrastive learning based on the detail feature maps to fully exploit the spatial information in each map, enabling the model to concentrate on invariant detail features. Extensive experiments on four public benchmark remote sensing datasets demonstrate the competitive performance of our proposed DCN.

The goal of General Continual Learning (GCL) is to preserve learned knowledge and learn new knowledge with constant memory from an infinite data stream where task boundaries are blurry. Distilling the model's response of reserved samples between the old and the new models is an effective way to achieve promise performance on GCL. However, it accumulates the inherent old model's response bias and is not robust to model changes. To this end, we propose an Online Contrastive Distillation Network (OCD-Net) to tackle these problems, which explores the merit of the student model in each time step to guide the training process of the student model. Concretely, the teacher model is devised to help the student model to consolidate the learned knowledge, which is trained online via integrating the model weights of the student model to accumulate the new knowledge. Moreover, our OCD-Net incorporates both relation and adaptive response to help the student model alleviate the catastrophic forgetting, which is also beneficial for the teacher model preserves the learned knowledge. Extensive experiments on six benchmark datasets demonstrate that our proposed OCD-Net significantly outperforms state-of-the-art approaches in 3.26%~8.71% with various buffer sizes. Our code is available at https://github.com/lijincm/OCD-Net.

Remote sensing image captioning (RSIC) has great significance in image understanding, which describes the image content in natural language. Existing methods are mainly based on deep learning and rely on the encoder–decoder model to generate sentences. In the decoding process, recurrent neural network (RNN) and long short-term memory (LSTM) are normally applied to sequentially generate image captions. In this letter, the transformer encoder–decoder is combined with grid features to improve the RSIC performance. First, the pretrained convolutional neural network (CNN) is used to extract grid-based visual features, which are encoded as vectorial representations. Then, the transformer outputs semantic descriptions to bridge visual features and natural language. Besides, the self-critical sequence training (SCST) strategy is applied to further optimize the image captioning model and improve the quality of generated sentences. Extensive experiments are organized on three public datasets of RSCID, UCM-Captions, and Sydney-Captions. Experimental results demonstrate the effectiveness of SCST strategy and the proposed method achieves superior performance compared with the state-of-the-art image captioning approaches on the RSCID dataset.