Semantic change detection (SCD) aims to provide the change locations and extend the detailed semantic change categories before and after the observation intervals, being a pivotal task in remote sensing community. Recent studies indicate that multi-task learning paradigm is an efficient solution for modeling the SCD. However, the temporal dependency information and semantic feature separability are not efficiently explored. To overcome the above limitations, this paper proposes a new Triplet Multi-task learning Network (TMNet) for SCD, in which a differential image branch is introduced to extract the temporal information reflecting the difference in original image pairs and two temporal branch to extract the semantic representation of each image. Then, a class-wise contrastive loss is employed to deal with the change types imbalance and improve the category discrimination. Finally, experiments are carried on a public SCD dataset to demonstrated the effectiveness of the proposed method.

This paper presents a Siamese network architecture based on a multi-scale hybrid convolution-Transformer (CTUNet) for Change Detection (CD) in a pair of co-registered optical remote sensing images. Different form CD frameworks based on convolution neural networks (CNNs) and pure Transformer networks, this method combines a convolution-Transformer hybrid encoder with a multi-scale change information extraction decoder in a Siamese network architecture. It overcomes the inherent limitations of CNN and Transformer and effectively integrates the multi-scale information required for accurate CD. To learn better discriminative representations from various scales, we propose a masked auto-encoder scheme (CTMAE) to adapt to building targets with varying morphological scales, further unleashing the potential of CTUNet. Experiments on two CD datasets show that the proposed self-supervised pre-trained hybrid convolution-Transformer CTUNet architecture achieves better CD performance than previous methods.

Recently, semantic segmentation technology has been a research hotspot in optical remote sensing urban use classification. However, because of coupled semantic relations in very high-resolution and complex urban scenes, a more effective semantic description for pixelwise urban use interpretation has become a challenge. Then, aiming to set up a more effective semantic description, the effective receptive field (ERF) is analyzed in general convolutional neural networks. The unreasonable ERF distribution in the stacked convolutional layers of the encoder would lead to a large amound of small ERFs and fewer not large enough ERFs that form a naive semantic description in decoder. Therefore, in this article, a novel full semantic constructed network (FSCNet) is proposed to improve the naive semantic description and set up an effective semantic description. First, to avoid noise from shallow feature layers, a residual refinement convolution is designed to optimize the full-scale skip connections based on the U-shaped encoder–decoder. Second, an interscale fusion module is newly designed for multiscale feature fusion, which can generate three initial semantic modalities that are prepared for redefining the full semantic description. Third, a multiscale local context spatial attention module and boundary supervision are designed for an initial shallow semantic modality to capture the pure boundary information, and then, pyramid spatial pooling is employed for an initial deep semantic modality to further enlarge the ERF and obtain more abstract global information. Next, a self-calibration convolution combined with the atrous spatial pyramid pooling is designed to rectify and enrich an initial middle semantic modality, which can improve the naive semantic description and bridge the semantic gap between the redefined shallow and deep semantic modalities to advance the full semantic feature fusion. Finally, extensive experiments are carried out on three benchmarks (e.g., ISPRS Vaihingen, Potsdam, and DLRSD), and comparative results show that the proposed FSCNet can get remarkable performance compared to state-of-the-art (SOTA) methods. Besides, the code is available athttps://github.com/DorisCV/FSCNet.

Change detection (CD) is an essential task in optical remote sensing, and it can be used to extract the valid information from sequential multitemporal images. However, since the character of long-term revisiting and very high resolution (VHR) development, the great differences of illumination, season, and interior textures between bitemporal images bring considerable challenges for pixel-wise CD. In this letter, focusing on accurate pixel-wise CD, a bilateral semantic fusion Siamese network (BSFNet) is proposed. First, to better map bitemporal images into semantic feature domain for comparison, a novel BSFNet is designed to effectively integrate shallow and deep semantic features, which can provide pixel-wise CD results with complete regions and clear boundary locations. Then, in order to facilitate the reasonable convergence of the proposed BSFNet, a scale-invariant sample balance (SISB) loss is designed for metric learning to avoid the problems of sample imbalance and scale variance. Finally, extensive experiments are carried out on two published CDD and LEVIR CD datasets, and results indicate that the proposed BSFNet can provide superior performance than the other state-of-the-art methods. Our work is available athttps://github.com/ClarissaDHL/BSFNet.

Scene classification of optical remote sensing is a basic but important task because of its broad application in a range of fields. Due to the powerful feature extraction capabilities, convolutional neural networks (CNNs) have been widely used in optical remote sensing scene classification tasks. Despite the remarkable efforts have been achieved, there are still several problems existed, including the effective multiscale feature description for complex scene, rotation, and low interclass diversity problems. In this letter, to address these mentioned problems and construct a powerful CNN for optical remote sensing scene classification, an effective multiscale residual network with a high-order feature representation (MRHNet) is proposed. First, data preprocessing is utilized to adapt the rotation invariance problem. Second, related to the original residual module, a pyramid convolution is introduced to realize the multiscale feature extraction, and then, its feature description ability is further improved by an effective channel attention module. Third, inspired by the tensor decomposition and its completion, a high-order feature representation structure is designed for recovering discriminative fine-scale details into deep layers to solve the low interclass diversity problem. Finally, extensive experiments are carried on two widely used scene classification datasets (e.g., AID and NWPU-RESISC45), and comparing results show that the proposed MRHNet can achieve superior performances.

Semantic segmentation of remote sensing images aims to label every pixel with the correct semantic category. The core challenge of the current deep convolutional network (ConvNet)-based methods lies in the difficulty of effectively aggregating high-level categorical semantics and low-level local details along the hierarchy of backbone. Most current approaches consider only fusing adjacent feature layers gradually with short-range feature connections, which lack the diversity of feature interactions, such as long-range cross-scale connections. To this end, we propose a novel dual-path sparse hierarchical network that is characterized by rich cross-scale feature interactions. Multiscale features are first sparsely grouped with a predefined interval, which is then aggregated via both long-range and short-range cross-scale connections in a hierarchical manner. Moreover, in order to further enrich the diversity of feature interactions, we also introduce another fusion path in parallel but with different sparsity for feature grouping, forming a dual-path network. In this way, our model is able to effectively aggregate multilevel features by incorporating both long-range and short-range feature interactions in both parallel and hierarchical manner. Meanwhile, the semantic and resolution gap between multilevel features can also be bridged.

Object detection is an essential task in computer vision. Recently, several convolution neural network (CNN)-based detectors have achieved a great success in natural scenes. However, for optical remote sensing images with a large scale of view, lower proportion of foreground target pixels and drastic differences in object scale present considerable challenges. To address these problems, we propose a novel one-stage detector called the full-scale object detection network (FSoD-Net) which consists of proposed multiscale enhancement network (MSE-Net) backbone cascaded with scale-invariant regression layers (SIRLs). First, MSE-Net provides the multiscale description enhancement by integrated the Laplace kernel with fewer parallel multiscale convolution layers. Second, SIRLs contain three different isolated regression branch layers (i.e., corresponding to small, medium, and large scales), which make default discrete scale bounding boxes (bboxes) cover full-scale object information in regression procedure. A novel specific scale joint loss is also designed that uses the softmax function combined with a strong$L_{1}$-norm constraint in each regression branch layer. It can further speed up the convergence and improve the classification scores of predicted bboxes. Finally, extensive experiments are carried on challenge data sets of large-scale dataset for object detection in aerial images (DOTA) and object detection in optical remote sensing images (DIOR) which contain multiple instances from different imaging platforms, and these results demonstrate that FSoD-Net can achieve better performance than other state-of-the-art one-stage detectors, and it can reach a mean average precision (mAP) of 75.33% on DOTA and 71.80% mAP on DIOR, respectively. Especially, the average precision (AP) of tiny object detection can improve 10%–20% approximately.

Optical remote sensing object detection is a challenging task, because of the complex background interference, ambiguous appearances of tiny objects, densely arranged circumstances, and multiclass object with vaster scale variances and irregular aspect ratios. The performance of object detection is seriously restricted. Thus, in this article, inspired by the anchor-free object detection framework, and aiming to solve these difficulties to improve the optical remote sensing object detection performance, a powerful one-stage detector of multiscale semantic fusion-guided fractal convolution network (MSFC-Net) is proposed. First, facing these strong-coupled semantic relations in each complex scene, a compound semantic feature fusion (CSFF) way is designed for generating an effective semantic description, which is a benefit to pixel-wise object center point interpretation. In addition, it can be easily extended into a semantic segmentation task. Second, in view of accurate multiclass pixel-wise center point predictions based on an effective compound semantic description, a novel fractal convolution (FC) regression layer is designed, which adaptively achieves the regression of multiscale bounding boxes (bboxes) with irregular aspect ratio under no priori information. Third, related to the set up FC regression layer, a specific hybrid loss is designed to make the proposed MSFC-Net converge better. Finally, the extensive experiments on challenge data sets of large-scale dataset for object detection in aerial images (DOTA) and object detection in optical remote sensing images (DIOR) datasets are carried out, and comparisons indicate that the proposed MSFC-Net can perform the remarkable performance than other state-of-the-art one-stage detectors, as it can reach 80.26% mean average precision (mAP) and 79.33% mF1 on DOTA and 70.08% mAP and 73.45% mF1 on DIOR. Then, our work is available athttps://github.com/ZhAnGToNG1/MSFC-Net.

Land cover classification is a popular research field in remote sensing applications, which have to both consider the pixel-level classification and boundary mapping comprehensively. Although multi-scale features in deep learning (DL) network have a powerful classification ability, how to use multi-scale feature description to produce an accurate land cover classification from very high resolution (VHR) optical remote sensing image is still a challenging task because of large intraclass or small interclass difference of land covers. Therefore, aiming at achieving more accurate pixel-level land cover classification, we proposed a novel feature ensemble network (FE-Net), which includes the multi-scale feature encapsulation and enhancement two phases. First, there are encapsulated shallow, middle, and deep scale feature layers from Resnet-101 backbone. Second, related to multi-scale feature description enhancement, these 2-D dilation convolutions with different sample rates are employed on each scale feature layer. After that, optimal channel selection works on each intrascale and interscale feature layers sequentially. Finally, extensive experiments proved that the proposed FE-Net combined with a special joint loss function outperforms state-of-the-art DL based methods. It can achieve the 68.08% and 65.16% of the mean of class-wise intersection over union (mIoU) on ISPRS and GID data sets, respectively.

Deep learning (DL) technique is widely applied in remote sensing (RS) applications because of its outstanding nonlinear feature extraction ability. However, with regard to the issues of large-scale and very high-resolution (VHR) land cover classification, multi-object distributions and clear appearance with large intraclass difference become challenges for refined pixelwise land cover mapping. Focusing on these problems, the letter proposed a novel encoding-to-decoding method called the full receptive field (RF) network (FRF-Net) based on two types of attention mechanism. In the FRF-Net, ResNet-101 is used as the basic backbone. Then, the ensemble feature is generated by encoding the high-level features based on the self-attention mechanism which could achieve full RF to capture long-range semantic. Next, the encoding result is decoded by the fusion attention mechanism combined with the low-level feature to produce a fusion feature which contains a refined semantic description for accurate land cover mapping. Extensive experiments based on the GID and ISPRS data sets proved that the proposed network outperforms the state-of-the-art methods. The FRF-Net achieved 66.71% and 64.17% of the mean of classwise Intersection over Union (mIOU) with smaller computation cost on ISPRS and GID, respectively.

For water resource analysis applications, due to very high resolution and large observation scope, optical remote sensing images can present more visible object characters. Water-land segmentation from optical remote sensing images is wildly used and becomes a hot research topic. However, since large scale complex background scenes include many interferences, the water-land segmentation from optical remote sensing images becomes a challenge task. Aim to achieve better water area feature description from complex land cover background, we apply a sub-neighbor system convolutional neural network (SNS-CNN) to the water-land segmentation in harbor scene areas. First, on the basis of the U-net structure, an optimized up-sampling process is proposed to enhance water area feature expression. Second, a novel sub-neighbor system constraint of each predicted pixel point is leaded into the loss function to make the model producing water mask more coherent. Furthermore, experiments on our collected variety of optical remote sensing images demonstrate that this paper proposed water-land segmentation method can produce better performance than state of the art methods.

It is shown that it is possible to provide an effective, patient-specific correction to geometrical distortion in MRI (magnetic resonance imaging) for use in stereotaxy. The authors examine two correction techniques, each based on the acquisition of an additional image of the patient utilizing a reversed readout gradient. Information from the additional image is combined with that of the standard image to provide geometrically corrected coordinates for targets identified by a neurosurgeon. They have applied the technique retrospectively to images of a patient who has undergone stereotaxic surgery, and they show that large geometrical errors are corrected.>