Siamese network based trackers develop rapidly in the field of visual object tracking in recent years. The majority of Siamese network based trackers now in use treat each channel in the feature maps generated by the backbone network equally, making the similarity response map sensitive to background influence and hence challenging to focus on the target region. Additionally, there are no structural links between the classification and regression branches in these trackers, and the two branches are optimized separately during training. Therefore, there is a misalignment between the classification and regression branches, which results in less accurate tracking results. In this paper, a Target Highlight Module is proposed to help the generated similarity response maps to be more focused on the target region. To reduce the misalignment and produce more precise tracking results, we propose a corrective loss to train the model. The two branches of the model are jointly tuned with the use of corrective loss to produce more reliable prediction results. Experiments on 5 challenging benchmark datasets reveal that the method outperforms current models in terms of performance, and runs at 38 fps, proving its effectiveness and efficiency.

Multi-unmanned aerial vehicle (UAV) collaborative 3-D object detection can comprehend complex environments by integrating complementary information, with applications encompassing traffic monitoring, delivery services, and agricultural management. However, the extremely broad observations in aerial remote sensing and significant perspective differences across multiple UAVs make it challenging to achieve precise and consistent feature mapping from 2-D images to 3-D space in multi-UAV collaborative 3-D object detection paradigm. To address the problem, we propose an unparalleled camera-based multi-UAV collaborative 3-D object detection paradigm called UCDNet. Specifically, the depth information from the UAVs to the ground is explicitly utilized as a strong prior to provide a reference for more accurate and generalizable feature mapping. Additionally, we design a homologous point geometric consistency loss as an auxiliary self-supervision, which directly influences the feature mapping module, thereby strengthening the global consistency of multiview perception. Experiments on AeroCollab3D and CoPerception-UAVs datasets show that our method increases 4.7% and 10% mean Average Precision (mAP) respectively compared to the baseline, which demonstrates the superiority of UCDNet.

The encoder–decoder framework is prevalent in existing remote-sensing image captioning (RSIC) models. The appearance of attention mechanisms brings significant results. However, current attention-based caption models only build up the relationships between the local features without introducing the global visual feature and removing redundant feature components. It will cause caption models to generate descriptive sentences that are weakly related to the scene of images. To solve the problems, this article proposed a global visual feature-guided attention (GVFGA) mechanism. First, GVFGA introduces the global visual feature and fuses them with local visual features to build up their relationships between them. Second, an attention gate utilizing the global visual feature is proposed in GVFGA to filter out redundant feature components in the fused image features and provide more salient image features. In addition, to relieve the hidden state’s burden, a linguistic state (LS) is proposed to specifically provide textual features, making the hidden state only guiding visual–textual attention process. What’s more, to further refine the fusion of visual features and textual features, a LS-Guided Attention (LSGA) mechanism is proposed. It can also filter out the irrelevant information in the fused visual–textual feature with the help of an attention gate. The experimental results show that this proposed image captioning model can achieve better results on three RSIC datasets, UCM-Captions, Sydney-Captions, and RSICD datasets.

Feature pyramid is the mainstream method for multi-scale object detection. In most detectors with feature pyramid, each proposal is predicted based on feature grids pooled from only one feature level, which is assigned heuristically. Recent studies report that the feature representation extracted using this method is sub-optimal, since they ignore the valid information exists on other unselected layers of the feature pyramid. To address this issue, researchers present to fuse valid information across all feature levels. However, these methods can be further improved: the feature fusion strategies, which use common operation (element-wise max or sum) in most detectors, should be replaced by a more flexible way. In this work, a novel method called feature adaptive selection subnetwork (FAS-Net) is proposed to construct effective features for detecting objects of different scales. Particularly, its adaption consists of two level: global attention and local adaptive selection. First, we model the global context of each feature map with global attention based feature selection module (GAFSM), which can strengthen the effective features across each layer adaptively. Then we extract the features of each region of interest (RoI) on the entire feature pyramid to construct a RoI feature pyramid. Finally, the RoI feature pyramid is sent to the feature adaptive selection module (FASM) to integrate the strengthened features according to the input adaptively. Our FAS-Net can be easily extended to other two-stage object detectors with feature pyramid, and supports to analyze the importance of different feature levels for multi-scale objects quantitatively. Besides, FAS-Net can also be further applied to instance segmentation task and get consistent improvements. Experiments on PASCAL07/12 and MSCOCO17 demonstrate the effectiveness and generalization of the proposed method.

Considering the lack of labeled training data sets for the supervised change detection task, in this letter, we try to relieve this problem by proposing a convolutional neural network (CNN)-based change detection method with a newly designed loss function to achieve transfer learning among different data sets. To reach this goal, we first pretrain a U-Net model on an open source data set by taking advantages of the relatively sufficient training data used for the supervised semantic segmentation task. Then, we minimize a skillfully designed loss function to combine the high-level features extracted from the pretrained model and the semantic information contained in the change detection data set, by which a transfer learning is achieved. Third, we compute the distance between the feature vectors obtained from the above step and produce a difference map. Finally, a simple clustering method used on the difference map can even obtain satisfied change map. Experiments carried out on typical optical aerial image data sets validate that the proposed approach compares favorably to the state-of-the-art unsupervised methods.

Building reconstruction from SAR images is a hot topic in recent years. Currently, related methods mainly deal with on flat-roofed buildings. In this paper, we extend the research scope to gable-roofed buildings, and try to present a parameterized geometrical model for the layover of gable-roofed buildings. Based on this model, a top-down building reconstruction technique based on MCMC method is proposed. Through representing the layover with parameterized geometrical models, building reconstruction is converted into an optimization problem under the Bayesian scheme. In order to obtain global optima, simulated annealing algorithm with MCMC is used in the optimization stage. Two groups of transmission kernels which are responsible for model updates are designed according to the model. Experiments show that the layover model is accurate and the reconstruction method is effective.

Semantic labeling in high-resolution aerial images is important for its wide range of applications. In this paper, we propose an end-to-end dual multi-scale feature fusion network (DMFFNet) for high-resolution aerial multi-source images. DMFFNet aims to further improve the semantic labeling results of the region where the multispectral features are indistinguishable. Specifically, we design a channel fusion strengthen (CFS) module, which can fuse features adaptively by modelling interdependencies between channels. Furthermore, a multiscale context aggregation (MCA) module is utilized to obtain larger receptive field and more contextual information. The experiment results confirm the DMFFNet with CFS and MCA improve the semantic labeling performance by utilizing multi-source data.

In this paper, we achieve a semantic segmentation of aerial imagery based on the fusion of multi-modal data in an effective way. The multi-modal data contains a true orthophoto and the corresponding normalized Digital Surface Model (nDSM), which are stacked together before they are fed into a Convolutional Neural Network (CNN). Though the two modalities are fused at the early stage, their features are learned independently with group convolutions firstly and then the learned features of different modalities are fused at multiple scales with standard convolutions. Therefore, the multi-scale fusion of multi-modal features is completed in a single-branch convolutional network. In this way, the computational cost is reduced while the experimental results reveal that we can still get promising results.

The performance of object instance segmentation in remote sensing images has been greatly improved through the introduction of many landmark frameworks based on convolutional neural network. However, the object densely issue still affects the accuracy of such segmentation frameworks. Objects of the same class are easily confused, which is most likely due to the close docking between objects. We think context information is critical to address this issue. So, we propose a novel framework called SLCMASK-Net, in which a sequence local context module (SLC) is introduced to avoid confusion between objects of the same class. The SLC module applies a sequence of dilation convolution blocks to progressively learn multi-scale context information in the mask branch. Besides, we try to add SLC module to different locations in our framework and experiment with the effect of different parameter settings. Comparative experiments are conducted on remote sensing images acquired by QuickBird with a resolution of 0.5m - 1m and the results show that the proposed method achieves state-of-the-art performance.

In this paper, we present a semi-supervised remote sensing change detection method based on graph model with Generative Adversarial Networks (GANs). Firstly, the multi-temporal remote sensing change detection problem is converted as a problem of semi-supervised learning on graph where a majority of unlabeled nodes and a few labeled nodes are contained. Then, GANs are adopted to generate samples in a competitive manner and help improve the classification accuracy. Finally, a binary change map is produced by classifying the unlabeled nodes to a certain class with the help of both the labeled nodes and the unlabeled nodes on graph. Experimental results carried on several very high resolution remote sensing image data sets demonstrate the effectiveness of our method.

Deep learning based object detection has achieved great success. However, these supervised learning methods are data-hungry and time-consuming. This restriction makes them unsuitable for limited data and urgent tasks, especially in the applications of remote sensing. Inspired by the ability of humans to quickly learn new visual concepts from very few examples, we propose a training-free, one-shot geospatial object detection framework for remote sensing images. It consists of (1) a feature extractor with remote sensing domain knowledge, (2) a multi-level feature fusion method, (3) a novel similarity metric method, and (4) a 2-stage object detection pipeline. Experiments on sewage treatment plant and airport detections show that proposed method has achieved a certain effect. Our method can serve as a baseline for training-free, one-shot geospatial object detection.

Semantic segmentation in high-resolution aerial images is a fundamental research problem in remote sensing field for its wide range of applications. However, it is difficult to distinguish regions with similar spectral features using only multispectral data. Recent research studies have indicated that the introduction of multisource information can effectively improve the robustness of segmentation method. In this letter, we use digital surface models (DSMs) information as a complementary feature to further improve the semantic segmentation results. To this end, we propose a lightweight and simple DSM fusion (DSMF) branch structure module. Compared with the existing feature extraction structures, proposed DSMF module is simple and can be easily applied to other networks. In addition, we investigate four fusion strategies based on DSMF module to explore the optimal feature fusion strategy and four end-to-end DSMFNets are designed according to the corresponding strategies. We evaluate our models on International Society for Photogrammetry and Remote Sensing Vaihingen data set and all DSMFNets achieve promising results. In particular, DSMFNet-1 achieves an overall accuracy of 91.5% on the test data set.

An algorithm based on optical flow is proposed to detect a ground moving target via the single-channel synthetic aperture radar. First, the signal models of uniform moving targets are established and classified into three types. Next, the Doppler spectrum is divided to generate a multilook image sequence. Then, the motion feature of a moving target response is described in the image sequence, in which the optical flow is introduced to realize the moving target detection. The detection results of real moving targets are obtained after the false alarm elimination based on the response motion relevance. This algorithm has a large range of detectable velocity and can even be applied to detect the moving targets with acceleration. In addition, compared with constant false alarm rate method, the optical flow has a better anti-interference performance against the strong static scatters. Finally, some numerical experiments are provided to demonstrate the effectiveness of the proposed method.

This letter presents a novel supervised change detection method based on a deep siamese semantic network framework, which is trained by using improved triplet loss function for optical aerial images. The proposed framework can not only extract features directly from image pairs which include multiscale information and are more abstract as well as robust, but also enhance the interclass separability and the intraclass inseparability by learning semantic relation. The feature vectors of the pixels pair with the same label are closer, and at the same time, the feature vectors of the pixels with different labels are farther from each other. Moreover, we use the distance of the feature map to detect the changes on the difference map between the image pair. Binarized change map can be obtained by a simple threshold. Experiments on optical aerial image data set validate that the proposed approach produces comparable, even better results, favorably to the state-of-the-art methods in terms of F-measure.

Object detection has been playing a significant role in the field of remote sensing for a long time but it is still full of challenges. In this paper, we propose a novel detection framework based on rotational region convolution neural network to cope with the problem of non-maximum suppression in dense objects detection. The bounding boxes obtained by adopting our method is the minimum bounding rectangle of object with less redundant regions. Furthermore, we find the head direction of the object through prediction. There are three important changes to our framework over traditional detection methods, representation and regression of rotational bounding box, head direction prediction and rotational non-maximal suppression. Experiments based on remote sensing images from Google Earth for Object detection show that our detection method based on rotational region CNN has a competitive performance.

Road extraction from high-resolution remote sensing images has been applied in many domains, but it is still full of challenges. We focus on the problem of slender roads, proposing a new multiple feature pyramid network (MFPN), which is composed of an effective feature pyramid and the tailored pyramid pooling module based on PSPNet. These two designs can address the sparsity of roads in remote sensing images via using multi-level semantic features. Experiments on remote sensing images from Quick Bird show that our MFPN model achieves competitive performance, especially for slender roads.

Semantic segmentation of aerial images refers to assigning one land cover category to each pixel. This is a challenging task due to the great differences in the appearances of ground objects. Many attempts have been made during the past decades. In recent years, convolutional neural networks (CNNs) have been introduced in the remote sensing field, and various solutions have been proposed to realize dense semantic labeling with CNNs. In this letter, we propose shuffling CNNs to realize semantic segmentation of aerial images in a periodic shuffling manner. This approach is a supplement to current methods for semantic segmentation of aerial images. We propose a naive version and a deeper version of this method, and both are adept at detecting small objects. Additionally, we propose a method called field-of-view (FoV) enhancement that can enhance the predictions. This method can be applied to various networks, and our experiments verify its effectiveness. The final results are further improved through an ensemble method that averages the score maps generated by the models at different checkpoints of the same network. We evaluate our models using the ISPRS Vaihingen and Potsdam data sets, and we acquire promising results using these two data sets.

Water-body segmentation is an important issue in remote sensing and image interpretation. Classic methods for counteracting this problem usually include the construction of index features by combining different spectra, however, these methods are essentially rule-based and fail to take advantage of context information. Additionally, as the quality of image resolution improves, these methods are proved to be inadequate. With the rise of convolutional neural networks (CNN), the level of research about segmentation has taken a huge leap, but the field is still facing an increasing demand for data and the problem of blurring boundaries. In this letter, a new segmentation network called restricted receptive field deconvolution network (RRF DeconvNet) is proposed, with which to extract water bodies from high-resolution remote sensing images. Compared with natural images, remote sensing images have a weaker pixel neighborhood relativity; in consideration of this challenge, an RRF DeconvNet compresses the redundant layers in the original DeconvNet and no longer relies on a pretrained model. In addition, to tackle the blurring boundaries that occur in CNN, a new loss function called edges weighting loss is proposed to train segmentation networks, which has been shown to significantly sharpen the segmentation boundaries in results. Experiments, based on Google Earth images for water-body segmentation, are presented in this letter to prove our method.

Cloud and cloud shadow detection in remote sensing imagery is important for its wide range of applications. Traditionally, the detection is usually based on the manually designed thresholds from multiband, which is complicated and of multistage. To simplify the process of cloud and cloud shadow detection and improve the performance, we propose a multilevel feature fused segmentation network (MFFSNet), which can be trained end-to-end without any hand-tuned parameters. Specifically, a fully convolutional network is proposed for cloud and cloud shadow features learning. Then, we utilize a novel pyramid pooling module to extract contextual relation between cloud and shadow. Furthermore, a special multilevel feature fused structure is designed to combine semantic information with spatial information from different levels, so that we can better handle the multiscale objects and produce detailed segmentation boundaries. Experiments show that the MFFSNet outperforms the state-of-the-art methods and achieves high accuracies of 98.69% and 98.92% for cloud and cloud shadow detection.

Building extraction from remote sensing images is a longstanding topic in land use analysis and applications of remote sensing. Variations in shape and appearance of buildings, occlusions and other unpredictable factors increase the hardness of automatic building extraction. Numerous methods have been proposed during the last several decays, but most of these works are task oriented and lack of generalization. This paper applys deep learning to building extraction in a supervised manner. A deep deconvolution neural network with 27 Convolution/Deconvolution weight layers is designed to realize building extraction in pixel level. As such a deep network is prone to overfitting, a data augment method that suits pixel-wise prediction tasks in remote sensing is suggested. Moreover, an overall training and inferencing architecture is proposed. Our methods are finally applied to building extraction tasks and get competitive results with other methods published.

Semantic segmentation for remote sensing images is a critical process in the workflow of object-based image analysis. Recently, convolutional neural networks(CNNs) are powerful visual models that yield hierarchies of features. In this paper, we propose a deep convolutional encoder-decoder model for remote sensing images segmentation. Specifically, we rely on the encoder network to extract the high-level semantic feature of ultra-high resolution images and the decoder network is employed to map the low resolution encoder feature maps to full input resolution feature maps for pixel-wise labeling. Also the fully connected conditional random field (CRF) is integrated into the model so that the network can be trained end-to-end. Experiments on the Vaihingen dataset demonstrate that our model can make promising performance.

Automatic inshore ship recognition, which includes target localization and type recognition, is an important and challenging task. However, existing ship recognition methods mainly focus on the classification of ship samples or clips. These methods rely deeply on the detection algorithm to complete localization and recognition in large scene images. In this letter, we present an integrated framework to automatically locate and recognize inshore ships in large scene satellite images. Different from traditional object recognition methods using two steps of detection-classification, the proposed framework could locate inshore ships and identify types without the detection step. Considering ship size is a useful feature, a novel multimodel method is proposed to utilize this feature. And an Euclidean-distance-based fusion strategy is used to combine candidates given by models. This fusion strategy could effectively separate side-by-side ships. To handle large scene images efficiently, scale-invariant feature transform registration is also integrated into the framework to utilize geographic information. All of these make the framework an end-to-end fashion which could automatically recognize inshore ships in large scene satellite images. Experiments on Quickbird images show that this framework could achieve the actual applied requirements.

In this letter, we propose a novel supervised change detection method based on a deep siamese convolutional network for optical aerial images. We train a siamese convolutional network using the weighted contrastive loss. The novelty of the method is that the siamese network is learned to extract features directly from the image pairs. Compared with hand-crafted features used by the conventional change detection method, the extracted features are more abstract and robust. Furthermore, because of the advantage of the weighted contrastive loss function, the features have a unique property: the feature vectors of the changed pixel pair are far away from each other, while the ones of the unchanged pixel pair are close. Therefore, we use the distance of the feature vectors to detect changes between the image pair. Simple threshold segmentation on the distance map can even obtain good performance. For improvement, we use a k-nearest neighbor approach to update the initial result. Experimental results show that the proposed method produces results comparable, even better, with the two state-of-the-art methods in terms of F-measure.

In this letter, we propose a novel method to reconstruct accurate building structures from high-resolution multiview aerial imagery, using layered contour fitting (LCF) with a density-based clustering algorithm. Initially, the complicated 3-D scene is reconstructed by a probabilistic volumetric modeling algorithm. Subsequently, the reconstructed 3-D scene model is projected into layer images based on the height information. At last, we combine an extended layered density-based clustering approach with a generative LCF approach to remove noise and extract accurate building contours in every layer image at the same time. The final accurate 3-D building model is generated from these contours in layer images with a smoothing operation. Experiments on the aerial image sets demonstrate effectiveness and precision of our method.