Self-supervised representation learning has proven constructive for skeleton-based action recognition. For better performance, existing methods mainly focus on 1) multi-modal data augmentations and 2) triplet contrastive samples construction. However, designing these strategies is always heuristics and hard. Instead of exploring more similar strategies, this paper addresses this issue with a different view and proposes a novel Contrastive Spatio-Temporal Clustering (CSTC) module. CSTC constructs a supervised signal (pseudo-label) of action sequences in an online clustering manner, and it is complementary to the recent data augmentations or triplet contrastive samples construction strategies. Specifically, CSTC can be formulated as an optimal transport problem. we introduce the spatio-temporal regularizations into the original optimal transport term to guide the pseudo-label generation, i.e., a semantic regularization learned by frame index is proposed to constrain the frame order, and a prior normal distribution regularization based on sampling characteristics of samples is proposed to maintain the dependability of spatial cluster assignments. Furthermore, to enhance the learning of latent features, we propose a Bidirectional Cross-modal Clustering Consistency Objective (B3CO) to enforce cluster assignments consistency for different modalities of the same sample. Last, since fusing spatial and temporal clustering losses directly during back-propagation will confuse the learned dimension-specific semantics, we propose a simple yet effective training strategy to fix it by training the model using these two losses alternately. By integrating the above designs into the MoCo framework, we propose a Contrastive Spatio-Temporal Clustering Network (CSTCN), which can excavate cross-modal discriminative spatio-temporal features in the clustering space. Experimental results on NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD II datasets show that CSTCN achieves state-of-the-art performance in both single- and multi-modal models, especially in the KNN and semi-supervised evaluation protocols. Besides, the key module CSTC shows good generalization capability, and achieves consistent performance improvement on the basis of several state-of-the-art methods which focus on data augmentations and triplet contrastive samples construction.

The popular networks for change detection (CD) in very-high-resolution (VHR) remote sensing (RS) images usually suffer from two problems. First, it is difficult for these networks to model simultaneously the local and global features of changed targets, which leads to the limited feature representation ability of popular CD networks. Second, these networks often have a large number of parameters and high computational costs due to complex network architecture. To address the above issues, we propose a local-global siamese network (LGS-Net) for CD in VHR RS images. First, we design an encoder with a parallel dual-branch structure consisting of convolutional neural networks (CNNs) and Transformer to extract rich features from bi-temporal images. Furthermore, we design a local-global feature enhancement (LGFE) module to help our encoder improve its feature representation ability. Second, we design a compact and efficient convolution module called inter-scale separable convolution (ISSConv). This module first divides feature maps into multiple groups, and then performs depthwise separable convolution in each group using atrous convolution with different dilation rates, which can not only capture changed targets across scales but also effectively reduce the number of model parameters. Experiments demonstrate that the proposed LGS-Net is superior to the state-of-the-art CD networks in terms of parameters, computational costs, and detection accuracy.

Building footprint extraction in remote sensing remains challenging due to the diverse appearances of buildings and confusing scenarios. Recently, researchers have revealed that both the globality and locality are vitally important in building footprint extraction tasks and proposed to incorporate the local context and global long-range dependency in the segmentation models. However, the inadequate integration of the globality and locality still leads to incomplete, fake or missing extraction results. To alleviate these problems, a novel segmentation method named Bi-branch Cross-fusion Transformer Network (BCTNet) is proposed in this study. Two parallel branches of the convolutional encoder branch (CB) and the transformer encoder branch (TB) are designed to extract multi-scale feature maps. A concatenation-then-cross-fusion transformer block (CCTB) is put forward to integrate the locality from the CB and globality from the TB in a cross-fusion way at each stage of the encoding process. Then, an adaptive gating module (AGM) is proposed to gate the feature maps from the CCTB to strengthen the important features while suppressing the irrelevant interference information. After that, the segmentation results can be obtained through a simple decoding process. Comprehensive experiments on two benchmark datasets demonstrate that the proposed BCTNet can achieve superior performance compared to the current state-of-the-art (SOTA) segmentation methods.

Facing the challenge of small open labeled data sets in remote sensing change detection, this letter proposes a novel supervised change detection method by taking advantages of capsule network which can reach the same performance as traditional convolutional neural networks (CNNs) but with less training data. To achieve this aim, we propose a pseudo-Siamese capsule network which takes both rotational invariance and spatial hierarchies between features into account for aerial images change detection. First, the features of image pairs are extracted by two identical nonshared weights convolutional capsule networks. Second, the extracted features are directly concatenated and sent to another convolutional capsule layer. The change probability map is obtained by calculating the length of the capsule vectors in the final layer. Additionally, to reduce the influence of imbalance samples when we optimize our network, we design a margin-focal loss function to pay more attention to the misclassified samples. Finally, binary change map can be produced by a simple threshold. Experimental results carried out on the SZTAKI AirChange Benchmark Set show that the proposed method achieves comparable and even better results with existing state-of-the-art methods in terms of F-measure.

The state-of-the-art deep neural networks have made a great breakthrough in remote sensing image classification. However, the heavy dependence on large-scale data sets limits the application of the deep learning to synthetic aperture radar (SAR) automatic target recognition (ATR) field where the target sample set is generally small. In this work, a metalearning framework named MSAR, consisting of a metalearner and a base-learner, is proposed to solve the sample restriction problem, which can learn a good initialization as well as a proper update strategy. After training, MSAR can implement fast adaptation with a few training images on new tasks. To the best of our knowledge, this is the first study to solve a few-shot SAR target classification via metalearning. In particular, the few-task problem is defined by analyzing the effect of available training classes on the performance of metalearning models. In order to reduce the metalearning difficulties caused by the few-task problem, three transfer-learning methods are employed, which can leverage the prior knowledge from the pretraining phase. Besides, we design a hard task mining method for effective metalearning. Based on the Moving and Stationary Target Acquisition and Recognition (MSTAR) data set, a specialized data set named NIST-SAR is devised to train and evaluate the proposed method. The experiments on NIST-SAR have shown that the proposed method yields better performances with the largest absolute improvements of 1.7% and 2.3% for 1-shot and 5-shot, respectively, over the next best, which indicates that the proposed method is promising and metalearning is a feasible solution for few-shot SAR ATR.

Object detection in aerial images is important for a wide range of applications. The most challenging dilemma in this task is the arbitrary orientation of objects, and many deep-learning-based methods are proposed to address this issue. In previous works on oriented object detection, the regression-based method for object localization has limited performance due to the shortage of spatial information. And the models suffer from the divergence of feature construction for object recognition and localization. In this article, we propose a novel architecture, i.e., point-based estimator to remedy these problems. To utilize the spatial information explicitly, the detector encodes an oriented object with a point-based representation and operates a fully convolutional network for point localization. To improve localization accuracy, the detector takes the manner of coarse-to-fine to lessen the quantization error in point localization. To avoid the discrepancy of feature construction, the detector decouples localization and recognition with individual pathways. In the pathway of object recognition, the instance-alignment block is involved to ensure the alignment between the feature map and oriented region. Overall, the point-based estimator can be easily embedded into the region-based detector and leads to significant improvement on oriented object detection. Extensive experiments have demonstrated the effectiveness of our point-based estimator. Compared with existing works, our method shows state-of-the-art performance on oriented object detection in aerial images.

Motivated by the development of deep convolution neural networks (DCNNs), aircraft detection has gained tremendous progress. State-of-the-art DCNN-based detectors mainly belong to top-down approaches, which enumerate massive potential locations of aircraft with the form of rectangular regions, and then identify whether they are objects or not. Compared with these top-down detectors, this article shows that aircraft detection via a type of bottom-up method can have better performances in the era of deep learning. In this article, we propose a novel bottom-up detector named X-LineNet. It formulates the aircraft detection task as prediction and clustering of paired intersecting line segments inside each target. Aircraft detection is then a purely appearance-based line segments estimation problem, without any rectangular regions classification or implicit features learning. With simple postprocessing, X-LineNet can simultaneously provide multiple representation forms of the detection result: the horizontal bounding box, the oriented bounding box, and the pentagonal mask. The pentagonal mask is a more accurate representation form of aircraft which has less redundancy than that of a rectangular box. Experiments show that X-LineNet outperforms prevalent top-down and region-based detectors on UCAS-AOD, NWPU VHR-10, and DIOR public data sets in the field of aircraft detection.

As common cancer, breast cancer kills thousands of women every year. It’s significant to provide doctors computer-aided diagnosis (CAD) to ease their workload as well as improve detection quality. Patch-level CNNs are usually used to classify the breast tissue slice, and the CNNs classify each patch independently ignoring the spatial correlations, resulting in wrong isolated label map. However, the probability distribution of cancer type is related to their adjacent patches. In this paper, we propose a framework integrating CNN and filter algorithm aimed at extracting spatial information and improving the performance of the classification. The network was trained on a breast cancer dataset provided by ICIAR18. For 4-class classification, compared to CNN methods without using spatial correlations, the proposed method achieved about 10% improvement on accuracy over the validation dataset and get smoother probability maps. Our experiments also show that larger kernel size gets better performance. The code is available at https://github.com/dong100136/Breast-Cancer-Image-Classification-On-WSI-With-Spatial-Correlations.

Object detection has been a building block in computer vision. Though considerable progress has been made, there still exist challenges for objects with small size, arbitrary direction, and dense distribution. Apart from natural images, such issues are especially pronounced for aerial images of great importance. This paper presents a novel multi-category rotation detector for small, cluttered and rotated objects, namely SCRDet. Specifically, a sampling fusion network is devised which fuses multi-layer feature with effective anchor sampling, to improve the sensitivity to small objects. Meanwhile, the supervised pixel attention network and the channel attention network are jointly explored for small and cluttered object detection by suppressing the noise and highlighting the objects feature. For more accurate rotation estimation, the IoU constant factor is added to the smooth L1 loss to address the boundary problem for the rotating bounding box. Extensive experiments on two remote sensing public datasets DOTA, NWPU VHR-10 as well as natural image datasets COCO, VOC2007 and scene text data ICDAR2015 show the state-of-the-art performance of our detector. The code and models will be available at https://github.com/DetectionTeamUCAS.

Building reconstruction from SAR images is a hot topic in recent years. Currently, related methods mainly deal with on flat-roofed buildings. In this paper, we extend the research scope to gable-roofed buildings, and try to present a parameterized geometrical model for the layover of gable-roofed buildings. Based on this model, a top-down building reconstruction technique based on MCMC method is proposed. Through representing the layover with parameterized geometrical models, building reconstruction is converted into an optimization problem under the Bayesian scheme. In order to obtain global optima, simulated annealing algorithm with MCMC is used in the optimization stage. Two groups of transmission kernels which are responsible for model updates are designed according to the model. Experiments show that the layover model is accurate and the reconstruction method is effective.

In this paper, we achieve a semantic segmentation of aerial imagery based on the fusion of multi-modal data in an effective way. The multi-modal data contains a true orthophoto and the corresponding normalized Digital Surface Model (nDSM), which are stacked together before they are fed into a Convolutional Neural Network (CNN). Though the two modalities are fused at the early stage, their features are learned independently with group convolutions firstly and then the learned features of different modalities are fused at multiple scales with standard convolutions. Therefore, the multi-scale fusion of multi-modal features is completed in a single-branch convolutional network. In this way, the computational cost is reduced while the experimental results reveal that we can still get promising results.

Accurate automatic conversion between aerial images and maps is a valuable and challenging task in computer vision and computer graphics. Deep convolutional neural networks (CNN) have achieved promising results on this task but the results accuracy is not ideal. In this paper, we propose a solution to improve the precision and quality of the transforming results. The core learning method is based on generative adversarial networks (GANs). A novel generator and a multi-scale discriminator are introduced in our network. The generator operates at the progressive method to gurantee the spatial consistency between the inputs and outputs, and our multi-scale discriminator focuses on increasing the capacity of the network and guides the generator to generate better results. In particular, our architecture can also be used as a general neural network for style translation. Analytic experiments on the aerial-to-map dataset show that our network outperforms the existing method, advancing both accuracy and visual appearance.

The local climate zone (LCZ) classification divides the urban areas into 17 categories, which are composed of 10 manmade structures and 7 natural landscapes. Though originally designed for temperature study, LCZ classification can be used for studies on economy and population. In this paper, we achieve a LCZ classification with convolutional neural networks based on the multi-source remote sensing data, including the polarimetric synthetic aperture radar (PolSAR) data and the corresponding multi-spectral imagery (MSI). Through experiments we attempt to reveal the contributions of the SAR data and the MSI to the classification performance. Furthermore, we emphasize the crucial importance of the preprocessing on the training data to derive a balanced dataset. We are ranked second in the Tianchi competition rankings when we submit our results.

Deep learning based object detection has achieved great success. However, these supervised learning methods are data-hungry and time-consuming. This restriction makes them unsuitable for limited data and urgent tasks, especially in the applications of remote sensing. Inspired by the ability of humans to quickly learn new visual concepts from very few examples, we propose a training-free, one-shot geospatial object detection framework for remote sensing images. It consists of (1) a feature extractor with remote sensing domain knowledge, (2) a multi-level feature fusion method, (3) a novel similarity metric method, and (4) a 2-stage object detection pipeline. Experiments on sewage treatment plant and airport detections show that proposed method has achieved a certain effect. Our method can serve as a baseline for training-free, one-shot geospatial object detection.

Image-to-sketch translation is to learn the mapping between an image and a corresponding human drawn sketch. Machine can be trained to mimic the human drawing process using a training set of aligned image-sketch pairs. However, to collect such paired data is quite expensive or even unavailable for many cases since sketches exhibit various level of abstractness and drawing preferences. Hence we present an approach for learning an image-to-sketch translation network via unpaired examples. A translation network, which can translate the representation in image latent space to sketch domain, is trained in unsupervised setting. To prevent the problem of representation shifting in cross-domain translation, a novel cycle+ consistency loss is explored. Experimental results on sketch recognition and sketch-based image retrieval demonstrate the effectiveness of our approach.

An algorithm based on optical flow is proposed to detect a ground moving target via the single-channel synthetic aperture radar. First, the signal models of uniform moving targets are established and classified into three types. Next, the Doppler spectrum is divided to generate a multilook image sequence. Then, the motion feature of a moving target response is described in the image sequence, in which the optical flow is introduced to realize the moving target detection. The detection results of real moving targets are obtained after the false alarm elimination based on the response motion relevance. This algorithm has a large range of detectable velocity and can even be applied to detect the moving targets with acceleration. In addition, compared with constant false alarm rate method, the optical flow has a better anti-interference performance against the strong static scatters. Finally, some numerical experiments are provided to demonstrate the effectiveness of the proposed method.

Deeper architectures are proven to be beneficial for the classification performance obviously in computer vision field. Inspired by this, deep CNN s are expected to make progress in the SAR target classification problem as well. However, it is hard to train deeper CNNs for SAR images. Such CNNs have millions of parameters to be determined in the network (for example the VGGNet has more than 130 million parameters), hence large-scale dataset is indispensable when training a deep CNN. But there is no large-scale annotated SAR target dataset, and data acquisition and annotation is much more costly for SAR images. With inadequate data, the network is easy to be overfitting. Several methods based on deep learning have been proposed for SAR image classifications, but they cannot get rid of the aforementioned data limitation of labelled SAR images. To solve this problem, this paper proposes a microarchitecture called CompressUnit (CU). With CU, we design a deeper CNN. Compared with the network with the fewest parameters for SAR image classification in literature so far, our network is 2X deeper with only about 10% of parameters. In this way, we get a deeper network with much fewer parameters. This network is easier to be trained with limited SAR data and is more likely to get rid of overfitting.

Road extraction from high-resolution remote sensing images has been applied in many domains, but it is still full of challenges. We focus on the problem of slender roads, proposing a new multiple feature pyramid network (MFPN), which is composed of an effective feature pyramid and the tailored pyramid pooling module based on PSPNet. These two designs can address the sparsity of roads in remote sensing images via using multi-level semantic features. Experiments on remote sensing images from Quick Bird show that our MFPN model achieves competitive performance, especially for slender roads.

In this paper, we propose a top-down building reconstruction technique based on layover modelling and MCMC method. Through representing the layover with parameterized geometrical models, the problem is converted into an optimization problem under the Bayesian scheme. The energy function consists of two parts: region part and edge part. In order to obtain global optima, simulated annealing algorithm with MCMC is used in the optimization stage. Two groups of transmission kernels which are responsible for model updates are designed according to the model. This method is tested both on simulated SAR image and HR TanDEM-X data. At this moment, only qualitative analysis for this method is provided. It proves the effectiveness of the presented method. Detailed quantitative evaluation will be added when we submit the final version of this paper.

In this paper, a fully automatic building reconstruction method for high resolution interferometric synthetic aperture radar (InSAR) data is presented. This method is based on stochastic geometrical model. Firstly, a building detection procedure is implemented on the big image and the entire scene is divided into building clips. After that, the reconstruction process is utilized for each building clip. In the reconstruction process, a building in 3D space is projected to the image plane and then decomposed to feature regions including layover, corner line, roof and shadow. We explore the statistic properties of the each region, and include it in the posterior function, together with the edge term and the prior we defined. Finally, in order to overcome local optima, a group of special transmission kernels are designed. The experimental results on TanDEM-X data demonstrate the effectiveness of our method.

The interferometric coherence map is derived from the cross-correlation of two registered synthetic aperture radar (SAR) images. It can give additional information complementary to the intensity image, or act as an independent information source in many applications. Compared to the plenty of work on SAR intensity statistics, there are quite fewer researches on the statistical characters of interferometric SAR (InSAR) coherence. And to our knowledge, all of the existing work that related to InSAR coherence statistics, models the coherence with Gaussian distribution with no discrimination on data resolutions or scene types. Our main contribution is the investigation on the accuracies of several typical models for high resolution coherence statistics over urban areas. We select three typical land classes including trees, buildings, and shadow, as the representatives of urban areas. And different models including Gaussian, Weibull, Rayleigh, Nakagami and Beta are evaluated. Experiment results on TanDEM-X data illustrate that the Beta model reveals a better performance than other distributions. Finally, the Beta model is used in the detection of buildings.

Layover in slant range synthetic aperture radar (SAR) images contains rich 3-D information of building geometry. In this letter, a coarse-to-fine method for building reconstruction from high-resolution (HR) SAR layover map using restricted parametric geometrical models is presented. First, we propose a new restricted parametric geometrical model for building layover and for corner line, respectively. Under the guidance of these models, a hierarchical coarse-to-fine layover estimation scheme is designed. Owning to the coarse-to-fine scheme, this method is resistant to various flaws of layover. At last, the building is reconstructed from the well-estimated layover. Experiments on HR TanDEM-X data demonstrate the effectiveness and precision of our method.