This article studies change detection within pairs of optical images remotely sensed from overhead views. We consider that a high-performance solution to this task entails highly effective multi-level feature interaction. With that in mind, we propose a novel approach characterized by two attentive feature aggregation schemes that handle cross-level features in different processes. For the Siamese-based feature extraction of the bi-temporal image pair, we attach emphasis on constructing semantically strong and contextually rich pyramidal feature representations to enable comprehensive matching and differencing. To this end, we leverage a feature pyramid network and re-formulate its cross-level feature merging procedure as top-down modulation with multiplicative channel attention and additive gated attention. For the multi-level difference feature fusion, we progressively fuse the derived difference feature pyramid in an attend-then-filter manner. This makes the high-level fused features and the adjacent lower-level difference features constrain each other, and thus allows steady feature fusion for specifying change regions. In addition, we build an upsampling head as a replacement for the normal heads followed by static upsampling. Our implementation contains a stack of upsampling modules that allocate features for each pixel. Each has a learnable branch that produces attentive residuals for refining the statically upsampled results. We conduct extensive experiments on four public datasets and results show that our approach achieves state-of-the-art performance. Code is available at https://github.com/xingronaldo/CLAFA.

Practical applications ask for object detection models that achieve high performance at low overhead. Knowledge distillation demonstrates favorable potential in this case by transferring knowledge from a cumbersome teacher model to a lightweight student model. However, previous distillation methods are plagued with massive misleading background information in remote sensing images and ignore investigating the relationships between different instances. In this article, we propose an instance-aware distillation (InsDist for short) method to derive efficient remote sensing object detectors. Our InsDist combines feature-based and relation-based knowledge distillation to make the most of instance-related information in the knowledge transfer from the teacher to the student. On one hand, we propose a parameter-free masking module to decouple instance-related foreground from instance-irrelevant background in multiscale features. On the other hand, we construct the relationships between different instances to enhance the learning of intraclass compactness and interclass dispersion. The student comprehensively imitates both features and relationships from the teacher, yielding considerable effectiveness in dealing with complex remote sensing images. In addition, our InsDist can be easily built on mainstream object detectors with negligible extra cost. Extensive experiments on two large-scale remote sensing object detection datasets, namely DIOR and DOTA, show that our InsDist obtains noticeable gains over other distillation methods for both one-stage and two-stage, as well as both anchor-based and anchor-free detectors. The source code will be publicly available athttps://github.com/swift1988/InsDist.

Detecting objects in remote sensing images (RSIs) using oriented bounding boxes (OBBs) is flourishing but challenging, wherein the design of OBB representations is the key to achieving accurate detection. In this article, we focus on two issues that hinder the performance of the two-stage oriented detectors: 1) the notorious boundary discontinuity problem, which would result in significant loss increases in boundary conditions, and 2) the inconsistency in regression schemes between the two stages. We propose a simple and effective bounding box representation by drawing inspiration from the polar coordinate system and integrate it into two detection stages to circumvent the two issues. The first stage specifically initializes four quadrant points as the starting points of the regression for producing high-quality oriented candidates without any postprocessing. In the second stage, the final localization results are refined using the proposed novel bounding box representation, which can fully release the capabilities of the oriented detectors. Such consistency brings a good trade-off between accuracy and speed. With only flipping augmentation and single-scale training and testing, our approach with ResNet-50-FPN harvests 76.25% mAP on the DOTA dataset with a speed of up to 16.5 frames/s, achieving the best accuracy and the fastest speed among the mainstream two-stage oriented detectors. Additional results on the DIOR-R and HRSC2016 datasets also demonstrate the effectiveness and robustness of our method. The source code is publicly available athttps://github.com/yanqingyao1994/QPDet.

As one of the fundamental research topics in remote sensing image analysis, hyperspectral image (HSI) classification has been extensively studied so far. However, how to discriminatively learn a low-dimensional feature space, in which the mapped features have small within-class scatter and big between-class separation, is still a challenging problem. To address this issue, this paper proposes an effective framework, named compact and discriminative stacked autoencoder (CDSAE), for HSI classification. The proposed CDSAE framework comprises two stages with different optimization objectives, which can learn discriminative low-dimensional feature mappings and train an effective classifier progressively. First, we impose a local Fisher discriminant regularization on each hidden layer of stacked autoencoder (SAE) to train discriminative SAE (DSAE) by minimizing reconstruction error. This stage can learn feature mappings, in which the pixels from the same land-cover class are mapped as nearly as possible and the pixels from different land-cover categories are separated by a large margin. Second, we learn an effective classifier and meanwhile update DSAE with a local Fisher discriminant regularization being embedded on the top of feature representations. Moreover, to learn a compact DSAE with as small number of hidden neurons as possible, we impose a diversity regularization on the hidden neurons of DSAE to balance the feature dimensionality and the feature representation capability. The experimental results on three widely-used HSI data sets and comprehensive comparisons with existing methods demonstrate that our proposed method is effective.

The performance of object detection has recently been significantly improved due to the powerful features learnt through convolutional neural networks (CNNs). Despite the remarkable success, there are still several major challenges in object detection, including object rotation, within-class diversity, and between-class similarity, which generally degenerate object detection performance. To address these issues, we build up the existing state-of-the-art object detection systems and propose a simple but effective method to train rotation-invariant and Fisher discriminative CNN models to further boost object detection performance. This is achieved by optimizing a new objective function that explicitly imposes a rotation-invariant regularizer and a Fisher discrimination regularizer on the CNN features. Specifically, the first regularizer enforces the CNN feature representations of the training samples before and after rotation to be mapped closely to each other in order to achieve rotation-invariance. The second regularizer constrains the CNN features to have small within-class scatter but large between-class separation. We implement our proposed method under four popular object detection frameworks, including region-CNN (R-CNN), Fast R- CNN, Faster R- CNN, and R- FCN. In the experiments, we comprehensively evaluate the proposed method on the PASCAL VOC 2007 and 2012 data sets and a publicly available aerial image data set. Our proposed methods outperform the existing baseline methods and achieve the state-of-the-art results.

Despite the promising progress made in recent years, person reidentification (re-ID) remains a challenging task due to the complex variations in human appearances from different camera views. This paper proposes to tackle this task by jointly learning feature representation and distance metric in an end-to-end manner. Existing deep metric learning-based re-ID methods usually encounter the following two weaknesses: 1) most works based on pairwise or triplet constraints often suffer from slow convergence and poor local optima, partially because they use very limited samples for each update and 2) hard negative sample mining has been widely applied in existing works. However, hard positive samples, which also contribute to the training of network, have not received enough attention. To alleviate these problems, we develop a novel structural metric learning objective for person re-ID, in which each positive pair is allowed to be compared against all negative pairs in a minibatch and each positive pair is adaptively assigned a hardness-aware weight to modulate its contribution. The introduced positive pair weighting strategy enables the algorithm to focus more on the hard positive samples. Furthermore, we propose to enhance the proposed loss function by adding a global loss term to reduce the variances of positive/negative pair distances, which is able to improve the generalization capability of the network model. By this approach, person images can be nonlinearly mapped into a low-dimensional embedding space where similar samples are kept closer and dissimilar samples are pushed farther apart. We implement the proposed algorithm using the inception architecture and evaluate it on three large-scale re-ID data sets. Experiment results demonstrate that our approach is able to outperform most state of the arts while using much lower dimensional deep features.

Image set classification has attracted much attention because of its broad applications. Despite the success made so far, the problems of intra-class diversity and inter-class similarity still remain two major challenges. To explore a possible solution to these challenges, this paper proposes a novel approach, termed duplex metric learning (DML), for image set classification. The proposed DML consists of two progressive metric learning stages with different objectives used for feature learning and image classification, respectively. The metric learning regularization is not only used to learn powerful feature representations but also well explored to train an effective classifier. At the first stage, we first train a discriminative stacked autoencoder (DSAE) by layer-wisely imposing a metric learning regularization term on the neurons in the hidden layers and meanwhile minimizing the reconstruction error to obtain new feature mappings in which similar samples are mapped closely to each other and dissimilar samples are mapped farther apart. At the second stage, we discriminatively train a classifier and simultaneously fine-tune the DSAE by optimizing a new objective function, which consists of a classification error term and a metric learning regularization term. Finally, two simple voting strategies are devised for image set classification based on the learnt classifier. In the experiments, we extensively evaluate the proposed framework for the tasks of face recognition, object recognition, and face verification on several commonly-used data sets and state-of-the-art results are achieved in comparison with existing methods.

Thanks to the powerful feature representations obtained through deep convolutional neural network (CNN), the performance of object detection has recently been substantially boosted. Despite the remarkable success, the problems of object rotation, within-class variability, and between-class similarity remain several major challenges. To address these problems, this paper proposes a novel and effective method to learn a rotation-invariant and Fisher discriminative CNN (RIFD-CNN) model. This is achieved by introducing and learning a rotation-invariant layer and a Fisher discriminative layer, respectively, on the basis of the existing high-capacity CNN architectures. Specifically, the rotation-invariant layer is trained by imposing an explicit regularization constraint on the objective function that enforces invariance on the CNN features before and after rotating. The Fisher discriminative layer is trained by imposing the Fisher discrimination criterion on the CNN features so that they have small within-class scatter but large between-class separation. In the experiments, we comprehensively evaluate the proposed method for object detection task on a public available aerial image dataset and the PASCAL VOC 2007 dataset. State-of-the-art results are achieved compared with the existing baseline methods.

Scene classification of high resolution remote sensing images plays an important role for a wide range of applications. While significant efforts have been made in developing various methods for scene classification, most of them are based on handcrafted or shallow learning-based features. In this paper, we investigate the use of deep convolutional neural network (CNN) for scene classification. To this end, we first adopt two simple and effective strategies to extract CNN features: (1) using pre-trained CNN models as universal feature extractors, and (2) domain-specifically fine-tuning pre-trained CNN models on our scene classification dataset. Then, scene classification is carried out by using simple classifiers such as linear support vector machine (SVM). In our work, three off-the-shelf CNN models including AlexNet [1], VGGNet [2], and GoogleNet [3] are investigated. Comprehensive evaluations on a publicly available 21 classes land use dataset and comparisons with several state-of-the-art approaches demonstrate that deep CNN features are effective for scene classification of high resolution remote sensing images.

Automatic semantic annotation of high-resolution optical satellite images is a task to assign one or several predefined semantic concepts to an image according to its content. The fundamental challenge arises from the difficulty of characterizing complex and ambiguous contents of the satellite images. To address this challenge, a diversity constrained joint multi-feature learning method is proposed to learn robust feature representations for annotating satellite images. The key motivation of our method is to make full use of the complementarity diversity information among the heterogeneous features in the learning process. Comprehensive experiments on an annotation dataset demonstrate the superiority and effectiveness of our method compared with baseline multi-feature learning method.

Object detection in very high resolution optical remote sensing images is a fundamental problem faced for remote sensing image analysis. Due to the advances of powerful feature representations, machine-learning-based object detection is receiving increasing attention. Although numerous feature representations exist, most of them are handcrafted or shallow-learning-based features. As the object detection task becomes more challenging, their description capability becomes limited or even impoverished. More recently, deep learning algorithms, especially convolutional neural networks (CNNs), have shown their much stronger feature representation power in computer vision. Despite the progress made in nature scene images, it is problematic to directly use the CNN feature for object detection in optical remote sensing images because it is difficult to effectively deal with the problem of object rotation variations. To address this problem, this paper proposes a novel and effective approach to learn a rotation-invariant CNN (RICNN) model for advancing the performance of object detection, which is achieved by introducing and learning a new rotation-invariant layer on the basis of the existing CNN architectures. However, different from the training of traditional CNN models that only optimizes the multinomial logistic regression objective, our RICNN model is trained by optimizing a new objective function via imposing a regularization constraint, which explicitly enforces the feature representations of the training samples before and after rotating to be mapped close to each other, hence achieving rotation invariance. To facilitate training, we first train the rotation-invariant layer and then domain-specifically fine-tune the whole RICNN network to further boost the performance. Comprehensive evaluations on a publicly available ten-class object detection data set demonstrate the effectiveness of the proposed method.

Effective representation and classification of scenes using very high resolution (VHR) remote sensing images cover a wide range of applications. Although robust low‐level image features have been proven to be effective for scene classification, they are not semantically meaningful and thus have difficulty to deal with challenging visual recognition tasks. In this study, the authors propose a new and effective auto‐encoder‐based method to learn a shared mid‐level visual dictionary. This dictionary serves as a shared and universal basis to discover mid‐level visual elements. On the one hand, the mid‐level visual dictionary learnt using machine learning technique is more discriminative and contains rich semantic information, compared with the traditional low‐level visual words. On the other hand, the mid‐level visual dictionary is more robust to occlusions and image clutters. In the authors' scene‐classification scheme, they use discriminative mid‐level visual elements, rather than individual pixels or low‐level image features, to represent images. This new image representation is able to capture much of the high‐level meaning and contents of the image, facilitating challenging remote sensing image scene‐classification tasks. Comprehensive evaluations on a challenging VHR remote sensing images data set and comparisons with state‐of‐the‐art approaches demonstrate the effectiveness and superiority of their study.

In this paper we present a conceptually simple but surprisingly effective multi-class geospatial object detection method based on Collection of Part Detectors (COPD), which can be easily scaled to a larger number of object classes. The presented COPD is composed of a set of representative and discriminative part detectors, where each part detector is a linear support vector machine (SVM) classifier trained using a weakly supervised learning method that only requires image labels indicating the presence of objects for the training data. Here, each part detector corresponds to a particular viewpoint of an object class, so the collection of them provides a feasible solution for rotation-invariant and simultaneous detection of multi-class geospatial objects. Comprehensive evaluations on high-spatial-resolution remote sensing images and comparisons with a number of state-of-the-art approaches demonstrate the effectiveness and superiority of the presented method.