Ship targets in remote sensing images can be categorized at various granularities due to variations in image quality, ranging from general ship categories to fine-grained classes like Nimitz-class carriers. Traditional studies mainly focus on fine-grained ship classification, often neglecting samples observed at coarser-grained levels. Samples distributed across multiple granularity levels exhibit semantic relationships among their annotated classes, enabling hierarchical knowledge transfer during model training. This paper incorporates two semantic relationships into deep learning-based representation learning and class prediction: parent-child relationships across levels and mutual exclusivity among sibling categories. For hierarchical representation learning, the proposed hierarchical contrastive learning algorithm extracts category-specific representations from input images and aligns them with their semantic relationships, ensuring that parent and child categories share similarities while sibling categories remain distinct. For hierarchical class predictions, a novel consistency loss ensures coherence in probability distributions between parent and child categories. Specially, cross-entropy loss is employed to impose mutual exclusivity among sibling categories. In this paper, a multi-modal dataset is also designedly developed for hierarchical classification, which integrates optical and synthetic aperture radar (SAR) images across multiple hierarchical levels. Experiments on two popular datasets and a multi-modal dataset demonstrate that the proposed method outperforms state-of-the-art approaches in hierarchical multi-granularity ship classification.

Hierarchical multi-granularity classification (HMC) assigns hierarchical multi-granularity labels to each object and focuses on encoding the label hierarchy, e.g., [“Albatross”, “Laysan Albatross”] from coarse-to-fine levels. However, the definition of what is fine-grained is subjective, and the image quality may affect the identification. Thus, samples could be observed at any level of the hierarchy, e.g., [“Albatross”] or [“Albatross”, “Laysan Albatross”], and examples discerned at coarse categories are often neglected in the conventional setting of HMC. In this paper, we study the HMC problem in which objects are labeled at any level of the hierarchy. The essential designs of the proposed method are derived from two motivations: (1) learning with objects labeled at various levels should transfer hierarchical knowledge between levels; (2) lower-level classes should inherit attributes related to upper-level superclasses. The proposed combinatorial loss maximizes the marginal probability of the observed ground truth label by aggregating information from related labels defined in the tree hierarchy. If the observed label is at the leaf level, the combinatorial loss further imposes the multi-class cross-entropy loss to increase the weight of fine-grained classification loss. Considering the hierarchical feature interaction, we propose a hierarchical residual network (HRN), in which granularity-specific features from parent levels acting as residual connections are added to features of children levels. Experiments on three commonly used datasets demonstrate the effectiveness of our approach compared to the state-of-the-art HMC approaches. The code will be available at https://github.com/MonsterZhZh/HRN.

Hierarchical multilabel classification (HMC) assigns multiple labels to each instance with the labels organized under hierarchical relations. In ship classification in remote sensing images, depending on the expert knowledge and image quality, the same type of ships in different remote sensing images may be annotated with different class labels from coarse to fine levels such as merchant ship (MS) or container ship (CTS). In this article, we propose a novel deep network with two output channels and their associated loss functions to learn an HMC classifier using samples labeled at different levels in the hierarchy. In the proposed network, a hierarchy and exclusion (HEX) graph is introduced to model the label hierarchy, which satisfies hierarchical constraints by encoding semantic relations between any two labels. The output nodes of the first channel are organized according to the HEX graph, and its corresponding probabilistic classification loss is built to reflect the hierarchical structure of the HEX graph. On the other hand, the output nodes of the second channel only represent the finest grained (last level in the hierarchy) classes, and its multiclass cross-entropy loss is designed to enhance the discriminative power of the HMC classifier on the last level labels, which is also compatible with constraints in the HEX graph. The combination of these two losses from two output channels can effectively transfer the hierarchical information of ship taxonomy during network training. Experimental results on two commonly used ship datasets demonstrate that the proposed method outperforms the state-of-the-art HMC approaches, and is especially advantageous when trained with fewer fine-grained samples.

When a detection model that has been well-trained on a set of classes faces new classes, incremental learning is always necessary to adapt the model to detect the new classes. In most scenarios, it is required to preserve the learned knowledge of the old classes during incremental learning rather than reusing the training data from the old classes. Since the objects in remote sensing images often appear in various sizes, arbitrary directions, and dense distribution, it further makes incremental learning-based object detection more difficult. In this article, a new architecture for incremental object detection is proposed based on feature pyramid and knowledge distillation. Especially, by means of a feature pyramid network (FPN), the objects with various scales are detected in the different layers of the feature pyramid. Motivated by Learning without Forgetting (LwF), a new branch is expended in the last layer of FPN, and knowledge distillation is applied to the outputs of the old branch to maintain the old learning capability for the old classes. Multitask learning is adopted to jointly optimize the losses from two branches. Experiments on two widely used remote sensing data sets show our promising performance compared with state-of-the-art incremental object detection methods.

Hierarchical multi-label classification (HMC) aims to assign multiple labels to every instance with the labels organized under hierarchical relations. In the application of ship recognition in remote sensing images, a ship can own coarse-to-fine hierarchical labels, e.g., the military ship, aircraft carrier, and nimitz class aircraft carrier. In this paper, we propose to combine two forms of loss functions to solve the HMC problem based on the neural network. The first probabilistic classification loss is to encode the hierarchical knowledge by introducing hierarchy and exclusion (HEX) graphs to impose constraints on hierarchical labels. The second cross-entropy loss imposes the softmax normalization on leaf nodes in the hierarchy to discriminate fine-grained classes. We evaluate our method on the high resolution satellite image dataset for ship recognition (HRSC), in which hierarchical labels are organized as the three-level tree. The proposed method shows comparative results compared to state-of-art HMC models.

We present a novel stacked autoencoder framework for feature extraction to improve classification of hyperspectral image, leveraging graph regularization to address the shortcomings of classical autoencoder that mainly focuses on learning spectral features. In the proposed method, we firstly construct a graph to represent the spectral-spatial similarity between pixels in a hyperspectral image by measuring their spatial and spectral distances. And then the graph regularized autoencoder is learned to transform the original spectral signatures of pixels into a new feature space used for the downstream pixel classification or other tasks. Our feature extraction method can preserve the intrinsic spectral-spatial distribution in a hyperspectral image and obtain more discriminative and robust features. The experiments on pixel classification show the competitive performance compared with classical autoencoder based and manifold learning based feature extraction approaches.

Multi-label remote sensing image classification is a significant yet difficult task due to intra-class variations and label dependencies among land-cover classes. In this paper, we propose a novel multi -label classification model based on deformable convolutions and graph neural networks. Specifically, we first use deformable convolutions to learn image features with geometric transformation invariance and adaptive receptive field. Then we adopt attention mechanism to extract label-related image features. After that, a directed graph is constructed to model the label dependencies, and the label-related features are fused through graph propagation mechanisms. Experiments on UC-Merced and DOTA data sets demonstrate its effectiveness.

High-dimensional spectral feature and limited training samples have caused a range of difficulties for hyperspectral image (HSI) classification. Feature extraction is effective to tackle this problem. Specifically, tensor factorization is superior to some prominent methods such as principle component analysis (PCA) and non-negative matrix factorization (NMF) because it takes spatial information into consideration. Recently, deep learning has gotten more and more attention for efficiently extracting hierarchical features for various tasks. In this paper, we propose a novel feature extraction method, deep tensor factorization (DTF), to extract hierarchical and meaningful features from observed HSI. This method takes advantage of tensor in representing HSI and the merits of convolutional neural network (CNN) in hierarchical feature extraction. Specifically, a convolution operation is firstly applied in the spectral dimension of HSI to suppress the effect of noise. Then, the convolved HSI is fed into tensor factorization to learn a low rank representation of data. After that, the above two process are repeated to learn a hierarchical representation of HSI. Experimental results on two real hyperspectral data sets show the superiority of the proposed method.

Hyperspectral unmixing aims at decomposing a hyperspectral image (HSI) into a number of constituted materials and associated proportions. Recently, nonnegative tensor factorization (NTF) based methods have been proved effective and natural for hyperspectral unmixing owing to their virtue of representing an HSI without any information loss. However, these methods take an HSI as a whole, partly ignoring the local information in distinct local regions. In addition, HSIs are high likely to be disturbed by various noise, making the global information unnecessarily reliable. To alleviate these drawbacks, we propose a superpixel-based matrix-vector nonnegative tensor factorization (S-MV-NTF) method for hyperspectral unmixing, where both the global information and local information are taken into consideration. In this method, the HSI is firstly partitioned into numerous superpixels, homogeneous regions with adaptive sizes and compact boundaries, representing the local spatial structure information. Then, such local information is integrated to the tensor factorization to make the pixels lying in the same superpixel share similar abundances. Experimental results on synthetic data and real-world data show that the proposed method dominates the state-of-the-art methods.

Small object detection has drawn increasing interest in computer vision and remote sensing image processing. The Region Proposal Network (RPN) methods (e.g., Faster R-CNN) have obtained promising detection accuracy with several hundred proposals. However, due to the pooling layers in the network structure of the deep model, precise localization of small-size object is still a hard problem. In this paper, we design a network with a deconvolution layer after the last convolution layer of base network for small target detection. We call our model Deconv R-CNN. In the experiment on a remote sensing image dataset, Deconv R-CNN reaches a much higher mean average precision (mAP) than Faster R-CNN.