Generative models are widely used to produce synthetic images with annotations, alleviating the burden of image collection and annotation for training deep visual models. However, challenges such as limited image diversity, noisy pseudo labels, and domain gaps between synthetic and real images often undermine their effectiveness in downstream visual tasks. This paper introduces the Iterative Self-Training with Class-Aware Text-to-Image Synthesis (IST-CATS) framework, which addresses these challenges by integrating a class-aware text-to-image synthesis (CATS) component with an iterative self-training (IST) strategy. CATS innovatively introduces a class-aware chain approach to generate detailed descriptions. These descriptions act as prompts for a diffusion model, enabling the creation of a diverse of images accompanied by distinguishable objects against the background. The generated images can be easily pseudo-labeled by an unsupervised instance segmentation method, and then noisy pseudo labels can be effectively purified by a novel feature similarity-based filtering mechanism. The generated images underpin our IST, which progressively enhances vision models and refines pseudo labels through self-training and our proposed label filtering strategy (LabFilt). LabFilt meticulously improves the quality of pseudo labels by employing class-adaptive techniques at both the pixel and object levels, ensuring refined pseudo-label accuracy. IST-CATS demonstrates superior performance in object detection and semantic segmentation compared to traditional synthetic and semi/weakly-supervised methods, effectively addressing data collection and annotation challenges.

Deep learning-based object detection models rely heavily on large-scale and precise annotations for training. However, manually annotating bounding-box annotations for such data is both time-consuming and costly, especially when dealing with high-resolution satellite imagery containing densely packed small-sized objects. To alleviate the burden of manual annotation, we propose a simple yet effective approach, called progressive self-training object detection (PSTDet), to enable accurate object detection in remote sensing imagery without relying on manual annotations. Our PSTDet framework consists of two main components: initial pseudo label generation (IPLG) and progressive self-training with relabeling (PST-R). In IPLG, we leverage unsupervised image clustering, unsupervised instance detection, and geometric constraints to automatically generate high-quality bounding-box annotations for the initial training dataset. This innovative approach significantly reduces the time and expense associated with data annotation, laying a solid foundation for the subsequent progressive self-training stage. The annotations produced by IPLG serve as the training data for PST-R, which enhances the detector and pseudo labels through progressive self-training and our proposed noisy pseudo label filtering strategy (NPLFilter). Our NPLFilter purifies the quality of pseudo labels by integrating geometric constraints, prior knowledge, and category-adaptive thresholds. Experimental results demonstrate that our method achieves significant performance improvement on challenging NWPU VHR-10.v2 and DIOR datasets. Notably, our method far outperforms state-of-the-art weakly supervised methods and compares favorably with fully supervised methods.

The performance of deep networks for object detection in remote sensing images (RSIs) largely depends on the availability of large-scale training images whose labels are given at the bounding-box level through a labor-intensive manual labeling process. To alleviate the huge burden of providing bounding-box annotations manually for movable objects, we propose a new approach, called dynamic automatic learning (DAL), to progressively learn object detectors. Specifically, a novel initial annotation generation (IAG) strategy is first designed to produce bounding-box annotations for movable objects in multi-temporal remote sensing images. During this process, image-level labels need to be manually labeled for the generated candidates. Next, a detection network learns the detection knowledge from multi-temporal remote sensing images with bounding-box annotations and then transfers the knowledge to generate pseudo boxes for the unlabeled data. Finally, with these pseudo boxes, the object detector can be optimized for generating accurate pseudo boxes iteratively. Furthermore, we introduce a pseudo box filtering (PBF) strategy to purify the quality of pseudo boxes to obtain accurate supervision. Our experiments on the challenging NWPU VHR-10.v2 and DIOR datasets have demonstrated that our DAL approach can achieve competitive results compared to state-of-the-art methods.

Cloud detection is one of the key technologies in the field of remote sensing. Although extensive deep learning-based cloud detection methods achieve good performance, their detection results in confusing areas such as cloud boundaries and thin clouds are often not satisfactory due to the potential inter-class similarity and intra-class inconsistency of objects. To this end, we propose a Boundary-Aware Bilateral Fusion network (BABFNet), which effectively enhances cloud detection in confusing areas by introducing a boundary prediction branch as an auxiliary. To avoid the loss of details, the boundary prediction branch is designed to run at full resolution with a shallow architecture, while some Semantic Enhancement Modules (SEMs) are used to supplement high-level semantic information by introducing multi-level encoder features of the cloud detection branch. This feature sharing in turn drives the cloud detection branch to focus more on cloud boundaries during training. At the end of the network, a Bilateral Fusion Module (BFM) is added for information complementarity between features from these two branches. The features from the cloud detection branch provide multi-scale features to the boundary prediction branch for more accurate boundary prediction, while the features from the boundary prediction branch further serve as prior knowledge to help the cloud detection branch aggregate contextual information. To verify the effectiveness of the proposed method, we select four different networks as cloud detection branches and conduct comparative experiments on two public datasets, GF-1 WFV and MODIS. The experimental results show that the proposed method significantly enhances cloud detection in confusing areas.

Deep learning has achieved promising performance on semantic segmentation, but few works focus on semantic segmentation at the fine-grained level. Fine-grained semantic segmentation requires recognizing and distinguishing hundreds of sub-categories. Due to the high similarity of different sub-categories and large variations in poses, scales, rotations, and color of the same sub-category in the fine-grained image set, the performance of traditional semantic segmentation methods will decline sharply. To alleviate these dilemmas, a new approach, named Class Guided Channel Weighting Network (CGCWNet), is developed in this paper to enable fine-grained semantic segmentation. For the large intra-class variations, we propose a Class Guided Weighting (CGW) module, which learns the image-level fine-grained category probabilities by exploiting second-order feature statistics, and use them as global information to guide semantic segmentation. For the high similarity between different sub-categories, we specially build a Channel Relationship Attention (CRA) module to amplify the distinction of features. Furthermore, a Detail Enhanced Guided Filter (DEGF) module is proposed to refine the boundaries of object masks by using an edge contour cue extracted from the enhanced original image. Experimental results on PASCAL VOC 2012 and six fine-grained image sets show that our proposed CGCWNet has achieved state-of-the-art results.

In recent years, a large number of convolutional neural networks-based cloud detection algorithms have been proposed for remote sensing image preprocessing and most of them have an encoder-decoder structure. However, downsampling and upsampling operations, as the basic components of these methods, inevitably lead to the loss of detailed information in high-level features, which affects cloud detection performance. At the same time, the physical characteristics of the cloud, such as the variable size and irregular structure, also put forward requirements for the multi-scale feature representation ability of the network. To this end, we propose a novel cloud detection network named DMNet, which contains a Dense Feature Enhancement Module (DFEM) and a Multi-scale Context Fusion Spatial Attention Module (MCFSAM). DFEM aims to achieve information complementarity by exploiting the different properties of the features at different levels of the encoder, so as to strengthen the detailed information of high-level features and make the low-level features have more semantics. MCFSAM introduces a Multi-scale Context Fusion Block (MCFB) in spatial attention, which enables the network to densely capture contextual information at different scales and further emphasize useful features in the spatial dimension. Extensive experiments on GF-1 wide field-of-view satellite imagery (GF-1 WFV) dataset and Moderate-Resolution Imaging Spectroradiometer (MODIS) dataset demonstrate that our method outperforms other state-of-the-art cloud detection algorithms.

The existing publicly available datasets with pixel-level labels contain limited categories, and it is difficult to generalize to the real world containing thousands of categories. In this paper, we propose an approach to generate object masks with detailed pixel-level structures/boundaries automatically to enable semantic image segmentation of thousands of targets in the real world without manually labelling. A Guided Filter Network (GFN) is first developed to learn the segmentation knowledge from an existed dataset, and such GFN then transfers the learned segmentation knowledge to generate initial coarse object masks for the target images. These coarse object masks are treated as pseudo labels to self-optimize the GFN iteratively in the target images. Our experiments on six image sets have demonstrated that our proposed approach can generate object masks with detailed pixel-level structures/boundaries, whose quality is comparable to the manually-labelled ones. Our proposed approach also achieves better performance on semantic image segmentation than most existing weakly-supervised, semi-supervised, and domain adaptation approaches under the same experimental conditions.

Abstract Image classification is one of the mainstream tasks of computer vision. However, the most existing methods use labels of the same granularity level for training. This leads to ignoring the hierarchy that may help to differentiate different visual objects better. Embedding hierarchical information into the convolutional neural networks (CNNs) can effectively regulate the semantic space and thus reduce the ambiguity of prediction. To this end, a multi‐task learning framework, named as Hierarchical Bilinear Convolutional Neural Network (HB‐CNN), is developed by seamlessly integrating CNNs with multi‐task learning over the hierarchical visual concept structures. Specifically, the labels with a tree structure are used as the supervision to hierarchically train multiple branch networks. In this way, the model can not only learn additional information (e.g. context information) as the coarse‐level category features, but also focus the learned fine‐level category features on the object properties. To smoothly pass hierarchical conceptual information and encourage feature reuse, a connectivity pattern is proposed to connect features at different levels. Furthermore, a bilinear module is embedded to generalise various orderless texture feature descriptors so that our model can capture more discriminative features. The proposed method is extensively evaluated on the CIFAR‐10, CIFAR‐100, and ‘Orchid’ Plant image sets. The experimental results show the effectiveness and superiority of our method.

Abstract Fine‐grained image classification is a challenging topic in the field of computer vision. General models based on first‐order local features cannot achieve acceptable performance because the features are not so efficient in capturing fine‐grained difference. A bilinear convolutional neural network (CNN) model exhibits that a second‐order statistical feature is more efficient in capturing fine‐grained difference than a first‐order local feature. However, this framework only considers the extraction of a second‐order feature descriptor, using a single convolutional layer. The potential effective classification features of other convolutional layers are ignored, resulting in loss of recognition accuracy. In this paper, a multilayer feature descriptors fusion CNN model is proposed. It fully considers the second‐order feature descriptors and the first‐order local feature descriptor generated by different layers. Experimental verification was carried out on fine‐grained classification benchmark data sets, CUB‐200‐2011, Stanford Cars, and FGVC‐aircraft. Compared with the bilinear CNN model, the proposed method has improved accuracy by 0.8%, 1.1%, and 5.5%. Compared with the compact bilinear pooling model, there is an accuracy increase of 0.64%, 1.63%, and 1.45%, respectively. In addition, the proposed model effectively uses multiple 1×1 convolution kernels to reduce dimension. The experimental results show that the multilayer low‐dimensional second‐order feature descriptors fusion model has comparable recognition accuracy of the original model.