On account of the extreme settings, stealing the black-box model without its training data is difficult in practice. On this topic, along the lines of data diversity, this paper substantially makes the following improvements based on our conference version (dubbed STDatav1, short for Surrogate Training Data). First, to mitigate the undesirable impacts of the potential mode collapse while training the generator, we propose the joint-data optimization scheme, which utilizes both the synthesized data and the proxy data to optimize the surrogate model. Second, we propose the self-conditional data synthesis framework, an interesting effort that builds the pseudo-class mapping framework via grouping class information extraction to hold the class-specific constraints while holding the diversity. Within this new framework, we inherit and integrate the class-specific constraints of STDatav1 and design a dual cross-entropy loss to fit this new framework. Finally, to facilitate comprehensive evaluations, we perform experiments on four commonly adopted datasets, and a total of eight kinds of models are employed. These assessments witness the considerable performance gains compared to our early work and demonstrate the competitive ability and promising potential of our approach.

Fine-grained object detection (FGOD) in remote sensing images is an emerging and challenging task in the field of image intelligent interpretation. It aims to localize objects while classifying them into different fine-grained categories. Modern FGOD methods are mainly derived from well-developed detectors and have made compelling progress. Despite this, these methods struggle to perform well in classifying objects at the subordinate level due to the limitations of their representation manners. In this paper, we propose a network capable of learning discriminative representation (DR) for fine-grained object detection in remote sensing images, named DRNet. First, a fine-grained branch that works in parallel with other task branches is introduced, where objects’ features are re-encoded with dual refinement to generate discriminative representation, enabling accurate fine-grained classification. Second, we design a confusion-minimized loss that automatically scales loss contributions according to the separability of samples to train the fine-grained branch, further boosting discriminative ability of the representation and better addressing hard-to-distinguish objects. Moreover, we devise an interaction verification strategy that empowers the network to fully utilize the results of fine-grained classification and coarse classification for achieving robust inference. On large-scale FAIR1M-1.0 and FAIR1M-2.0 datasets, our DRNet with ResNet50 and$1\times $training schedule obtains 40.87% mAP and 47.04% mAP, respectively, establishing new state-of-the-arts for fine-grained object detection in remote sensing images. The source code is available athttps://github.com//54wb//DRNet.

In object detection, particularly within remote sensing images, the quality of selected samples is crucial for the accuracy and robustness of detection models. However, current sampling strategies demonstrate inherent limitations. They empirically define positive sample sets using fixed thresholds or preset areas, ignoring the actual shapes of the objects and failing to distinguish the intrinsic value of each sample point. To address these critical issues, this article proposes a novel centric probability-based sample selection approach that includes centering probability mapping (CPM), Expectation-Maximization-based boundary optimization (EBO), and probabilistic random sampling (PRS) technologies. Specifically, the CPM is constructed to assign various confidence levels for all sample points based on their proximity to the center of bounding box, effectively discerning the value of individual samples. Then, the EBO is utilized to dynamically optimize the boundaries for positive and negative samples based on the EM algorithm, thus avoiding the sample imbalance problem associated with empirical thresholds. Finally, the PRS strategy is proposed to select training samples from the sample space constructed by CPM and EBO in a manner of random probability sampling, which could improve the diversity of samples while guaranteeing their quality. Experimental validation on three remote sensing image datasets, including DOTA-v1.0, DOTA-v2.0, and DIOR-R, demonstrates that our method achieves robust performance improvements over baseline and significantly surpasses the advanced sample selection methods. The source code will be available athttps://github.com/yanqingyao1994/CPSS.

Few-shot object detection (FSOD) in remote sensing images is a marginally explored but highly challenging task that focuses on identifying unseen classes of objects with a limited number of annotations. Current FSOD approaches often fail to accurately localize the foreground and misalign targets with various orientations, resulting in poor detection performance. For this purpose, we develop a fresh and powerful meta-learning framework based on the idea of imprinting, which leverages tailored support information to model the regional correlation between query and support objects in different stages. Specifically, a global-integrated scheme is first proposed to guide the generation of high-quality proposals by increasing the activation of foreground features and integrating global support information. Considering the orientation discrepancy of objects in query and support sets, we introduce a drift-rectified technique to achieve adaptive alignment by implicitly capturing the positional correspondence between the instances in two sets. In stark contrast to conventional FSOD approaches, our method can extract key clues and establish directional relationships between objects from different training sets, leading to better generalization capability. Extensive experiments on two standard benchmarks (DIOR and NWPU VHR-10.V2) manifest the effectiveness, and our proposed method exhibits superior performance to other competitors with similar motivation. The source code is available athttps://github.com/Ybowei/GIDR

Within remote sensing images, complex atmospheric environments commonly bring about distinct variations in imaging visibility and ambient occlusions, significantly transforming the appearance of objects. Nevertheless, modern detectors generally struggle to maintain promising accuracy when encountering realistic scenarios. Devoting to alleviating the issues, we develop a noise incentive robust network (NIRNet) for remote sensing object detection under cloud corruption without relying on hazy images for training. The proposed NIRNet preserves discriminative representations and calibrates them using an incentive mechanism. Firstly, we design a noise perception module (NPM) to deal with diverse cloud corruption types, which generates point-wise calibration weights dependent on the perceived discrepancy between objects and environmental noise. Secondly, aiming to detect difficult-to-discern objects thoroughly, a dual-path incentive calibration (DPIC) strategy is proposed to combine intensity and stability features weighted by NPM. Profiting from its universal design, the DPIC could be treated as a plug-and-play module for existing detectors, enhancing robustness against adverse weather. To evaluate the reliability of aerial detectors under intricate cloud corruptions, we present an elaborate Hazy-DIOR dataset, which contains numerous images with different cloud conditions and severity levels. Finally, extensive experiments on the Hazy-DIOR and DOTA-Cloud datasets simultaneously demonstrate the robustness of NIRNet, which especially achieves state-of-the-art accuracy and gets 2.16% mAP and 2.56% rPC improvements on the Hazy-DIOR compared to solid Oriented R-CNN detector. The code is available at https://github.com/zhangpeng2001/nirnet.

In recent years, few-shot object detection (FSOD) in remote sensing images has attracted increasing attention. Numerous studies address the challenges posed by both intra-class and inter-class variance through strategies such as augmenting sample diversity and incorporating multi-scale features. However, these features still encompass a considerable amount of noise attributes due to the complex characteristic of satellite images, persistently and adversely affecting classification. In contrast, we advocate for the belief that a limited yet refined set of features surpasses a multitude of coarse features. Accordingly, we tackle above issues through the meticulous refinement of representative category features, enhancing performance by eliminating irrelevant attributes that interfere with classification. Specifically, two pivotal modules: retentive compensation module (RCM) and personality filtering module (PFM), are introduced. The former module RCM systematically scrutinizes features proximate to the category center, yielding prototypes that exhibit both intra-class compactness and inter-class distinctiveness. Furthermore, the latter module PFM utilizes previous obtained prototypes to supervise the filtering process, diminishing the intra-class variance by excluding personality features which could impede the classification task. The integration of the above two modules enables a holistic feature representation, capturing inherent similarities within individual classes while accentuating distinctions between classes. Experiments have been conducted on the DIOR and NWPU VHR-10.v2 datasets, and the results demonstrate that our proposed approach exceeds several state-of-the-art methods. Code is available at https://github.com/yomik-js/RP-FSOD.

Recently, Few-Shot Object Detection (FSOD) has received considerable research attention as a strategy for reducing reliance on extensively labeled bounding boxes. However, current approaches encounter significant challenges due to the intrinsic issue of incomplete annotation while building the instance-level training benchmark. In such cases, the instances with missing annotations are regarded as background, resulting in erroneous training gradients back-propagated through the detector, thereby compromising the detection performance. To mitigate this challenge, we introduce a simple and highly efficient method that can be plugged into both meta-learning-based and transfer-learning-based methods. Our method incorporates two innovative components: Confusing Proposals Separation (CPS) and Affinity-Driven Gradient Relaxation (ADGR). Specifically, CPS effectively isolates confusing negatives while ensuring the contribution of hard negatives during model fine-tuning; ADGR then adjusts their gradients based on the affinity to different category prototypes. As a result, false-negative samples are assigned lower weights than other negatives, alleviating their harmful impacts on the few-shot detector without the requirement of additional learnable parameters. Extensive experiments conducted on the PASCAL VOC and MS-COCO datasets consistently demonstrate that our method significantly outperforms both the baseline and recent FSOD methods. Furthermore, its versatility and efficiency suggest the potential to become a stronger new baseline in the field of FSOD. Code is available at https://github.com/Ybowei/UNP.

Object detection in remote sensing images has garnered significant attention due to its wide applications in real-world scenarios. However, most existing oriented object detectors still suffer from complex backgrounds and varying angles, limiting their performance to further improvement. In this paper, we propose a novel oriented detector withHierarchical mask prompting andRobust integrated regression, termed HRDet. Specifically, to cope with the first issue, we construct a hierarchical mask prompting module consisting of a semantic mask prediction branch and hierarchical Softmax technique. The former aims to isolate object instances from cluttered interferences guided by coarse box-wise masks, while the latter propagates differentiated features for adjacent layers using hierarchical attentive weights. To deal with the second issue, we strive for robust integrated regression and formulate an efficient oriented IoU loss, explicitly measuring the discrepancies of three geometric factors in oriented regression, i.e., the central point distance, side length, and angle. This innovative loss intends to overcome the problem that existing IoU-based losses are invariant during the regression of varying angles. We applied these two strategies to a simple one-stage detection pipeline, achieving a new level of trade-off between speed and accuracy. Extensive experiments on four large aerial imagery datasets, DOTA-v1.0, DOTA-v2.0, DIOR-R, and HRSC2016, demonstrate that our HRDet significantly improves the accuracy of the one-stage detector over refine-stage counterparts while maintaining the efficiency advantage. The source code will be available athttps://github.com/yanqingyao1994/HRDet.

Oriented object detection in aerial images has made significant advancements propelled by well-developed detection frameworks and diverse representation approaches to oriented bounding boxes. However, within modern oriented object detectors, the insufficient consideration given to certain factors, like contextual priors in aerial images and the sensitivity of the angle regression, hinder further improvement of detection performance. In this paper, we propose a dual-focused detector (DFDet), which simultaneously focuses on the exploration of contextual knowledge and the mitigation of angle sensitivity. Specifically, DFDet contains two novel designs: a contextual dependence mining network (CDMN) and a penalty-incentive allocation strategy (PIAS). CDMN constructs multiple features containing contexts across various ranges with low computational burden, and aggregates them into a compact yet informative representation that empowers the model for robust inference. PIAS dynamically calibrates the angle regression loss with a scalable penalty term determined by the angle regression sensitivity, incentivizing model to boost regression capacity for large aspect ratio objects challenging to be localized accurately. Extensive experiments on four widely-used benchmarks demonstrate the effectiveness of our approach, and new state-of-the-arts for one-stage object detection in aerial images are established. Without bells and whistles, DFDet with ResNet50 achieves 74.71% mAP running at 23.4 FPS on the most widely-used DOTA-v1.0 dataset. The source code is available at https://github.com/DDGRCF/DFDet.

Conventional deep CNN-based segmentation approaches have achieved satisfactory performance in recent years, however, they are essentially big data-driven technologies and are difficult to generalize to unseen categories. Few-shot segmentation is subsequently developed to perform pertinent operations in a low-data regime. Unfortunately, due to the training paradigm and network architecture factors, existing methods are prone to overfit the targets of base categories and yield inaccurate segmentation boundaries, which impedes the research progress to some extent. In this paper, we propose a Holistic Prototype Activation (HPA) network to alleviate these problems. Its novel designs can be summarized in three aspects: 1) A training-free scheme to derive the prior representations of base categories. 2) Prototype Activation Module (PAM) that generates reliable activation maps and well-matched query features by filtering the objects of irrelevant classes with high confidence. 3) Cross-Referenced Decoder (CRD) for interacted feature reweighting and multi-level feature aggregation. Extensive experiments on standard few-shot segmentation benchmarks (PASCAL-5$^{i}$and COCO-20$^{i}$) verify the effectiveness of our method. On top of that, the superior performance on multiple extended tasks, such as weak-label segmentation, zero-shot segmentation, and video object segmentation, also illustrates its flexibility and versatility. Our code is publicly available athttps://github.com/chunbolang/HPA.

Despite the progress made by few-shot segmentation (FSS) in low-data regimes, the generalization capability of most previous works could be fragile when countering hard query samples with seen-class objects. This paper proposes a fresh and powerful scheme to tackle such an intractable bias problem, dubbed base and meta (BAM). Concretely, we apply an auxiliary branch (base learner) to the conventional FSS framework (meta learner) to explicitly identify base-class objects, i.e., the regions that do not need to be segmented. Then, the coarse results output by these two learners in parallel are adaptively integrated to derive accurate segmentation predictions. Considering the sensitivity of meta learner, we further introduce adjustment factors to estimate the scene differences between support and query image pairs from both style and appearance perspectives, so as to facilitate the model ensemble forecasting. The remarkable performance gains on standard benchmarks (PASCAL-5$^{i}$, COCO-20$^{i}$, and FSS-1000) manifest the effectiveness, and surprisingly, our versatile scheme sets new state-of-the-arts even with two plain learners. Furthermore, in light of its unique nature, we also discuss several more practical but challenging extensions, including generalized FSS, 3D point cloud FSS, class-agnostic FSS, cross-domain FSS, weak-label FSS, and zero-shot segmentation. Our source code is available athttps://github.com/chunbolang/BAM.

Object detection is a fundamental yet challenging task in computer vision. Despite the great strides made over recent years, modern detectors may still produce unsatisfactory performance due to certain factors, such as non-universal object features and single regression manner. In this paper, we draw on the idea of mutual-assistance (MA) learning and accordingly propose a robust one-stage detector, referred as MADet, to address these weaknesses. First, the spirit of MA is manifested in the head design of the detector. Decoupled classification and regression features are reintegrated to provide shared offsets, avoiding inconsistency between feature-prediction pairs induced by zero or erroneous offsets. Second, the spirit of MA is captured in the optimization paradigm of the detector. Both anchor-based and anchor-free regression fashions are utilized jointly to boost the capability to retrieve objects with various characteristics, especially for large aspect ratios, occlusion from similar-sized objects, etc. Furthermore, we meticulously devise a quality assessment mechanism to facilitate adaptive sample selection and loss term reweighting. Extensive experiments on standard benchmarks verify the effectiveness of our approach. On MS-COCO, MADet achieves 42.5% AP with vanilla ResNet50 backbone, dramatically surpassing multiple strong baselines and setting a new state of the art.

Few-shot segmentation (FSS), which aims to determine specific objects in the query image given only a handful of densely labeled samples, has received extensive academic attention in recent years. However, most existing FSS methods are designed for natural images, and few works have been done to investigate more realistic and challenging applications,e.g., remote sensing image understanding. In such a setup, the complex nature of the raw images would undoubtedly further increase the difficulty of the segmentation task. To couple with potential inference failures, we propose a novel and powerful remote sensing FSS framework with global Rectification and decoupled Registration, termed R2Net. Specifically, a series of dynamically updated global prototypes are utilized to provide auxiliary non-target segmentation cues and to prevent inaccurate prototype activation resulting from the variability between query-support image pairs. The foreground and background information flows are then decoupled for more targeted and tailored object localization, avoiding unnecessary confusion from information redundancy. Furthermore, we impose additional constraints to promote the interclass separability and intraclass compactness. Extensive experiments on the standard benchmark iSAID-5idemonstrate the superiority of the proposed R2Net over state-of-the-art FSS models. The code will be made available.

In recent years, few-shot segmentation (FSS) has received widespread attention from scholars by virtue of its superiority in low-data regimes. Most existing research focuses on natural image processing, and very few studies are dedicated to the practical but challenging topic of remote sensing image understanding. Related experimental results show that directly transferring the previously proposed framework to the current domain is prone to produce unsatisfactory results withincomplete objectsandirrelevant distractors. Such phenomena can be attributed to the lack of modules specifically designed for the complex characteristics of remote sensing images,e.g., great intra-class diversity and low target-background contrast. In this paper, we propose a conceptually simple and easy-to-implement framework to tackle the aforementioned problems. Specifically, our innovative design embodies two main aspects: i) the support mask is progressively parsed into multiple valuable sub-regions that can be further exploited to compute local descriptors with segmentation cues about intractable parts; ii) the base-class memories stored in the meta-training phase are replayed and leveraged for the distillation of novel-class prototypes, where the commonalities between classes are adequately explored, more in line with the concept oflearning to learn. These two components, i.e., the progressive parsing module and commonality distillation module, contribute to each other and together constitute the proposed PCNet. We conduct extensive experiments on the standard benchmark to evaluate segmentation performance in few-shot settings. Quantitative and qualitative results illustrate that our PCNet distinctly outperforms previous FSS approaches and sets a new state-of-the-art.

Benefiting from advances in few-shot learning techniques, their application to dense prediction tasks (e.g., segmentation) has also made great strides in the past few years. However, most existing few-shot segmentation (FSS) approaches follow a similar pipeline to that of few-shot classification, where some core components are directly exploited regardless of various properties between tasks. We note that such an ill-conceived framework introduces unnecessary information loss, which is clearly unacceptable given the already very limited training sample. To this end, we delve into the typical types of information loss and provide a reasonably effective way, namely Retain And REcover (RARE). The main focus of this paper can be summarized as follows: (i) the loss of spatial information due to global pooling; (ii) the loss of boundary information due to mask interpolation; (iii) the degradation of representational power due to sample averaging. Accordingly, we propose a series of strategies to retain/recover the avoidable/unavoidable information, such as unidirectional pooling, error-prone region focusing, and adaptive integration. Extensive experiments on two popular benchmarks (i.e., PASCAL-5iand COCO-20i) demonstrate the effectiveness of our scheme, which is not restricted to a particular baseline approach. The ultimate goal of our work is to address different information loss problems within a unified framework, and it also exhibits superior performance compared to other methods with similar motivations. The source code will be made available at https://github.com/chunbolang/RARE.

Recently few-shot segmentation (FSS) has been extensively developed. Most previous works strive to achieve generalization through the meta-learning framework derived from classification tasks; however, the trained models are biased towards the seen classes instead of being ideally class-agnostic, thus hindering the recognition of new concepts. This paper proposes a fresh and straightforward insight to alleviate the problem. Specifically, we apply an additional branch (base learner) to the conventional FSS model (meta learner) to explicitly identify the targets of base classes, i.e., the regions that do not need to be segmented. Then, the coarse results output by these two learners in parallel are adaptively integrated to yield precise segmentation prediction. Considering the sensitivity of meta learner, we further introduce an adjustment factor to estimate the scene differences between the input image pairs for facilitating the model ensemble forecasting. The substantial performance gains on PASCAL-5iand COCO-20iverify the effectiveness, and surprisingly, our versatile scheme sets a new state-of-the-art even with two plain learners. Moreover, in light of the unique nature of the proposed approach, we also extend it to a more realistic but challenging setting, i.e., generalized FSS, where the pixels of both base and novel classes are required to be determined. The source code is available at github.com/chunbolang/BAM.

Few-shot segmentation, which aims to segment unseen-class objects given only a handful of densely labeled samples, has received widespread attention from the community. Existing approaches typically follow the prototype learning paradigm to perform meta-inference, which fails to fully exploit the underlying information from support image-mask pairs, resulting in various segmentation failures, e.g., incomplete objects, ambiguous boundaries, and distractor activation. To this end, we propose a simple yet versatile framework in the spirit of divide-and-conquer. Specifically, a novel self-reasoning scheme is first implemented on the annotated support image, and then the coarse segmentation mask is divided into multiple regions with different properties. Leveraging effective masked average pooling operations, a series of support-induced proxies are thus derived, each playing a specific role in conquering the above challenges. Moreover, we devise a unique parallel decoder structure that integrates proxies with similar attributes to boost the discrimination power. Our proposed approach, named divide-and-conquer proxies (DCP), allows for the development of appropriate and reliable information as a guide at the “episode” level, not just about the object cues themselves. Extensive experiments on PASCAL-5i and COCO-20i demonstrate the superiority of DCP over conventional prototype-based approaches (up to 5~10% on average), which also establishes a new state-of-the-art. Code is available at github.com/chunbolang/DCP.

Few-shot image classification has attracted extensive attention, which aims to recognize unseen classes given only a few labeled samples. Due to the large intraclass variances and interclass similarity of remote sensing scenes, the task under such circumstance is much more challenging than general few-shot image classification. Most existing prototype-based few-shot algorithms usually calculate prototypes directly from support samples and ignore the validity of prototypes, which results in a decline in the accuracy of subsequent inferences based on prototypes. To tackle this problem, we propose a Siamese-prototype network (SPNet) with prototype self-calibration (SC) and intercalibration (IC). First, to acquire more accurate prototypes, we utilize the supervision information from support labels to calibrate the prototypes generated from support features. This process is called SC. Second, we propose to consider the confidence scores of the query samples as another type of prototypes, which are then used to predict the support samples in the same way. Thus, the information interaction between support and query samples is implicitly a further calibration for prototypes (so-called IC). Our model is optimized with three losses, of which two additional losses help the model to learn more representative prototypes and make more accurate predictions. With no additional parameters to be learned, our model is very lightweight and convenient to employ. The experiments on three public remote sensing image datasets demonstrate competitive performance compared with other advanced few-shot image classification approaches. The source code is available athttps://github.com/zoraup/SPNet.

Oriented object detection is a practical and challenging task in remote sensing image interpretation. Nowadays, oriented detectors mostly use horizontal boxes as intermedium to derive oriented boxes from them. However, the horizontal boxes are inclined to get small Intersection-over-Unions (IoUs) with ground truths, which may have some undesirable effects, such as introducing redundant noise, mismatching with ground truths, detracting from the robustness of detectors, etc. In this paper, we propose a novel Anchor-free Oriented Proposal Generator (AOPG) that abandons horizontal box-related operations from the network architecture. AOPG first produces coarse oriented boxes by a Coarse Location Module (CLM) in an anchor-free manner and then refines them into high-quality oriented proposals. After AOPG, we apply a Fast R-CNN head to produce the final detection results. Furthermore, the shortage of large-scale datasets is also a hindrance to the development of oriented object detection. To alleviate the data insufficiency, we release a new dataset on the basis of our DIOR dataset and name it DIOR-R. Massive experiments demonstrate the effectiveness of AOPG. Particularly, without bells and whistles, we achieve the accuracy of 64.41%, 75.24% and 96.22% mAP on the DIOR-R, DOTA and HRSC2016 datasets respectively. Code and models are available at https://github.com/jbwang1997/AOPG.