Unsupervised Semantic Segmentation (USS) aims to learn semantically rich and dense representations without relying on labels. Recent advances in self-supervised learning have demonstrated the potential of pretrained vision transformers to capture patch-level semantic information, offering a promising direction to USS. However, existing methods face challenges in constructing a discriminative spatial token embedding space that consistently and effectively represents the well-structured semantic relationships among object components. Inspired by Edwin Hancock’s pioneer work on hierarchical pattern analysis, we highlight the critical role of hierarchical context to overcome this limitation. By modeling spatial relationships at multiple levels of granularity, hierarchical context helps align related object parts while distinguishing them across semantic groups. Based on this insight, we introduce Hierarchical Context Learning (HCL), a novel approach for USS that enhances semantic consistency by integrating hierarchical context. HCL incorporates a novel parallel multi-level vision transformer backbone to aggregate multi-level contextual information into object component tokens. To uncover the semantic structure of objects, we propose Momentum-based Global Foreground–Background Clustering (MoGoClustering) to cluster object components into coherent semantic groups and then calculate their semantic centroids. To enforce intra-group semantic consistency and maximize inter-group separation across spatial scales, we design a foreground–background-aware contrastive loss based on MoGoClustering. Our method achieves state-of-the-art performance on the COCO-Stuff and Pascal VOC datasets, demonstrating its ability to learn robust, context-aware, and discriminative object component semantics for USS. The code is available at: https://github.com/dbaofd/HCL .

Automatic open-pit mine extraction and change detection from high-resolution remote sensing images are of great importance to mineral resource management. However, the high spatial heterogeneity and spectral variations of mining area scenarios make these tasks challenging. Motivated by the strong correlation between the two tasks and their potential mutual benefits, this article presents a hybrid convolutional neural network (CNN)–Transformer multitask network (CTMNet). Constructed in an encoder-decoder manner, CTMNet has two sperate extraction paths (EPs) to localize the regions of interest for bi-temporal images, along with a change detection path (CDP) to identify discrepancies by differentiating the multiscale feature representations from the EPs. As the basic building block for the EP, a CNN-Transformer hybrid block is designed to enhance the global and local feature representation capacity. To cope with the variations in the bi-temporal images, we propose the feature alignment module for the CDP. A hard sample mining-based contrastive constraint loss is proposed to emphasize the contributions of hard samples to the training process. The experimental results on a collected open-pit mine extraction and change detection dataset (OMECSet) and two public datasets reveal the validity of the CTMNet when compared to the state-of-the-art methods. The OMECSet and the code of CTMNet have been made public available athttps://figshare.com/s/80519cb980ca54456447.

Adversarial attacks pose a significant threat to deep neural networks (DNNs) used for 3-D point cloud classification, especially in safety-critical applications. While previous works have proposed several defense model architectures and adversarial training strategies, they often either fall short in capturing the intricate geometric and topological aspects of point cloud data or grapple with challenges pertaining to model convergence. To solve these problems, in this article, we propose an innovative point cloud defense framework, called MAFFN-SAT, which contains a multiview adaptive feature fusion network (MAFFN) along with a smooth adversarial training (SAT) strategy. Specifically, we construct a multiview defense module to obtain multiview features in MAFFN, which uses geometric proximity and spatial queries to comprehensively explore the inherent characteristics of point cloud data. Subsequently, an adaptive feature fusion module is designed to integrate the multiview features. Furthermore, we introduce SAT, which uses an optimized regularization to measure the information divergence between two probability distributions, guiding the model to develop a smoother decision boundary, thereby more robust to adversarial attacks. Extensive experiments conducted on three benchmark datasets demonstrate the robustness of our approach against various attacks. Remarkably, our defense framework achieves 15.34% performance improvement under point dropping attacks on the ModelNet40 dataset. Our implementation:https://github.com/shenyu234/MAFFN-SAT.

This paper proposes a novel uncertainty-adjusted label transition (UALT) method for weakly supervised solar panel mapping (WS-SPM) in aerial Images. In weakly supervised learning (WSL), the noisy nature of pseudo labels (PLs) often leads to poor model performance. To address this problem, we formulate the task as a label-noise learning problem and build a statistically consistent mapping model by estimating the instance-dependent transition matrix (IDTM). We propose to estimate the IDTM with a parameterized label transition network describing the relationship between the latent clean labels and noisy PLs. A trace regularizer is employed to impose constraints on the form of IDTM for its stability. To further reduce the estimation difficulty of IDTM, we incorporate uncertainty estimation to first improve the accuracy of noisy dataset distillation and then mitigate the negative impacts of falsely distilled examples with an uncertainty-adjusted re-weighting strategy. Extensive experiments and ablation studies on two challenging aerial data sets support the validity of the proposed UALT.

Weakly supervised solar panel mapping has shown its advantages in automatically detecting solar panels from remote sensing images with low annotation costs. Considering the noisy nature of pseudo labels (PLs), which are frequently employed in weakly supervised methods, we propose to introduce uncertainty measure to guide the estimation of noise levels in PLs and develop a novel method based on uncertainty-aware label transition (UALT). The proposed method consists of three parts: uncertainty estimation network, uncertainty-aware label transition network, and target mapping network with forward correction. We first generate heteroscedastic uncertainty by learning an estimator under Bayes formalism. Then, with the uncertainty as guidance, a label transition network is trained to learn the mapping between clean labels, and Bayes optimal labels and predict the instance-dependent transition matrix. Finally, the transition matrix is employed in the forward correction process, where the target mapping network produces clean predictions for solar panels. Comparative experiments with six state-of-the-art weakly supervised methods on an aerial image data set show the superiority of the proposed UALT, especially in mapping accuracy and dis-covering small-scale objects.

Solar panel mapping from high-resolution aerial images is becoming increasingly crucial to grid planning and operation, where weakly supervised approach has been explored. To cope with the noisy nature of pseudo-labels (PLs) generated by weakly supervised object localization, we propose an effective uncertainty-aware forward correction (UA-FC) method to learn clean predictions from the noisy PLs. The proposed method consists of two steps: heteroscedastic uncertainty estimation and forward correction procedure. The purpose of the first step is to produce uncertainty as an indicator for the instance-dependent noise. The second step includes a target mapping network to produce clean predictions and a transition function to model the relationship between clean predictions and noisy PLs. As estimating every probability of one class flipped into another is difficult and time-consuming, we introduce heteroscedastic uncertainty as a measurement and propose an uncertainty-based transformation function to map clean predictions into noisy ones. By minimizing the errors between the noisy predictions and noisy labels, the target mapping network is able to offer clean predictions close to the actual objects. Extensive experiments on an aerial dataset reveal that the proposed method outperforms other state-of-the-art methods by a large margin, especially in recovering the boundary of the objects.

With the rapid development of the solar distribution, solar panel mapping is becoming increasingly valuable to decision-makers. Weakly supervised methods have been developed to reduce the cost in training sample collection, and the most successful ones follow the alternative training scheme, which first generates coarse object localizations as pseudo labels (PLs) and then utilizes these PLs to train an end-to-end network for object extraction. As remote sensing images are typically characterized by multiple occurrences of objects and complicated backgrounds, the alternative training scheme suffers from low mapping accuracy and deficient boundary maintenance due to the varying quality of PLs. In this article, we focus on addressing these problems by adaptively adjusting the contributions of quality-varying PLs and propose a novel self-paced residual aggregated network (SP-RAN) for solar panel mapping. Specifically, with the initial PLs generated by gradient-weighted class activation mapping, a residual aggregated network is designed for target mapping with special consideration for the capability in producing complete and well-shaped mapping results. Considering the inconsistent quality of PLs, an effective confidence-aware (CA) loss is developed to emphasize the contribution of high-quality PLs and alleviate the negative impacts brought by the bad-quality ones in the training phase. Moreover, to concentrate on boundary maintenance, a novel self-paced label correction (SP-LC) strategy is proposed to selectively update PLs by considering their reliability. Extensive experimental comparisons with state-of-the-art methods and ablation study on two aerial datasets and a remote sensing dataset demonstrate the superiority of the proposed method.

With the rapid development of solar distribution, mapping the locations and sizes of solar panels is becoming increasingly valuable. Weakly supervised methods have been proposed to reduce the reliance on expensive pixel-wise annotations by adopting weak labels, but inevitably suffer from low mapping accuracy and poor boundary maintenance. To solve these problems, a novel weakly supervised residual aggregated network (WS-RAN) is presented in this paper. In the WS-RAN, pixel-wise labels are automatically generated from image-level labels by a classification network. Then, the produced pixel-wise labels are used to train the residual aggregated network (RAN) for target mapping, which is designed to cope the variations in size and shapes of individual solar panel layout. Particularly, for better boundary maintenance, the residual aggregated block is developed as the basic module in the bottom-up path of the RAN. Experiment results reveal that the proposed WS-RAN significantly outperforms four state-of-the-art weakly supervised methods with 76.0% F1score and 61.3% IoU score.

Moving Object Detection (MOD) from satellite videos plays one of the most fundamental roles in satellite video surveillance. Training a supervised moving object detector typically requires boundary box annotations for object instances, however, this annotation process is time-consuming for satellite videos. In this paper, we propose a weakly supervised method for sidestepping this process, where the supervised information for detecting moving objects on a frame is instead provided by the unsupervised method based on motion knowledge across a video. We adopt the Extended Low-rank and Structured Sparse Decomposition (E-LSD) approach for generating pixel-wise pseudo labels for moving objects. Then the extracted pseudo labels are used for training a Deep Convolutional Neural Network (DCNN) following the Encoder-Decoder architecture with lateral connections for segmenting moving objects from a new frame. We demonstrate the effectiveness of the proposed method on a satellite video dataset, and, compared with five state-of-the-art MOD methods tested, it achieves both improved detection accuracy and promising frame rate.

In many remote sensing tasks, different types of regions or targets differ in requirements for spectral and spatial quality. The discrepancy reveals that a uniform pansharpening strategy applying to the entire image may not fulfill the varying demands of different regions appropriately. From this aspect, we resort to saliency analysis to distinguish regions with different spatial and spectral requirements and then propose a new saliency cascade convolutional neural network for pansharpening (SC-PNN). SC-PNN is composed of two parts: a dilated deformable convolutional network (DDCN) for saliency analysis and a saliency cascade residual dense network (SC-RDN) for pansharpening. DDCN is a fully convolutional network based on hybrid dilated convolution and deformable convolution, aiming to separate salient regions, such as residential areas from nonsalient areas, including mountains and vegetation areas, with well-defined boundaries and integrity. In the fusion process, SC-RDN is specially designed with the help of saliency analysis. We first construct a deep regression network to estimate a primarily sharpened image and subsequently leverage the saliency map produced by DDCN to develop a saliency enhancement module. In this module, the quality of salient and nonsalient areas is further improved by two independent deep residual dense networks. Thus, a precise fused image can be predicted. Experiments on SPOT5, GeoEye-1, and WorldView-3 data sets reveal that, compared to state-of-the-art pansharpening methods, our proposal has a superior ability to improve the spatial quality and preserve spectral information. The effectiveness of the saliency enhancement module is also validated in the experiment.

The visual characteristics of different regions in remote sensing images are significantly versatile, which poses a huge challenge to single image super-resolution. Although generative adversarial network (GAN) has shown great potential in generating photo-realistic results, it provides unsatisfactory performance in objective metrics owning to pseudo textures brought by adversarial learning. In this paper, we propose a new saliency-driven feedback GAN to cope with these problems. We design a saliency-driven feedback generator based on paired-feedback blocks (PFBBs) and recurrent structure to provide strong reconstruction ability. In the PFBB, the saliency map serves as an indicator to reflect the texture complexity, so different reconstruction principles can be applied to restore areas with varying levels of saliency. Besides, we propose to measure the visual quality of salient areas, non-salient areas, and the whole image with multi-discriminators, which can dramatically eliminate pseudo textures. Comprehensive evaluations and ablation studies validate the superiority of our proposal.

Recently, convolutional neural networks have shown superior performance in single-image superresolution. Although existing mean-square-error-based methods achieve high peak signal-to-noise ratio (PSNR), they tend to generate oversmooth results. Generative adversarial network (GAN)-based methods can provide high-resolution (HR) images with higher perceptual quality, but produce pseudotextures in images, which generally leads to lower PSNR. Besides, different regions in remote sensing images (RSIs) reflect discrepant surface topography and visual characteristics. This means a uniform reconstruction strategy may not be suitable for all targets in RSIs. To solve these problems, we propose a novel saliency-discriminated GAN for RSI superresolution. First, hierarchical weakly supervised saliency analysis is introduced to compute a saliency map, which is subsequently employed to distinguish the diverse demands of regions in the following generator and discriminator part. Different from previous GANs, the proposed residual dense saliency generator takes saliency maps as a supplementary condition in the generator. Simultaneously, combining the characteristic of RSIs, we design a new paired discriminator to enhance the perceptual quality, which measures the distance between generated images and HR images in salient areas and nonsalient areas, respectively. Comprehensive evaluations validate the superiority of the proposed model.

Remote-sensing images (RSIs) generally have strong spatial characteristics for surface features. Various ground objects, such as residential areas, roads, forests, and rivers, differ substantially. According to this visual attention characteristic, regions with complicated texture features require more realistic details to reflect a better description of the topography, while regions such as farmlands should be smooth and have less noise. However, most existing single-image superresolution (SISR) methods fail to fully utilize these properties and therefore apply a uniform reconstruction strategy to the whole image. In this article, we propose a novel saliency-driven unequal single-image reconstruction network in which the demands of various regions in the superresolution (SR) process are distinguished by saliency maps. First, we design a new gradient-based saliency analysis method to produce more accurate saliency maps with imagewise annotations. The method utilizes the superiority of a multireception field to extract both high-level features and low-level features. Second, we propose a novel saliency-driven gate conditional generative adversarial network, where the saliency map is regarded as a medium during the training procedure of the whole network. The saliency map is regarded as a pixelwise condition in a generator to enhance the training capability of the network. Additionally, we design a new loss function that combines normalized content loss, saliency-driven perceptual loss, and gate-control adversarial loss to further refine details of texture-complex areas for RSIs. We evaluate the performance of our algorithm and compare it with many other state-of-the-art SR methods using a remote-sensing data set. The experimental results show that our approach achieves the optimal outcome in salient areas. Our method attains the best effect on global quality and visual performance.

Pansharpening is a fundamental and crucial task in the remote sensing community. For remote sensing images, there is a significant difference in demands for spatial and spectral resolution in different regions. From this perspective, we propose a new pansharpening method using objectness based saliency analysis and saliency guided deep residual network to boost the fusion accuracy. We first develop an objectness based saliency analysis by incorporating texture feature and objectness measurements to estimate saliency values in images and thereby help discriminate different demands for spatial improvement and spectral preservation. Inspired by the impressive performance of deep learning, we subsequently construct a saliency guided deep residual network to implement pansharpening. In addition, in order to produce images with subtler details, we design a new loss function, the normalized mean square error, particularly for the pansharpening task. Experiments support the superiority of our proposal over six competing methods.

The goal of the remote sensing image fusion is to inject the detail information extracted from panchromatic (PAN) images to multispectral (MS) images with minimized spectral distortion. However, different regions in the image may practically have different demands on the spatial and spectral resolution. In this paper, a new fusion method for remote sensing images based on salient region extraction is proposed. By introducing the hybrid visual saliency analysis, information in the PAN and MS image are automatically partitioned into two categories: salient and non-salient regions. Then, a sub-region fusion strategy is applied to fuse the non-salient and salient regions respectively. For non-salient regions, such as farmland and mountains, the wavelet transform is used in the process of spatial infusion to suppress spectral distortion. As for salient regions like residential areas, the windowed IHS transform is carried out for its merits of effective integration of spatial and spectrum information. Experimental results demonstrate that our proposal achieves a better balance between spatial injection and spectral maintenance in different regions.

In remote sensing images, demands for spectral and spatial resolution vary from region to region. Regions with abundant texture and well-defined boundaries (like residential areas and roads) need more spatial details to provide better descriptions of various ground objects while regions such as farmland and mountains are mainly discriminated by spectral characteristic. However, most existing fusion algorithms for remote sensing images execute a unified processing in the whole image, leaving those important needs out of consideration. The employment of diverse fusion strategy for regions with different needs can provide an effective solution to this problem. In this letter, we propose a new saliency-driven fusion method based on complex wavelet transform. First, an adaptive saliency detection method based on clustering and spectral dissimilarity is presented to generate saliency factor for indicating diverse needs of the two kinds of resolutions in regions. Then, we combine nonlinear intensity-hue-saturation transform with multiresolution analysis based on dual-tree complex wavelet transform in order to complement each other's advantages. Finally, saliency factor is employed to control the detail injection in the fusion, helping to satisfy different needs of different regions. Experiments reveal the validity and advantages of our proposal.