Optical remote sensing imagery is often compromised by cloud cover, making effective cloud-removal techniques essential for enhancing the usability of such data. We designed a novel structural representation-guided generative adversarial network (GAN) framework for cloud removal, in which structure and gradient branches are integrated into the network, helping the model focus on the structural representations of ground objects during image reconstruction. Different from previous methods that concentrate on recovering pixel information, we emphasize learning the structural information of remote sensing images. We then utilize error feedback to fuse features from the structural auxiliary branch, guiding the image reconstruction process. During the training phase, synthetic cloud images are used to supervise the optimization of the cloud-removal network, while real cloud images are employed in an adversarial training manner for unsupervised learning to improve the generalization ability of the network. Additionally, multitemporal revisit images from remote sensing satellites are employed as auxiliary inputs, aiding the network to remove thick clouds reliably. We evaluated our framework on a dataset derived from SEN12MS-CR, and the proposed method outperformed classical cloud-removal methods in both objective performance and subjective visual quality. Furthermore, compared to other methods, our approach achieved superior cloud-removal results on real images.

The recent advancement of generative foundational models has ushered in a new era of image generation in the realm of natural images, revolutionizing art design, entertainment, environment simulation, and beyond. Despite producing high-quality samples, existing methods are constrained to generating images of scenes at a limited scale. In this paper, we present MetaEarth - a generative foundation model that breaks the barrier by scaling image generation to a global level, exploring the creation of worldwide, multi-resolution, unbounded, and virtually limitless remote sensing images. In MetaEarth, we propose a resolution-guided self-cascading generative framework, which enables the generating of images at any region with a wide range of geographical resolutions. To achieve unbounded and arbitrary-sized image generation, we design a novel noise sampling strategy for denoising diffusion models by analyzing the generation conditions and initial noise. To train MetaEarth, we construct a large dataset comprising multi-resolution optical remote sensing images with geographical information. Experiments have demonstrated the powerful capabilities of our method in generating global-scale images. Additionally, the MetaEarth serves as a data engine that can provide high-quality and rich training data for downstream tasks. Our model opens up new possibilities for constructing generative world models by simulating Earth's visuals from an innovative overhead perspective.

Hyperspectral remote sensing images (HSIs) have unique advantages in urban planning, precision agriculture, and ecology monitoring since they provide rich spectral information. However, hyperspectral imaging usually suffers from low spatial resolution and high cost, which limits the wide application of hyperspectral data. Spectral super-resolution provides a promising solution to acquire hyperspectral images with high spatial resolution and low cost, taking RGB images as input. Existing spectral super-resolution methods utilize neural networks following a single-shot framework, i.e., final results are obtained by one-stage spectral super-resolution, which struggles to capture and model the complex relationships between spectral bands. In this article, we propose a spectral-cascaded diffusion model (SCDM), a coarse-to-fine spectral super-resolution method based on the diffusion model. The diffusion model fits the real data distribution through stepwise denoising, which is naturally suitable for modeling rich spectral information. We cascade the diffusion model in the spectral dimension to gradually refine the spectral trends and enrich spectral information of the pixels. The cascade solves the highly ill-posed problem of spectral super-resolution step-by-step, mitigating the inaccuracies of previous single-shot approaches. To better utilize the potential of the diffusion model for spectral super-resolution, we design image condition mixture guidance (ICMG) to enhance the guidance of image conditions and progressive dynamic truncation (PDT) to limit cumulative errors in the sampling process. Experimental results demonstrate that our method achieves state-of-the-art performance in spectral super-resolution. Codes can be found athttps://github.com/Mr-Bamboo/SCDM.

Deep neural networks (DNNs) have been widely applied in remote sensing, and the research on its adversarial attack algorithm is the key to evaluating its robustness. Current adversarial attack methods primarily prioritize maximizing the attack success rate, disregarding the imperceptibility of the generated adversarial noise to human visual perception. Moreover, research on adversarial sample transferability has mostly focused on cross-model and cross-dataset scenarios, overlooking the investigation of adversarial attacks across different resolutions, while the rarely studied cross-resolution adversarial attacks are critical for remote sensing with different resolutions. In this article, we propose a novel method for generating imperceptible adversarial samples for cross-resolution remote sensing images based on implicit neural representations (INRs). By mapping the discrete images to a continuous neural functional space, we explicitly guarantee the visual quality of adversarial samples and decouple the model input from the image resolution. To enhance the visual fidelity of the generated adversarial samples, a multiscale discriminative learning scheme is proposed for the optimization process. For cross-resolution adversarial attacks, we align with images of different resolutions and generate cross-resolution adversarial perturbation by benefiting from the natural properties of the continuous resolution of INRs. To validate the effectiveness of our method, we compare it with the existing adversarial attacking methods using four evaluation metrics. Experiments show that our method achieves the best results in terms of attack success rate, imperceptibility, and cross-resolution attack transferability. Our code will be made publicly available.

When the state feedback control is carried out for the positioning anti-swing of bridge crane, the values of state variables such as trolley displacement and trolley speed, load swing angle and swing angle speed need to be obtained in real time. In practical engineering application, it is easy to measure the displacement of trolley with sensor, but it is difficult to measure the load swing angle and swing angle speed. To solve this problem, a fuzzy state feedback control method on the base of T-S fuzzy observer is proposed in this paper. Firstly, a nonlinear T-S fuzzy observer is designed to estimate the trolley speed, load swing angle and swing angle speed signal online by the trolley position signal. Then the fuzzy state feedback controller is designed, the original system state variables and observer errors are formed into an augmented state vector, and the stability of the augmented system is proved by Lyapunov stability theory. Finally, the feasibility and effectiveness of this method are verified by MATLAB simulation.

Hyperspectral image synthesis overcomes the limitations of imaging sensors and enables low-cost acquisition of hyperspectral images with high spatial resolution. Using RGB as a conditional input for hyperspectral generation is promising and valuable, as it can leverage abundant existing multispectral/RGB images without the intervention of hyperspectral sensors. However, most existing generation methods follow one-to-one mapping frameworks and ignore generation diversity. In addition, the current evaluation metrics of hyperspectral generation are based on the similarity with the reference image, which cannot reflect the diversity of the generated spectra. In this paper, we propose a novel method for diverse hyperspectral remote sensing image generation based on the diffusion model. The diffusion model uses a denoising model to gradually remove noise from the normal distribution and generates the hyperspectral data step-by-step with the conditional RGB image as input. To address the high-dimensional noise prediction problem caused by a large number of bands in the hyperspectral image, we introduce a conditional VQGAN that maps the high-dimension hyperspectral data into a low-dimension latent space and conduct the diffusion process in the latent space. The latent-diffusion process makes the diffusion process faster and more stable. The conditional VQGAN decodes hyperspectral images from the latent code generated by diffusion, with the conditional RGB image as input, which restricts the diversity to a specific object distribution. We also design two new metrics to evaluate the generation spectral diversity. Experiments on the IEEEgrss_dfc_2018dataset demonstrate that our method can synthesize highly diverse hyperspectral data. In addition, the rationality of the proposed metrics is also verified.

Hyperspectral image (HSI) synthesis, as an emerging research topic, is of great value in overcoming sensor limitations and achieving low-cost acquisition of high-resolution remote sensing HSIs. However, the linear spectral mixing model used in recent studies oversimplifies the real-world hyperspectral imaging process, making it difficult to effectively model the imaging noise and multiple reflections of the object spectrum. As a prerequisite for hyperspectral data synthesis, accurate modeling of nonlinear spectral mixtures has long been a challenge. Considering the above difficulties, we propose a novel method for modeling nonlinear spectral mixtures based on implicit neural representations (INRs) in this article. The proposed method learns from INR and adaptively implements different mixture models for each pixel according to their spectral signature and surrounding environment. Based on the above neural mixing model, we also propose a new method for HSI synthesis. Given an RGB image as input, our method can generate an accurate and physically meaningful HSI. As a set of by-products, our method can also generate subpixel-level spectral abundance as well as the solar atmosphere signature. The whole framework is trained end-to-end in a self-supervised manner. We constructed a new dataset for HSI synthesis based on a wide range of Airborne Visible Infrared Imaging Spectrometer (AVIRIS) data. Our method achieves a mean peak signal-to-noise ratio (MPSNR) of 52.36 dB and outperforms other state-of-the-art hyperspectral synthesis methods. Finally, our method shows great benefits to downstream data-driven applications. With the HSIs and abundance directly generated from low-cost RGB images, the proposed method improves the accuracy of HSI classification tasks by a large margin, particularly for those with limited training samples.

Few-shot learning provides a new way to solve the problem of insufficient training samples in hyperspectral classification. It can implement reliable classification under several training samples by learning meta-knowledge from similar tasks. However, most existing works perform frequency statistics, which may suffer from the prevalent uncertainty in point estimates (PEs) with limited training samples. To overcome this problem, we reconsider the hyperspectral image few-shot classification (HSI-FSC) task as a hierarchical probabilistic inference from a Bayesian view and provide a careful process of meta-learning probabilistic inference. We introduce a prototype vector for each class as latent variables and adopt distribution estimates (DEs) for them to obtain their posterior distribution. The posterior of the prototype vectors is maximized by updating the parameters in the model via the prior distribution of HSI and labeled samples. The features of the query samples are matched with prototype vectors drawn from the posterior; thus, a posterior predictive distribution over the labels of query samples can be inferred via an amortized Bayesian variational inference approach. Experimental results on four datasets demonstrate the effectiveness of our method. Especially given only three to five labeled samples, the method achieves noticeable upgrades of overall accuracy (OA) against competitive methods.

Self-supervised learning (SSL) has recently been introduced to remote sensing (RS) to learn in-domain transferable representations. Here, we propose a semantic decoupled representation learning for RS image change detection (CD). Typically, the object of interest (e.g., building) is relatively small compared to the vast background. Different from existing methods expressing an image into one representation vector that may be dominated by irrelevant land-covers, we disentangle representations of different semantic regions by leveraging the semantic mask. We additionally force the model to distinguish different semantic representations, which benefits the recognition of objects of interest in the downstream CD task. We construct a dataset of bitemporal images with semantic masks in an effort-less manner for pre-training. Experiments on two CD datasets show our model outperforms ImageNet, indomain supervised pre-training, and several recent SSL methods.

Generating hyperspectral images (HSI) from RGB imagery can obtain HSI with both high spatial and spectral resolution, which overcomes the limitations of imaging hardware conditions. Many HSI generation methods target learning a 3-n mapping from RGB to HSI, lacking concern of the spectral categories and spatial distribution. In this paper, we propose an HSI generation method preserving the band structure similarity and semantic information. We design an MLP based classifier and trained it on many spectra of known semantic categories. Then we use it to map the spectra to semantic space and constrain the distance between the embedding of generated spectra and that of the real ones. Meanwhile, a structure similarity loss is added to constrain the spatial information. Experiment results verified the superiority of the proposed method.

In deep learning based road extraction from remote sensing images, the network often learns some features that are not essential to road discrimination, such as trees, buildings, etc. In fact, there is no causal relationship between these features and road discrimination, which will lead to error and omission in final results. In this paper, we propose a novel road extraction network to enhance essential features, including local and global line features and geometric features along the road direction. Multi-scale Line Enhancement Module utilize hough transform to enhance line featues of different scales. Neighboring road prediction branch make the network pre-dict the distance and direction of each pixel to the neighboring road, which helps the network to focus on geometric features along the road direction. Experimental results on the deepglobe dataset show that the network is able to obtain bet-ter road extraction results by enhancing essential features that have a causal relationship with the task. Codes are available at https://github.com/zaoyifan/EssentialFeatures.

High-resolution hyperspectral remote sensing images are of great significance to agricultural, urban, and military applications. However, collecting and labeling hyperspectral images are time-consuming, expensive, and usually heavily rely on domain knowledge. In this article, we propose a new method for generating high-resolution hyperspectral images and subpixel ground-truth annotations from RGB images. Given a single high-resolution RGB image as its conditional input, unlike previous methods that directly predict spectral reflectance and ignores the physics behind it, we consider both imaging mechanism and spectral mixing, introduce a deep generative network that first recovers the spectral abundance for each pixel, and then generate the final spectral data cube with the standard USGS spectral library. In this way, our method not only synthesizes high-quality spectral data existing in the real world but also generates subpixel-level spectral abundance with well-defined spectral reflectance characteristics. We also introduce a spatial discriminative network and a spectral discriminative network to improve the fidelity of the synthetic output from both spatial and spectral perspectives. The whole framework can be trained end-to-end in an adversarial training paradigm. We refer to our method as “Physics-informed Deep Adversarial Spectral Synthesis (PDASS).” On the IEEEgrss_dfc_2018dataset, our method achieves an MPSNR of 47.56 on spectral reconstruction accuracy and outperforms other state-of-the-art methods. As latent variables, the generated spectral abundance and the atmospheric absorption coefficients of sunlight also suggest the effectiveness of our method.