Land Surface Temperature (LST) is an important parameter representing surface energy, which is of great significance for monitoring urban heat islands, agricultural drought, and global climate. The high-resolution LST observations will address new applications in hydrology. The spatiotemporal fusion method can generate LST with high temporal and spatial resolution. However, missing data due to cloud cover becomes a main limitation to improving the accuracy of spatiotemporal fusion models. The purpose of the fusion of LST is image prediction and generation, and deep learning generative models provide an effective idea to solve this problem. Therefore, we proposed a conditional generative model fusion network (CGMFN) for LST generation in this paper. Firstly, based on the generated model, we construct an unsupervised generation network that simultaneously learns and iterates, which can generate fine-spatiotemporal-resolution LST data from reference images with missing values. Then, spectral normalization was applied to generators and discriminators to stabilize the training process. The pre-training mechanism was adopted to improve the iteration efficiency of the model. We tested and evaluated the model in the Heihe River Basin using FY-4A LST and MODIS LST datasets. Compared to different methods, CGMFN produces a lower RMSE (average0.97). In practical applications, CGMFN can reduce the influence of reference image missing values on fusion results and generate land surface temperature products with reliable accuracy.

The potential applications of high spatiotemporal resolution land surface temperature (LST) products are extensive. However, the tradeoff between spatial and temporal resolutions of remote sensing data has significantly constrained the availability of such LST products. Existing spatiotemporal fusion methods appear to encounter certain limitations. This article proposes a two-stage hierarchical spatiotemporal fusion network (THSTNet) to fuse MODIS LST products and Landsat LST products. THSTNet adopts the shift windows (Swin) transformer architecture and employs a two-stage process to reconstruct the fine-resolution image at the target time. The innovation within THSTNet is multifaceted. It combines spatiotemporal mapping (ST mapping) with deep learning, enhancing the extraction of global information for fusion results; leveraging self-attention computation, it adopts a two-stage structure to improve the understanding of intricate LST changes. In addition, it incorporates a texture converter module aimed at enhancing spatial details within the reconstruction results. Validation of the model’s predictions was conducted using actual images and ground observations, affirming the high reliability of THSTNet’s predictive outcomes. Compared with two traditional methods [spatial and temporal adaptive reflectance fusion model (STARFM) and enhanced STARFM (ESTARFM)] and four deep learning-based methods [enhanced deep convolutional spatiotemporal fusion network (EDCSTFN), generative adversarial network (GAN)-based spatiotemporal fusion model (GANSTFM), spatiotemporal temperature fusion network (STTFN), and multistream fusion network (MSNet)], THSTNet demonstrated superior performance (average root-mean-square error (RMSE) is below 1.3 K and average structural similarity index (SSIM) is 0.939). The prediction results of THSTNet also maintain high consistency with ground observations (the average RMSE is 2.3 K and the average$R^{2}$is 0.9). The code will be available athttps://github.com/HuPengHua2021/THSTNet.

Crop identification is a fundamental task in remote sensing image interpretation. The rapid development of Unmanned Aerial Vehicle (UAV) has revolutionized the acquisition of super-high-resolution images. Compared with remote sensing ones, the fact that UAV images are easier to be flexibly acquired and contain more information brings opportunities for refined semantic segmentation. Recently deep learning methods have gained substantial popularity in the field of semantic segmentation. However, the practical application of deep learning methods is often hindered by heavy labeling tasks and computational resources. The object-based Markov random field (OMRF) offers a both time and labor cost-effective unsupervised approach. Nevertheless, the high-spatial heterogeneity exhibited by crops in UAV images brings great difficulties to the application of this method. To address these challenges, this letter introduces RA-OMRF model, an unsupervised approach that improves OMRF model through our newly proposed pre-processing step named Region Aid (RA). RA is used to increase the amount of data for categories with fewer samples to alleviate the problem of high-spatial heterogeneity of ground object categories, thereby improving the performance of the OMRF model. Compared to five deep learning models, our method achieves matching or even higher segmentation accuracy (for instance an overall accuracy of 97.43% on the UAV Dataset DWST) in crop identification tasks of UAV images, while reducing time and labor costs.

Semantic segmentation is one of the most important tasks in the field of remote sensing. As the spatial resolution increases, the remote sensing images can capture more detailed information and make hierarchical semantic interpretation possible. However, hierarchical semantic segmentation encounters high heterogeneity not only within the intra-layer classes but also among inter-layer classes. It brings challenges to semantic segmentation methods such as the convolutional neural network (CNN). In this article, a hierarchical self-learning knowledge inference model (HSKIM) based on the Markov random field (MRF) model is proposed for hierarchical semantic segmentation of remote sensing images. The HSKIM model introduces a new framework that integrates the advantages of CNN-based data feature learning and MRF-based hierarchical semantic inference. It contains three modules: data learning module ($\boldsymbol {D}$), inference units generation module ($\boldsymbol {I}$), and self-learning knowledge inference module ($\boldsymbol {S}$). The module$\boldsymbol {D}$uses CNN to learn specific data features layer by layer and extract preliminary geographical objects as the initial results. The module I refines the geographical objects using a novel boundary-preservation trick to generate more accurate inference units with clear geographical meaning. The module S introduces a hierarchical object-based MRF model to implement semantic inference among intra-layer and inter-layer inference units, guided by the spatial interactions and geographical criteria. This module can self-learn and update the relationship between classes iteratively and provide the final result. Experiments on the GID dataset with hierarchical classes, alongside 12 state-of-the-art CNN-based methods, validate the effectiveness and robustness of the proposed HSKIM model. The code of this article is available athttps://github.com/iichengzi/HSKIM.

Convolutional neural network (CNN) has been widely used in semantic segmentation for remote sensing images, and it has achieved great success. Due to the diversity of the spatial distribution of terrestrial objects in remote sensing images, it is difficult to effectively learn general geographical laws and apply them to a specific image. To introduce geographical knowledge into the CNN model more effectively, a self-learning-update CNN model (SLU-CNN) is proposed in this letter. It learns the representation of specific spatial dependence among different objects according to the CNN result, and then incorporates it with the CNN result to make semantic inference available. The proposed method mainly involves two modules. First, geographical objects generated from the CNN result are used as inference units. Second, the spatial dependence between inference units is learned to build a specific adaptive geographical relationship. And then, it is embedded as an adaptive penalty term into an object-based Markov random field model to achieve the collaboration between the CNN result and the semantic inference. Our method provides a general data-knowledge dual-driven framework for the deep neural network. Experiments of the GID and Sentinel-2 datasets validate the effectiveness of the proposed method by comparing it with different state-of-the-art CNN methods.

Deep learning methods have been widely studied in the semantic segmentation field of the remote sensing image. Training images play an important role in these methods; however, each training image usually contains not only the generalization information of each land category but also the specific interclass context between different categories. The specific interclass context prevents deep learning methods from focusing on generalization information learning during training and limits the performance on different data distributions. This article proposes a generalization sampling learning method of deep convolutional neural network (GSL-CNN) to emphasize generalization information learning for the semantic segmentation of remote sensing images. The proposed method develops a new CBR sampling strategy that contains three modules: category grouping ($\mathbf {\boldsymbol {C}}$), basic unit extraction ($\mathbf {\boldsymbol {B}}$), and random combination ($\mathbf {\boldsymbol {R}}$). Module$\mathbf {\boldsymbol {C}}$collects each land category map and strips away the specific interclass context from the raw annotated image. Module$\mathbf {\boldsymbol {B}}$extracts basic units with different granularities from each land category map, and each basic unit can keep the generalization information of this category. Module$\mathbf {\boldsymbol {R}}$aims to enhance the robustness against different data distributions by randomly picking basic units of different categories and randomly generating their interclass context. The new GSL-CNN method integrates the CBR sampling strategy with the convolutional neural network (CNN) model for semantic segmentation. Experiments on different remote sensing datasets and 15 state-of-the-art CNN models validated that the proposed method has the potential of improving the generalization ability of the CNN method from a sampling perspective.

Semantic segmentation is one of the most important tasks in the field of remote sensing image processing. Many methods have been proposed to realize it at the pixel granularity or object granularity. Specifically, the pixel-based methods usually can effectively extract the detailed information and edges, and the object-based methods can keep the internal consistency of each land cover or land use. The Markov random field (MRF) model provides a statistical way to combine the advantages of both pixel and object granularities together. However, current MRF-based methods still face a problem, that is, how to ensure that the advantages of different granularities will complement each other, not that disadvantages will affect advantages. To solve this problem, a new multigranularity edge-preservation optimization is proposed in this letter. The proposed method first represents the image with a series of granularities from the object to the pixel by downsampling. Then, the MRF model is defined on each granularity. By defining an edge set for each granularity, during the process of downsampling, the proposed method can continuously correct edges while maintaining intraclass consistency. Experiments of Gaofen-2 and SPOT5 demonstrate the effectiveness of the proposed method. Moreover, the proposed method can be also used as the postprocessing step for deep learning. The experiment of the Pavia University hyperspectral image illustrates it for an instance of DeepLab v3+.

High spatiotemporal resolution land surface temperature (LST) data are essential for dynamic monitoring and prediction in climate change research. Due to the limitations of remote sensing instruments, the current platforms have difficulty achieving a compromise between high spatial and temporal resolutions for LST products. In this study, we propose a spatiotemporal fusion network for land surface temperature based on a conditional variational autoencoder (CVAE-LSTFM). First, an improved network is designed based on the CVAE by reconstructing an encoder and a decoder. To generate fine LST images based on dense time series, a variational inference model is formulated to integrate coarse and fine LST image pairs in variational autoencoded latent space. In addition, a new compound loss function for the proposed training method is designed to reduce the effects of noise and outliers. Then, a pretraining mechanism is adopted to optimize the network training process, and the parameters can be transferred to the training network of the CVAE to accelerate network convergence. Finally, a novel weighting strategy that considers spatiotemporal variations in LST (LST consistency weighting) is employed to solve the spatiotemporal heterogeneity problem caused by the rapid changes in LST. The method is quantitatively tested and evaluated in the Heihe River Basin using FY-4A LST and MODIS LST from September 2019. Compared with two traditional models and two deep learning-based models, CVAE-LSTFM yields lower RMSE (average<1.26 K) and LPIPS (learning perceptual image patch similarity, average<0.13) values and higher SSIM (structural similarity, average>0.96). In practice, CVAE-LSTFM can generate high spatiotemporal resolution LST values (hourly LST with a 1 km spatial resolution) with high accuracy, quality, and robustness.